Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX

Nitin Vengurlekar – Oracle Corporation

6/26/2004
 Introduction................................................................................................................. 3
1. Adopt a standard where all disks are “protected”................................................... 5
2. Choose Appropriate RAID level based on the Service level Agreements.............. 5
3. Standardize on hyper-volume size(s)...................................................................... 6
4. Start utilizing automated resource provisioning tools ............................................ 7
5. Utilize a storage provisioning model. ..................................................................... 7
6. Database System growth definition and extrapolation. .......................................... 8
7. Striping Methodlogy .............................................................................................. 8
8.Operating System (OS) Queue Depth for MetaVolumes .......................................... 10
9. Multiplex Online Redo Logs ................................................................................ 11
10. Adopt a standard of 2 HBAs for every 600GB of storage allocated to a host.... 11
11. Standardize on the use of dynamic multi-pathing software................................ 12
12. Adopt a standard 6:1 fan-out ratio for SAN ports............................................... 12
13. Deploy file mapping tools and automated back-end performance tuning tools to
automatically detect and possibly correct “hot spots” on physical spindles............. 13
14.Implement Oracle enterprise Manager (EM). ...................................................... 13
Conclusion .................................................................................................................... 14

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
2
Oracle and EMC have invested in a joint Global Alliance since 1995, designed to better
service the over 25,000 joint customer installations we share. The focus of our Alliance is
reducing the time, cost and effort of designing and maintaining Oracle on EMC
environments:
• Dedicated on-site EMC engineers at Oracle headquarters in Redwood Shores, CA.
• Dozens of jointly documented engineering best practices whitepapers available at
www.emc.com
• A suite of joint Accelerator deployment services build by Oracle Consulting and
EMC Services to provide customer rapid deployment options, available at
www.emc.com/solutions
• A Joint Service Center (JSC) with Oracle where dedicated Support staff from both
companies reside in the same facility, providing faster support resolution
• For the past five years, Oracle and EMC have both run core production Global IT
operations on a combination of Oracle databases and Applications with EMC
Automated Networked Storage

The purpose of this document is to share best practices that are field tested and promoted
by Oracle and EMC, and to provide a set of defined standards and the methodology
necessary to achieve a lower Total Cost of Ownership though higher availability and
reduction of cost.

•
o
o
•
o
o
o

This paper outlines standard system configurations that can be applied to the majority of
systems that are running supported versions of Oracle and EMC. Additionally, this
document does not specifically address 10g ASM. There are two other documents on
OTN that discuss ASM Best Practices (one of which specifically addresses ASM-EMC
Best Practices). Therefore, 10g customers wanting to deploy ASM as the storage
manager over EMC Symmetrix and DMX systems should review the ASM Best Practices
as well as this document. However, all the best practices listed here are complimentary
and applicable to those defined in the ASM Best Practices paper.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
3
This white paper will provide useful information for EMC and Oracle environments,
using the 80-20 rule. This implies that 80% of systems deployed, the requirements will
be met in this white paper, while the environment making up the other 20% will require
additional analysis. These best practices define and streamline the methodology by
establishing configuration standards that reduce the number of decision points while still
providing sufficient flexibility and adaptability for future changes. A second aspect of the
Oracle and EMC recommendations centers around standardization on operational and
support tools that automate the system diagnostic and analysis process. This
standardization will assist customers in deploying systems with less effort and greater
success while increasing system availability and reliability by proactively identifying and
diagnosing potential system problems before they occur. The audience for this document
will be anyone who will be responsible for implementing these standards, such as storage
and systems administrators as well as database administrators. Additionally the audience
should be familiar with Oracle and EMC storage concepts.

! " # $ % &
'( % % % '

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
4
Benefit: Loss of one disk drive will not preclude operations from continuing.

All database systems test or production should be built over RAID configurations. This
configuration can either be done at the hardware, software, or both hardware and
software levels. Each method will be discussed in, “7.Stripping Methodologies”.
Currently the EMC’s Symmetrix product line supports RAID 10 or Parity RAID on
DMX systems.

Benefit: All frames can be deployed uniformly, and time to implement is greatly
reduced.

Determining the Service Level Agreements, or SLA, required is the first step. The next
step requires cost, availability, and performance to be prioritized and compared to SLA
requirements. Typically, lower cost deployments, will mean lower availability and
performance and visa versa. Mirroring offers the highest level of performance and
availability, but with higher cost. Parity RAID provides the lowest cost implementation
while providing modest performance and availability. Lastly, when considering
availability, determine the recovery requirements. Recovery requirements must be
considered during the data layout process along with the various technologies that can be
used for recovery.

Mirroring technologies (a.k.a. RAID1) have higher costs due to 100% overhead that is
required to provide the hardware protection [redundancy] for the volumes. Within the
storage array, there is a redundant physical volume for every device seen by the host.
Thus, if a physical disk is lost, the application and database will not experience any down
time or outage.

When choosing mirroring protection, there are a couple of choices, including

RAID1, RAID10, and RAID01. RAID1 is the foundation of hardware RAID
mirroring and provides the highest availability and performance. RAID10 and
RAID01 are similar in that they both provide hardware-protected volumes and
striped data; however, the difference is in the order that this is accomplished.
RAID 10 mirrors the volume on two different physical disks, and then stripes the
data across the logically grouped RAID1 protected volumes. RAID01 stripes the
data across logically grouped volumes, and then mirrors this logical RAID group
across physical disks. While RAID10 and RAID01 both provide the highest
performance, the difference is in the availability. RAID10 provides the highest
level of performance and availability, while RAID01 provides high performance
but only slightly better availability then PARITY RAID. RAID1 and RAID10 are
both recommended for availability; however, RAID10 will provide the highest
performance.
Oracle – EMC Best Practices and Standardization
For Symmetrix and DMX
5
Parity RAID allows more usable space per raw volume than mirroring. However, with
Parity RAID, the Parity calculation may add overhead to the IO performance, and drive
rebuild times are typically elongated. For applications that are looking for the lowest cost
solution, with Mean Time to Recover (MTTR) requirements that are not stringent, Parity
RAID should be utilized. For applications that have increased performance requirements
with more demanding MTTR requirements, Mirroring should be utilized.

It is also recommended to incorporate Hot Spares into the storage configuration. Hot
spares are physical disks that are not utilized until a physical disk is lost. Once a physical
disk is lost the hot spare will be enabled and began rebuilding. The rebuild time for
RAID1 (and RAID10) is typically shorter then the rebuild time for Parity RAID.

Benefit: This practice will decrease the time to implement dramatically and provide
a standardized deployment practice.

Standard hypervolume or Logical Units (LUNs) sizing provides an even distribution of

LUN size-to-capacity. It also provides management flexibility for storage administration.

On EMC’s Symmetrix and DMX storage arrays, using a standard LUN size(s) will
reduce complexity and management costs. It is recommended that the number of LUN
sizes within a Storage frame to two sizes. For example, maintain 8Gb LUN sizes for
physical disks smaller than 100Gb (such as the 36 or 73 Gb disks), and 12 Gb LUN sizes
for physical disks greater than 100Gb (such as the 146Gb disks). Standardized LUN sizes
will provide maximum flexibility and increase the manageability of allocations within the
array. Allocating storage similarly, regardless of the application type, helps to facilitate
shared storage in SAN based architectures. Once these standard LUN sizes are
determined, they can then be logically grouped together to create Meta-Volumes, or
RAID10 LUNs.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
6
!
Benefit: This tool will increase manageability and reduce time to implement

Automated Resource Provisioning tool(s) allow quick and easy creation of allocation
pools and storage provisioning policies based on business requirements, automate the
end-to-end provisioning process from the array through the fabric to the host, and quickly
identify free storage available from the hosts, HBAs, ports, arrays, or devices. Currently
for most customers, this is a manual process involving several weeks of deployment lead-
time for storage purchase, bin file change requests, zoning requests, implementation, and
fibre runs.
Utilization of provisioning tools in conjunction with a “storage on demand” model will
greatly reduce the lead-time for deployment. EMC’s Automated Resource Manager is
one such tool used to automate storage provisioning.

"
Benefit: Pre-populated frames provide rapid deployment, and reduced personnel
requirements.

Storage arrays with capacity according to forecasted usage are pre-staged in the storage
infrastructure. These arrays will be carved into the standard hyper-volume sizes, to
facilitate rapid configuration. When an application needs space, the administrator can
deploy storage interactively using automated provisioning tools, for example EMC’s
Automated Resource Manager. Time to deploy is now only dependant on the time
required to perform the fiber pulls from the host to the switch. This significantly reduces
the time to deploy and also reduces the personnel required for request fulfillment.
Additionally, a pre-populated array lends itself to more flexibility in wider database
striping. This is discussed in more detail in Striping Section.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
7
# $ %
Benefit: Request fulfillment is accurate and predictable.

To deploy a database system that will scale accordingly and meet the appropriate storage
needs, the expected growth rate and acceleration of growth should be clearly defined
before deployment. Using this information the Storage Administrator can succinctly
define and generate a storage map for a database system and reduce the Mean Time To
Respond for a storage request. For example, if the a database is initially sized at 200 GB,
but the growth rate is 50 percent w/in 6 months, and the growth acceleration is high, then
given this information, the Storage Admin would prudently build a 400Gb Meta-volume.
This Meta-volume should be rounded to the next 4 or 8 Meta-volume boundary.

In addition, by creating the initial database with the projected growth factored in, the
database can be spread out wider across the backend of the frame. This will take
advantage of the processing power on the backend of the storage array and maximize the
bandwidth and IOPs available to the database.

&
Benefit: I/O throughput will be increased and spread over multiple devices
simultaneously. Main goal here is to balance the load evenly across all Fibre
Adapters (FA) ports, Disk Adapter (DA) ports and disk spindles.

Striping may be accomplished in three ways: host based striping, meta-volumes striping,
or double-striping (plaidding). For storage management simplicity, we recommend that
meta-volumes be utilized for all database volumes. If this is not possible, then use LVM
striping where possible or necessary.

Meta-volumes are Symmetrix striped logical volumes with a stripe size of 2 cylinders
(960Kb). This stripe size is very accommodating for Oracle and has been tested to be the
optimum for the track size/cache implementation within the Symmetrix. By using meta-
volumes, the amount of cache utilized is also optimized providing increased I/O
efficiency. Meta-volumes have the additional benefit of reducing the number physical
LUNs presented to the host, thereby minimizing system startup time for hosts with large
storage requirements. Moreover, some platforms have a device limitation, which meta-
volumes help to mitigate. We recommend as a general rule to standardize on Meta-
volumes that contain either 4 disks or 8 disks. This general rule simplifies space
allocation and Meta-volume management. It also ensures that the members are placed on
disks that are controlled by different DA processors. Therefore, the Symmetrix will be
able to utilize the full potential of the back end.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
8
For those implementations where LVM striping is preferred, deploy a stripe depth of
1MB, 512KB, or 256KB when using LVM striping. Sufficient research studies
conducted by Oracle and EMC have concluded that 1MB is the optimum stripe size for
LVM striping, but 512 and 256 also provide good performance when spread across
maximum number of physical spindles. The stripe size usage will depend on the
appropriate O/S level configuration and may be affected by LVM limitations on some
systems. Make sure that your LVM and OS parameters are configured to accommodate
the largest possible IO request size.

When building/creating an Oracle database over the Symmetrix/DMX, determine total

size of the database; including all redo logfiles and controlfiles. Exclude the archive log
space for now, but factor in any predictable growth if possible. For large databases with
high growth, use the aggregate database size to determine how many 8 disk Meta-
volumes are required, rounding up to the next 8 disk meta-volume. For smaller databases
with moderate growth, use the aggregate size to determine how many 4 disk Meta-
volumes are required, rounding up to the next 4 disk meta-volume.

At this point a critical decision has to be made regarding host logical volume creation.
There are two possible options, and each one has merits and drawbacks. We will explore
each one.

1. Layout Option 1 – Double Stripping (Plaidding)

Use plaidding to create one large logical volume (and subsequently a filesystem).
Plaidding means to use a host based Volume Manager to stripe the data across
multiple Meta-volumes. For example, suppose the aggregate database size is 400 Gb,
then you would create four 8 disk Meta-volumes that are 12Gb each. Then use a
LVM to create a logical volume over those 4 Meta-volumes to create a 400Gb logical
volume and then create a filesystem over this logical volume.

This configuration strategy affords the greatest simplicity, since datafile (or any
Oracle physical file) placement becomes a non-factor, and file IO is evenly
distributed across all disks. However, this strategy may have two drawbacks:

a. It will create a very large filesystem, and recoverability may become as issue.
However, implementing a journal-ed filesystem can mitigate this issue.
b. Sequential read performance may suffer.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
9
 2. Layout Option 2 – Sets of Meta-volumes
Create as many 4 disk or 8 disk Meta-volumes as needed, and present these LUNs as
individual logical volumes. Create a filesystem over each logical volume, and then let
the DBA manage which Oracle datafiles reside in which filesystem.

The main benefit of this configuration is that it allows for unexpected growth as well
as provides a simpler chargeback policy. However, it does incur the costs of datafile
placement and manual file IO load balancing.

Lastly, unless disk space utilization is a high priority, we do not recommend

having multiple databases share the same LUNs (Meta-volumes).

The following are benefits of discrete database-to-LUN:

o Increased availability and performance of the database.
o Simpler streamlined approach to LVM volume management, and other tools; such
as EMC’s TimeFinder, and EMC’s Replication Manager. By having the database
on its own LUNs and Volume Group, the database can easily be cloned for
backup, recovery, testing, and development purposes.
o Simplified movement of database files across the Storage Area Network.
o Simplified performance monitoring, tuning and load rebalancing tasks.

If this recommendation documented above does not fit into your environment, then
additional analysis maybe required.

' ( $
Benefit: Databases IO will not be limited by OS queuing policies.

The queue depth is the maximum number of IO commands that the SCSI driver will
attempt to queue to the HBA. Meta volumes can theoretically accept more physical I/Os
request than a single disk. However, since it is presented to the O/S as one ‘physical’
volume there may be a problem with I/O queuing in the O/S. This is where the operating
system ‘queue depth’ parameter should be adjusted in the O/S. The normal EMC default
is to take the O/S default and multiply it by the number of hyper-volmes in the Meta-
volume.
Additionally, with PowerPath implemented, there will be multiple active paths to the
Meta volumes which make it less likely that you will encounter queue depth issues.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
10
) %
Benefit: Critical recovery files will be protected from a majority of hardware-based
outages.

Online redo logs are crucial files for instance, crash, and media recovery. The loss of
these files can cause data to be lost irrevocably. Oracle recommends multiplexing of
redo logs due to the critical nature of these files.
Multiplex Oracle online redo logs to offer additional protection from failures, even in
RAID configurations. This will protect the database in case of loss of an active online
log. Ideally, each database will utilize redo log multiplexing with redundant log copies
spread across physically separate disks. However, achieving this goal is extremely
difficult from a storage management perspective. Therefore, multiplexing across Meta-
volumes or stripe sets is a generally accepted best practice

Most customers do not need to adopt a standard of multiplex archived redo logs, unless
MTTR and recoverability are critical to the application. Multiplexing of archived redo is
not as critical as protection of online redo. Loss of archived redo affects recoverability,
but does not result in immediate data loss.

* $ $ #** $

Benefit: Standardizing on HBA layout will reduce the time to implement

This calculation is a field-proven starting point for efficient I/O performance for systems
that do not have a current workload footprint. As I/O characteristics become available, re-
evaluation of HBA utilization will identify whether or not a system is efficiently using
the resources it currently has installed. If appropriate, HBAs may need to be added to
support additional growth. If total HBAs on a given system could be reduced, these
HBAs are candidates for re-deployment elsewhere in the enterprise. Tools identifying
the I/O utilization from HBA, down through SAN and into the storage array will provide
the needed insight for this evaluation. The Workload Analyzer component of the EMC
Control Center Suite, is one such tool providing a centralized view of all three areas of
I/O utilization.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
11
 $
$
Benefit: Hosts are able to instantly respond to changing dynamics of the shared
storage infrastructure.

It is recommended to implement more than one HBA per server, for purposes of failover
as well as for multi-pathed performance. It is also highly recommended to use a dynamic
multi-pathing tool, such as Power Path, to help distribute the IO across the multiple
HBAs, as well as help to assuage in HBA failures. As more systems are added to the
shared infrastructure, performance of static multi-pathing tools may be affected, causing
additional management overhead and possible application availability implications as
data needs to be re-distributed across other paths. Dynamic multi-pathing will continue
to adjust to changes in I/O response times from the shared infrastructure as the needs of
the application and usage of the shared infrastructure change over time.

#+ $ $
Benefit: This will reduce time to implement, by adopting a standard starting point

Fan out is the number of HBAs (from the host) that are mapped through the FC-Switch,
to a single EMC FA port. For example in a 6:1 fan-out ratio, six HBAs from a single
host or from various hosts, are plugged into the FC-Switch and mapped to a single FA
port.

For new deployments, particularly those with no known I/O profile, usage of a target 6:1
SAN fan-out is a good starting point. The 6:1 fan-out recommendation is based on 2Gb
FC infrastructure. Proactively monitor resource utilization to evaluate shared
infrastructure efficiency. This will guide future decisions for additional use of shared
infrastructure components, as well as evaluate the need for additional or fewer HBAs to
adequately service the application. For systems with low I/O requirements, sharing SAN
and storage resources over the target 6:1 ratio provides more efficient use of system
resources, while still providing the necessary response time and availability. The
Workload Analyzer component of the EMC Control Center Suite is a tool providing
insight to SAN and storage port utilization.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
12
 $
$

Benefit: These tools will provide insight into physical data placement and
proactively identify potential I/O hotspots within the shared storage infrastructure.

While the recommendations in this document are designed to provide the best average
performance on a shared resource, the possibility exists for two areas on the same
physical spindle to be in contention. For example, the redo logs from two separate
database implementations could potentially reside on the same physical spindle. Using
automated back-end storage tuning tool, such as EMC’s Optimizer, will not only detect
and correct this area of contention, but will also help in minimizing the effort to
investigate areas of contention, as well as to non-disruptively correct the issue increases
efficiency of all affected personnel.

For 9iR2 databases and above, it is possible for users with Oracle Enterprise Manager
(EM) installed, to use the File Mapping feature to view the database layout from an EMC
perspective. See the Oracle Documents on OTN for details on configuration and usage.

!
Benefit: This tool provides insight into the physical data placement of the database

Oracle Enterprise Manager (EM) is designed to address the specific complexities facing
administrators when managing the environment. Built with robust functionality for
managing both small and large sets of systems, EM automates critical operations to
reduce task time and the risk of errors which increases as the number of systems goes up.
Enterprise Manager supports the management of the complete Oracle environment - from
the end-user through to the backend storage supporting the applications being accessed.
Enterprise Manager provides functionality to view and manage the entire application
infrastructure - regardless of its complexity - and contains specific facilities to manage
each component of the application individually. All this functionality is integrated into a
single console providing a consistent user interface and context when managing multiple
systems that interrelate. Additionally, as applications and systems scale Enterprise
Manager provides rich functionality that simplifies and automates the management of
many systems.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
13
This paper touches on several topics at a high level and is intended to serve as quick start
guide. For further detailed reading on these topics as well as other Joint Oracle-EMC
papers see the following URLs:
http://www.emc.com/techlib/

http://www.emc.com/partnersalliances/partner_pages/oracle3b.jsp

See URL, http://otn.oracle.com/deploy/availability/index.html, for detailed documents on

o High Availability
o Backup and Recovery
o Oracle related storage technologies

Oracle and EMC are continuing to work together to provide not only integrated solutions,
but also on knowledge based materials such as best practices and white papers. Look for
new documents at the aforementioned EMC and Oracle websites.

Oracle – EMC Best Practices and Standardization

For Symmetrix and DMX
14