BEST PRACTICES FOR DEPLOYING APPLICATIONS ON ORACLE RAC

ENVIRONMENT
A Technical White Paper
By
Harish Kumar Kalra
EmailID: [email protected]

Page 1 of 8

TABLE OF CONTENTS
BEST PRACTICES FOR DEVELOPING APPLICATIONS ON ORACLE RAC
ENVIRONMENT ............................................................................................................. 1
1. PURPOSE ................................................................................................................ 3
2. SCOPE ..................................................................................................................... 3
3. BEST PRACTICES FOR DEPLOYING APPLCATION ON RAC.............................. 3
3.1.

Partitioning ....................................................................................................... 3

3.2.

Connection Management ................................................................................. 5

3.3.

Database Sequence Objects............................................................................ 5

3.4.

Using Full Table Scans Sensibly...................................................................... 6

3.5.

Data Change Rate / Transaction Commit Frequency....................................... 6

3.6.

Sorted Hash Cluster......................................................................................... 6

4. REFERENCES ......................................................................................................... 8

Page 2 of 8

1.

PURPOSE

A well-designed and tuned database that services a well-designed application can scale admirably on RAC
with multiple nodes, as compared to single SMP box. The performance of an application on RAC depends
on how well the application performs on a single instance. In theory, no changes are required to run an
application on RAC. There are, however, specific issues, which must be considered when movingan
application to RAC, or developing a new application for RAC.
Cache Fusion is the primary feature of Oracle Real Application Cluster that enables applications to scale on
multiple instances. Cache Fusion enables each instance to locate the most useful version of a block of
data for its needs.
This document identifies the known best practices to develop and deploy applications for Oracle Real
Application Cluster Database 10g. It also identifies potentially sub optimal design and deployment
elements.

2.

SCOPE

This document covers the best practices known for developing applications for RAC, moving existing
applications to RAC, and provides basic guidelines to gain performance and scalability benefits, while
designing an application for RAC databases.
This documents is focussed on Oracle Real Application Clusters and must not to be compared or used with
other cluster database technologies available in the market.

3.

BEST PRACTICES FOR DEPLOYING APPLCATION ON RAC

When applications run in parallel environments, database sessions may contend for global resources,
which may result in high response time for end users. There are various reasons for contention for global
resources. Contention may occur when different components of the same application are running the same
pattern of transaction on different nodes within the cluster, executing the same set of DML statements, high
rate of data insertion etc.
There are no definitive design and coding standards for Real Application Cluster. These best practices
have been drawn from past experience and from various other resources. The aim of these guidelines is to
reduce contention for global resources like data/index blocks, locks, CPU cycles or memory in a Real
Application Cluster environment. Some of the guidelines and known best practices are documented below
and each guideline or best practice can have priority depending upon urgency of the problem.

3.1.

Partitioning

Partitioning is a technique used to distribute the workload among different nodes within a cluster either by
redirecting task of the same nature to a specific node within the cluster or partitioning the underlying data
like partitioning the tables and indexes etc.

3.1.1. Application Partitioning

Application partitioning is not a requirement to deploy the Real Application Cluster but it is a widely used
methodology to distribute the workload among available nodes within the cluster. Applications with a
modular design are the best candidates for application partitioning.
First, identify the business functional components of the application and identify the areas of the
application, which can be segregated
Once the main business functional components have been segregated, schedule the component(s) or
module(s) to run on a specific node within the cluster using Oracle 10g services or connection load
balancing.
Lets consider an application with 2 modules MA and MB, which have few inter-module dependencies and
operate on their own datasets. Now if we have an Oracle 10g 2 node cluster then we can direct the
connections for module MA on instance 1 and connections for module MB on instance 2 using the Oracle
10g Services or the Client side workload distribution function. As show in the below diagram, one database

Page 3 of 8

service for each module is defined. All the transactions of the application module MA will execute on
instance 1 because instance 1 is the preferred instance defined for service MA. In the same way all
transactions from the application module MB will execute on instance 2 because instance 2 is the preferred
instance for service MB. In case of failure of any instance, both services can fail over to surviving database
instance.

Database Tier

Mid-Tier
SERVICE MA

Preferred Instance

SERVICE MB

INSTANCE 1

INSTANCE 2
Preferred Instance

Guidelines:

Identify the main business functional components / modules within the application.

Execute the module specific requests / transactions on nodes dedicated for the application module.

Implement a failover option for the application module to run on other nodes within the cluster in
case of node failure using Oracle 10g Services.

3.1.2. Data Partitioning

Data partitioning is an approach to divide large data tables into a number of smaller tables. This division
eases the management of data as well as improves the overall performance of accessing the data in an
efficient manner. There are different methods available within Oracle to partition the data. The type of
application will determine the most appropriate option. Partitioning can help minimize a lot of contention for
data blocks provided that the right method of partitioning is chosen.
The type of environment/application is a crucial factor in choosing the type of partitions to create.
The application could be a straightforward OLTP system or a mix of OLTP and batch processing. Some
environments are pure batch or DSS in nature, sometimes referred to as data warehouses. Each of these
environments has a predominant type or nature of transaction. It could be INSERT intensive, UPDATE
intensive, or SELECT intensive, or a mixture of these. It is common to see OLTP applications that do a lot
of all three types of activities, including DELETEs. Some applications also have lots of DDL embedded in
their systems. Transaction volume (rate of inserts, updates, and deletes) is another critical factor that needs
to be considered while choosing the correct partitioning strategy.

Page 4 of 8

Guidelines:

For applications being developed for Oracle Real Application Cluster databases, it is recommended
that logic is built into the application code so that data is directed into each partition based on the
dataset that will be used for processing on that node. It can be termed a guidance system, which
ensures that data is inserted and processed on a particular node only. A guidance system can be
application logic or any third party data dependant routing software like Tuxedo. Assume a fournode cluster where each node is processing data for particular region. Upon arrival of data for
insertion, the guidance system will route the data so that specific nodes insert region specific data.

3.1.3. Index Partitioning

Index partitioning is equally or even more important than table partitioning, as index blocks can become
hotter than the table blocks because of high concurrency at block level due to the smaller row length of
index keys. Also, index blocks are the source of contention for the database because they are high
concurrency in nature.
Guidelines:

Use Global indexes for OLTP applications wherever possible.

Applications causing lot of inserts may benefit from using Reverse Key indexes as they are
designed to eliminate index hot spots on insert applications. These indexes are excellent for insert
performance, but they are limited in that they cannot be used for index range scans.

3.2.

Connection Management

In OLTP kind of applications, it is a common scenario that applications disconnect and reconnect to the
database server repeatedly. Every new connection to the database server incurs an operating system cost
of creating a new process or thread. Most of recursive SQLs are executed at the database server level to
initiate the new database connection, which causes lots of soft parsing because of a new session. This
adds to the load on the library cache latches.
All these overheads can be avoided if database connections are retained as long as they are required.
Guidelines:

Use Pro*C/C++ Connection Pooling and OCI Session pooling features for applications built in
Pro*C/C++.

Use third party transaction monitors like Tuxedo, CORBA etc while building the application.

Use Oracle ODBC, OCI and JDBC connection pooling features wherever possible

3.3.

Database Sequence Objects

Oracle Sequences are used for generating numbers that grow sequentially. There is no need to develop a
local sequence generator because Oracle sequences are so powerful and configurable. The Oracle
RDBMS updates the data dictionary when a database session selects from a sequence, when it is not
cached. So every select internally invokes a DML and commit, which is very expensive in RAC, since
System Commit Numbers (SCNs) must be synchronized across the cluster when sequences are used
extensively.
In RAC environments, performance degradation due to heavy (ab)use of sequences and a high insert rate
can manifest in different ways:

Contention for the SQ enqueue. If you see contention for SV enqueue, the problem is complicated
and Oracle Support Services might be required.

Buffer busy waits for various classes of blocks such as data, index leaves, or segment headers.

Page 5 of 8

TX enqueue waits on the index blocks may also be visible.

In certain cases, ITL waits are also possible when a sufficient number of transaction slots are not reserved
or available. This could be due to a very low setting for PCTFREE combined with a default value for
INITRANS.
Guidelines:

Use a different sequence on each instance to avoid single point of contention

Cache large number of sequence values without any order using CACHE with the NOORDER
clause of Oracle sequence. Please note that the CACHE option is usable only if the application is
able to accept gaps. This would be the easiest method.

Use ASSM (Automatic Segment Space Management) Tablespaces

3.4.

Using Full Table Scans Sensibly

Frequent full table scans in a Real Application Cluster can impede performance and scalability as the
private interconnect is used for transferring blocks, and that can eat into network bandwidth. This can
cause a delay for other required and important blocks such as index blocks, undo blocks, header blocks,
and any messages. The goal is to reduce interconnect traffic and bring down latency. With a lot of full table
scans, interconnects could get overloaded, thereby flooding the private network.
Full Table Scan are not always bad; accessing the data through full table scan could be better in some
cases, but it has been observed that they are a common issue in RAC performance problems. There are
various ways to identify the database sessions performing lot of full table scans. The best way to identify
whether this is an issue is to monitor the wait events in V$SESSION_WAIT dynamic performance view.
Database sessions waiting on db file scattered read wait event are performing full table scans.

Guidelines:

Use correct Index after evaluating the selectivity of data.

Using Application partitioning and guidance system, explained in section 3.1.2 could be helpful to
reduce traffic over interconnect.

3.5.

Data Change Rate / Transaction Commit Frequency

In a single database, when a user commits or rolls back any transaction, the database server creates a
commit record, flushes the redo log buffer to disk for durability and releases the held locks for current
transactions.
In addition to work being done by a single instance upon transaction commit, the RAC instance needs SCN
propagation, global synchronization and may also require control file synchronization. Hence, extra caution
must be exercised in RAC environments when choosing a commit interval or frequency preferably for the
batch programs. Some business applications perform commits more often than required. Developers and
users must be aware of the performance trade offs that accompanies a high commit rate.
Guidelines:

Wherever possible keep higher commit sizes for batch programs.

Design provision in the application so that end user can set the commit frequency or interval
(Number of transactions per commit) for the large batch programs.

3.6.

Sorted Hash Cluster

Page 6 of 8

Consider a call detail record (CDR) data structure that is common in telecommunication environments. For
billing and auditing purposes, each call is to be recorded and associated with its origin. Usually, calls are
stored as they arrive from the switch. This data can be retrieved later in FIFO order or a customized order
when customer bills are generated.
Though it is common practice to capture this data in two standard tables, a sorted hash cluster is a better
solution as it allows inexpensive access to the sorted list of call records for any given subscriber and allows
the billing application reading the call records to walk through the sorted list in a FIFO manner. Any
application in which data is always consumed in the order in which it is inserted will see tremendous
performance improvements by using this type of optimized table structure.

Guidelines:

Use Sorted Hash Cluster wherever feasible and possible. This table structure better supports firstin first-out (FIFO) type data processing applications, where data is processed in the order it was
inserted. Such applications are typically found in the telecommunications and manufacturing
environments.

Page 7 of 8

4.

REFERENCES
Document Number

Title

B14251-01

Application Developers Guide Fundamentals

Oracle 10g Real Application Cluster Handbook by K
Gopalkrishanan

Page 8 of 8