Oracle Real Application Cluster & High

Availability
Oracle Real Application Cluster (RAC) is a cluster database. Compared to
traditional shared-nothing architecture of single instance database, oracle Real
Application Cluster has shared cache architecture and also shared-disk approaches to
provide highly available and scalable solution for any business applications.

One of the key components of Oracle enterprise grid architecture by oracle is the Real
application Cluster (RAC)

Oracle RAC is a unique technology that provides high availability and scalability for all
application types. The Oracle RAC infrastructure is also a key component for implementing
the Oracle enterprise grid computing architecture. Having multiple instances access a single
database prevents the server from being a single point of failure. Oracle RAC enables you to
combine smaller commodity servers into a cluster to create scalable environments that
support mission critical business applications. Applications that you deploy on Oracle RAC
databases can operate without code changes.

Unique feature called Fast Application Notification (FAN) configuration in oracle Real
application cluster helps load balancing when available service status change. Available
service status can change during a scheduled outage for patching or any other regular
maintenance task. Service status can also change due to unexpected faults such as node
reboot, database service unavailability etc.

Oracle Real Application Cluster has some cost effective options while scaling up or scaling
down your size of application usage volume. Olden days normally businesses have to pre-
plan the scale of their application few years in advance. Though they did not made use of
the full capacity at the initial stage of deployment, the hardware had to be in place in
anticipation of high growth in the future. This kind of planning did add additional cost to
Businesses in planning their hardware and also software licensing. In case of miscalculation
of capacity, did cost further cost in moving their application to higher capacity servers in
later stages. Once the application is deployed and up and running and if you decide to
migrate to different server, it does cost additional overhead in terms of building servers,
outages, manpower etc etc. So migrating the application is not an easy exercise and if your
business is cost conscious (All business are cost conscious I guess) this type of migration
related to scaling up or scaling down should be avoided.

So what is the cost effective solution to avoid these overheads? That is where Oracle Real
application Cluster comes to your rescue.

1
 Why Oracle Real application Cluster?

To accommodate unplanned, unanticipated growth in any business application, oracle real

application cluster can be built from standard, commodity-priced processors, with standard
network and storage components. Since oracle Real application Cluster is built on very
foundation of Grid computing, when you require more processing power or wish to scale up,
simply add another similar commodity priced server. Adding new server doesn’t require you
to bring down the database and this can be done without interrupting the service with users
still accessing the database. Oracle Real application cluster supports up to 100 nodes in any
given Cluster configuration.

If your business decides that you need to scale down your application for whatever reason,
you could de-commission some of the servers, without bringing down your database and
while users still accessing the database without any interruption.

What is a Cluster?

A cluster comprises of multiple interconnected servers or computers that appear as if they

are one single server to end users and applications.

Oracle Database 10g Real Application Clusters (RAC) enables the clustering of the Oracle
Database. A RAC database comprises of multiple instances residing on different computers
to access a common database residing on shared storage.

The basic principle behind the Real Application Cluster is greater throughput and scalability
due to the combined power of multiple instances running on multiple servers

Some of the main benefits of Oracle Real Application Cluster (RAC)

 Scalability: Service capacity can be expanded simply by adding servers to existing
cluster.
 Availability round the clock (24/7): Zero downtime for database applications.
 Relatively lower computing cost: Cost can be relatively reduced by using low-cost
commodity hardware.
 Grid Computing: Very foundation of Oracle Grid computing is oracle Real Application
Cluster (RAC.

Main Components of Oracle Real Application Cluster 10g.

It comprises of two main components

1. Oracle Clusterware
2. Oracle Software

In RAC10g Clusterware is called CRS layer which resides below Oracle software layer.
Second layer is the Oracle software itself

2
 RAC is the Oracle Database option that provides a single system image for multiple servers
to access one Oracle database. In oracle Real application cluster, each Oracle instance
usually runs on a separate server.

However when it comes to managing and looking after your production Real application
Cluster, it may not be practical to find the commands you needed in the right time and
possibly you don’t want to keep searching for the right command and tips when you have a
major production issue and the users cannot access the database.

This book "Oracle Real Application Cluster Field DBA Admin Handbook" describes how to
administer the Oracle Clusterware and Oracle Real Application Clusters (Oracle RAC)
architecture and provides an overview of these products. Describes services and storage
and how to use RAC scalability features to add and delete instances and nodes in RAC
environments.

This book "Oracle Real Application Cluster Field DBA Admin Handbook" also describes, how
to use the Server Control (SRVCTL) utility to start and stop the database and instances,
manage configuration information, and to delete/add or move instances and services

Troubleshooting section describes how to interpret the content of various RAC-specific log
files, search on Metalink and also useful reference section with relevant Metalink Document
reference and Weblinks

The Practical Handbook on Oracle Real Application Cluster for Oracle

DBA’s.

The “Oracle Real Application Cluster Field DBA Admin Handbook” is a practical guide
for Oracle DBA’s who manage and administer the Oracle Real Application Cluster High
availability solution.

This book provides practical steps to administer Oracle Real Application Cluster (RAC). It’s
like a field guide rather than an in depth manual. It’s written for the DBA’s to handle the day
to day challenges faced by DBA’s managing RAC on the job.

Architecture

 What is a Cluster
 Components of RAC
 RAC Storage
 ASM
 raw device
 Cluster File system
 Voting Disk and Cluster Registry

3
What is a cluster ?
A cluster comprises of multiple interconnected servers or computers that appear as if they
are one single server to end users and applications.

What is RAC ?

RAC stands for Real Application Clusters. It allows multiple nodes in a clustered system to
mount and open a single database that resides on shared disk storage. Should a single
system fail (node), the database service will still be available on the remaining nodes.

In RAC database, comprises of multiple instances, however there will be only one database.

A non-RAC database is only available on a single system. If that system fails, the database
service will be down (single point of failure).

Oracle Database 10g Real Application Clusters (RAC) enables the clustering of the Oracle
Database. A RAC database comprises of multiple instances residing on different computers
to access a common database residing on shared storage.

Why Real Application Cluster ?

The basic principle behind the Real Application Cluster is greater throughput and scalability
due to the combined power of multiple instances running on multiple servers

Real Application Cluster provides high availability and scalability for all application types.
The RAC infrastructure is also a key component for implementing the Oracle enterprise grid
computing architecture. Multiple instances access a single physical database prevents the
server from being a single point of failure. Combining smaller servers into a cluster can be
achieved to create scalable environment that support mission critical business applications.

Real application Cluster uses Oracle Cluster ware for the infrastructure to bind multiple
servers so that they operate as a single system.

Oracle Clusterware is a portable cluster management solution that is integrated with the
Oracle database. The Oracle Clusterware is also a required component for using Real
application Cluster.

In Oracle Database 10g, Oracle Provides you with an integrated software solution that
addresses cluster management, event management, application management, connection
management, storage management, load balancing and availability. These capabilities are
addressed while hiding the complexity through simple-to-use management tools and
automation.

Oracle Real Application Clusters 10g provides an integrated cluster ware layer that delivers
a complete environment for applications.

4
 Oracle Real Application Cluster (RAC) uses Oracle Clusterware for the infrastructure to bind
multiple servers so that they operate as a single system

Main Components of Oracle Real Application Cluster 10g

It comprises of two main components

1. Oracle Clusterware
2. Oracle Software

In RAC 10g Clusterware is called CRS layer which resides below Oracle software Layer.
Second layer is the Oracle software itself.

Oracle Real Application Cluster (RAC) is the Oracle Database option that provides a single
system image for multiple servers to access one Oracle database. In RAC, each Oracle
instance usually runs on a separate server.

Oracle Clusterware is software, enables the servers to be bound together to operate as if

they were one server. The Oracle Clusterware comprises of two clusterware components: a
voting disk to record node membership information and the Oracle Cluster Registry (OCR) to
record cluster configuration information. In Oracle Clusterware each node is connected to a
private network by way of a private interconnect.

The Oracle Clusterware comprises several background processes that facilitate cluster
operations such as Cluster Synchronization Service (CSS) and Event Management (EVM).

What are the Real Application Cluster Main Processes ?

The main processes involved in Oracle RAC are primarily used to maintain database
coherency among each instance. They manage what is called the global resources.

 LMON: Global Enqueue Service Monitor

 LMD0: Global Enqueue Service Daemon
 LMSx: Global Cache Service Processes. Where x can range from 0 to j
 LCK0: Lock Process
 DIAG : Diagnostibility Process

There are several tools that are used to manage the various resources available on the
cluster at a global level. Some of the tools used are Server Control (SRVCTL), DBCA and
Enterprise Manager.

Oracle Clusterware is a portable cluster management solution that is integrated with the
Oracle database. The Oracle Clusterware enables you to create a clustered pool of storage
to be used by any combination of single-instance and RAC databases.

5
 Oracle Clusterware is the only clusterware that you need for most platforms on which RAC
operates. You can also use clusterware from other vendors if the clusterware is certified for
RAC.

The combined processing power of the multiple servers can provide greater throughput and
scalability than is available from a single server. RAC is the Oracle Database option that
provides a single system image for multiple servers to access one Oracle database. In RAC,
each Oracle instance runs on a separate server.

RAC is a unique technology that provides high availability and scalability for all application
types. The RAC infrastructure is also a key component for implementing the Oracle
enterprise grid computing architecture. Having multiple instances access a single database
prevents the server from being a single point of failure. RAC enables you to combine smaller
commodity servers into a cluster to create scalable environments that support mission
critical business applications

What are the Storage Principles for RAC Software and CRS ?

The Oracle Software 10g Real Application Clusters installation is a two-phase installation in
the first Phase, You install CRS. In the second phase, you install the Oracle Database
software with RAC components and create a cluster database.

The oracle home that you use for the CRS software must be different from the one that is
used for the RAC Software. CRS and RAC software is installed on cluster shared storage.

Note:- Cluster Software and Oracle software is usually installed on a regular file system that
is local to each node. This permits online patch upgrades without shutting down the
database and also eliminates the software as a single point of failure.

Do you need special hardware to run RAC ?

RAC requires the following hardware components:

 A dedicated network interconnect - might be as simple as a fast network connection

between nodes; and a shared disk subsystem.
RAC and Shared Storage Technologies

1. Supported shared storage for Oracle grids:

 Network attached Storage
 Storage Area Network
2. Supported file systems for Oracle grids:-
 Raw Volumes
 Cluster file system
 ASM

6
 Components of RAC
Components of Real application Cluster

Oracle Real Application Cluster 10g provides you with an integrated software solution that
addresses cluster management, event management, application management, connection
management, storage management, load balancing and availability. These capabilities are
addressed while hiding the complexity through simple-to-use management tools and
automation.

Cluster platforms depend on the cluster management program to track the cluster node
status. Basically there are two types of cluster managers, first one is oracle supplied and
second one is vendor supplied.

In Oracle 10g , oracle provided clusterware can be used independently or on top of the
vendor provided cluster software.

Cluster Ready Service’ (CRS) is Oracle provided portable clusterware, which is portable and
works on all the platforms to implement RAC database solution.

RAC Software Principles

Addition to the background processes found in single instance database, You see few
additional background processes associated with RAC instance. These processes are
primarily used to maintain database coherency among each instance. They manage what is
called the global resources:-

 LMON:- Global Enqueue Service Monitor

 LMD 0:- Global Enqueue Service Daemon
 LMSx :- Global Cache Service Processes
 LCK 0 :- Lock Process
 DIAG :- Diagnosibility Process
The Oracle Clusterware Software Components

When Oracle Clusterware operates, several platform-specific processes or services will also
be running on each node in the cluster to support Oracle Clusterware.

Oracle Clusterware Processes on UNIX-Based Systems

 crsd—Performs high availability recovery and management operations such as

maintaining the OCR and managing application resources. This process runs as the root
user, or by a user in the admin group on Mac OS X-based systems. This process restarts
automatically upon failure

7
 evmd—Event manager daemon. This process also starts the racgevt process to manage
FAN server callouts
 ocssd—This process runs as oracle user and manages cluster node membership. Any
failure of this process results in cluster restart
 oprocd—Cluster Process monitor. This process only appears on platforms that do not use
vendor clusterware with Oracle Clusterware
Components of Oracle Clusterware

The Oracle Clusterware includes two important components: the voting disk and the Oracle
Cluster Registry (OCR). The voting disk is a file that manages information about node
membership and the OCR is a file that manages cluster and Oracle Real Application Clusters
(Oracle RAC) database configuration information

1. Voting Disk: - Manages cluster membership by way of a health check and arbitrates
cluster ownership among the instances in case of network failures. RAC uses the voting disk
to determine which instances are members of a cluster. The voting disk must reside on
shared disk. For high availability, Oracle recommends that you have multiple voting disks.
The Oracle Clusterware enables multiple voting disks.

Note:- If you define a single voting disk, then you should use external mirroring to provide
redundancy.

2. OCR File :- Cluster configuration information is maintained in Oracle Cluster Registry file.
OCR relies on a distributed shared-cache architecture for optimizing queries against the
cluster repository. Each node in the cluster maintains an in-memory copy of OCR, along with
an OCR process that accesses its OCR cache.

When OCR client application needs to update the OCR, they communicate through their
local OCR process to the OCR process that is performing input/output (I/O) for writing to the
repository on disk.

The OCR client applications are Oracle Universal Installer (OUI), SRVCTL, Enterprise Manger
(EM), Database Configuration Assistant (DBCA), Database Upgrade Assistant(DBUA), NetCA
and Virtual Internet Protocol Configuration assistant (VIPCA). OCR also maintains
dependency and status information for application resources defined within CRS, specifically
databases, instances, services and node applications.

Note:- The name of the configuration file is ocr.loc and the configuration file variable is
ocrconfig.loc

Oracle Cluster Registry (OCR) :- Maintains cluster configuration information as well as

configuration information about any cluster database within the cluster. The OCR also
manages information about processes that Oracle Clusterware controls. The OCR stores
configuration information in a series of key-value pairs within a directory tree structure. The
OCR must reside on shared disk that is accessible by all of the nodes in your cluster. The
Oracle Clusterware can multiplex the OCR and Oracle recommends that you use this feature
to ensure cluster high availability

8
 Note:- You can replace a failed OCR online, and you can update the OCR through supported
APIs such as Enterprise Manager, the Server Control Utility (SRVCTL), or the Database
Configuration Assistant (DBCA).

Cluster Synchronization Services (CSS) is the cluster manager on all platforms on oracle 10g

CSS - Cluster Synchronization Services - Component responsible for controlling which nodes
are members of the clusters. When node joins or leaves the cluster, CSS notifies the other
nodes of the change in configuration. If this process fails, then the cluster will be restarted.
Under Linux, CSS is implemented by the ocssd daemon, which runs as the root user.

New feature introduced in Oracle10g, Cluster Ready Services (CRS) provides many system
management services and interacts with the vendor clusterware to coordinate cluster
membership information.

CRS - Cluster Ready Services - Manages resources (databases, instances, services, listeners,
VIPs, application processes, GSD, ONS) Configuration is stored in OCR. When status of
resource changes CRS emits an event. CRS will try to restart resource up to 5 times before
giving up. Under Linux, CRS is implemented as the crsd process, which runs with root
privileges. In the event of failure this process restarts automatically.

RAC Storage
Storage in Oracle Real Application Clusters

Storage for Real application cluster databases must be shared. In other words, datafiles
must reside on cluster file system, shared raw devices or Automatic Storage Management
(ASM) disk group.

The shared storage must include datafiles, undo tablespaces for each instance and also
online redo log files. It is highly recommended by oracle to use server parameter file spfile
(SPFILE) instead of parameter file (PFILE).

Shared storage Technologies and RAC

1. Supported shared storage for Oracle grids
 Network attached Storage
 Storage area Network
2. Supported file systems for Oracle grids
 Raw Volumes
 Cluster file system
 ASM

Storage Area Network (SAN) represents the evolution of data storage technology.
Traditionally, on client server systems, data was stored on devices either inside or directly
attached to the server. Next in the evolution scale came Network Attached Storage (NAS)

9
 that took the storage devices away from the server and connected them directly to the
network.

In RAC deployment, choosing the appropriate file system is critical. Because traditional file
systems do not support simultaneous mounting by more than one system, you must store
files in either raw volumes without any file system, or on a file system that supports
concurrent access by multiple systems.

Oracle Cluster File systems

Oracle cluster file system (OCFS) is a shared file system designed specially for oracle Real
Application Cluster

Automatic Storage Management (ASM)

 Automatic and high-performance cluster file system

 Manages Oracle Database files
 Data spread across disks to balance load
 Integrated mirroring across disks
 Solves many storage management Challenges

The automatic storage management(ASM) is a new feature in oracle Database 10g. It

provides a vertical integration of the file system and the volume manager that is specially
built for Oracle Database files

The ASM distributes I/O load across all available resources to optimize performance while
removing the need for manual I/O tuning. ASM facilitates management of dynamic database
environment by allowing DBA’s to increase the database size without having to shut down
the database to adjust the storage allocations.

The ASM can maintain redundant copies of data to provide fault tolerance.

Note:- ASM is the oracle’s strategic and stated direction as to where oracle database files
should be stored. However OCFS will continue to be developed and supported for those who
are using it.

Comparison between RAW or CFS

 Using CFS
1. Simple management
2. Use of OMF with RAC
3. Single Oracle Software installation
4. Autoextend
 Using raw
1. Performance

10
 2. Use when CFS not available
3. Cannot be used for archive log files

You can use a cluster file system or place files on raw devices.

Cluster file systems provide the following advantages:-

 Greatly simplify the installation and administration of RAC
 Use of Oracle Managed Files with RAC
 Single Oracle Software installation
 Autoextend enabled on oracle data files
 Uniform accessibility to archive logs in case of physical node failure

RAW devices Implications

 Raw devices are always used when CFS is not available or not supported by oracle
 Raw devices offer best performance without any intermediate layer between oracle and
the disk

Autoextend fails on raw devices if the space is exhausted

ASM
What is Automatic Storage Management

Automatic storage management (ASM) is a new feature in Oracle Database 10g from oracle .
It integrates file system and the Logical Volume Manager (LVM) . In ASM Volume Manager is
specifically built for Oracle database files. The ASM can provide management for single SMP
machines or across multiple nodes of a cluster for Oracle Real Application Clusters support.

Automatic Storage Management (ASM) simplifies administration of Oracle related files by

allowing the administrator to reference disk groups rather than individual disks and files,
which are managed by ASM.

Manual I/O tuning can be eliminated while ASM distributes input/output (I/O) load across all
available resources to optimize performance while removing the need for manual I/O tuning.

The ASM has the flexibility of maintaining redundant copies of data to provide fault
tolerance, or it can be built on top of vendor-supplied reliable storage mechanisms. Data
management in ASM is basically done by choosing the desired reliability and performance
characteristics for classes of data rather than with human interaction of per-file basis.

Automated storage management gives the time to DBA’s by increasing their ability to
manage larger databases and more of them with increased efficiency.

11
 Automatic Storage Management (ASM) is a feature in Oracle Database 10g/11g that
provides the database administrator with a simple storage management interface that is
consistent across all server and storage platforms. ASM provides the performance of async
I/O with the easy management of a file system.

Some of the Key features of ASM

 Stripes files rather than logical volumes

 Enables online disk reconfiguration and dynamic rebalancing
 Provides adjustable rebalancing speed
 Provides file based redundancy
 Supports only Oracle files
 It’s cluster aware
Why ASM ?

Some of the storage management features with ASM include

 Striping
 Mirroring
 Asynchronous I/O
 Direct I/O
 SAME and Load Balancing
 Is automatically installed as part of the base code set

ASM includes striping and mirroring to provide balanced and secure storage. The level of
redundancy and the granularity of the striping can be controlled using templates. The new
ASM functionality can be used in combination with existing raw and cooked file systems,
along with OMF and manually managed files.

Direct I/O

By making use of Direct I/O, higher cache hit ratio can be achieved. Buffered I/O uses most
important resources like CPU and memory. In case of buffered I/O Oracle blocks are cached
both in the SGA and in the file system buffer cache.

Buffered I/O fills up the file system cache with Oracle Data, where as using the Direct I/O
allows non-Oracle data to be cached in the file system much more efficiently.

Key Features and Benefits of ASM

The ASM functionality is controlled by an ASM instance.

12
The main components of ASM are disk groups, each of which comprise of several physical
disks that are controlled as a single unit. The physical disks are known as ASM disks, while
the files that reside on the disks are know as ASM files.

The ASM divides a file into pieces and spreads them evenly across all the disks. The ASM
uses an index technique to track the placement of each piece. Traditional striping
techniques use mathematical functions to stripe complete logical volumes. The ASM includes
mirroring protection without the need to purchase a third-party Logical Volume Manager.
One unique advantage of ASM is that mirroring is applied on file basis, rather than on a
volume basis. Therefore, the same disk group can contain a combination of files protected
by mirroring, or not protected at all.

The ASM supports data files, log files, control files, archive logs, Recovery Manager (RMAN)
backup sets, and other Oracle database file types. The ASM supports Real Application
Clusters (RAC) and eliminates the need for a cluster Logical Volume Manager or a cluster file
system.

Note:- ASM is shipped with the database and available as part of base code set and there is
no need to go through a separate installation in the custom tree installation. It is available in
both the Enterprise Edition and Standard Edition installations.

One of the flexible feature of ASM is it does not eliminate any existing database functionality
which uses non ASM files. Existing database are able to operate as they always have been.
New files may be created as ASM files, while existing ones are administered in the old way
or can be migrated to ASM.

In ASM, at the top of the new hierarchy, you can find what are called ASM disk groups. Any
single ASM file is contained in only one disk group. However, a disk group may contain files
belonging to several databases, and a single database may use storage from multiple disk
groups.

ASM files are always spread across all ASM disks in the disk group.

The ASM disks are partitioned in allocation units (AU) of on megabyte each. An AU is the
smallest contiguous disk space that ASM allocates. The ASM does not allow physical blocks
to split across AUs.

ASM General Architecture

To use ASM, you must start a special instance called an ASM instance before you start your
database instance.

ASM instances manage the metadata needed to make ASM files available to ordinary
database instances. Both ASM instances and database instances have access to a common
set of disks call disk group. Database instances access contents of ASM files directly,
communicating with an ASM instance only to get information about the layout of these files.

An ASM instance is like any other database instance except it contains two new background
processes. First one coordinates rebalance activity for disk groups and it is called RBAL. The

13
second one performs the actual rebalance activity for AU movements. At any given time
there can be many of these, and they are called ARB0, ARB1, and so on. An ASM instance
also has most of the same background processes as a ordinary database instance (SMON,
PMON, LGWR, and so on.)

Each database instance using ASM has two new background processes called ASMB and
RBAL. RABL performs global opens of the disks in the disk groups. At database instance
startup, ASMB connects as a foreground process into the ASM instance. All communication
between the database and ASM instances is performed via this bridge. This includes physical
file changes such as data file creation and deletion. Over this connection, periodic messages
are exchanged to update statistics and to verify that both instances are healthy

It is quite possible to cluster ASM instances and run them as RAC, using the existing Global
Cache Services (GCS) infrastructure. There is one ASM instance per node on a cluster.

Storage in Oracle Real Application Clusters

Storage for RAC databases must be shared. In other words, datafiles must reside in an
Automatic Storage Management (ASM) disk group, on a cluster file system, or on shared raw
devices. This must include space for an undo tablespace for each instance if you are using
automatic undo management. Additionally, for each instance you must create at least two
redo log files that reside on shared storage. Oracle recommends, you can use one shared
server parameter file (SPFILE) with instance-specific entries. Or you can use a local file
system to store client-side parameter files (PFILEs).

If you do not use ASM, if your platform does not support a cluster file system, or if you do
not want to use a cluster file system to store datafiles, then create additional raw devices as
described in your platform-specific Oracle Real Application Clusters installation and
configuration guide.

Automatic Storage Management in Real Application Clusters

ASM automatically optimizes storage to maximize performance by managing the storage

configuration across the disks. ASM does this by evenly distributing the storage load across
all of the available storage within your cluster database environment. ASM partitions your
total disk space requirements into uniformly sized units across all disks in a disk group. ASM
can also automatically mirror data to prevent data loss. Due to these added features, ASM
significantly reduces administrative overhead.

As in single-instance Oracle databases, To use ASM in RAC, select ASM as your storage
option when you create your database with the Database Configuration Assistant (DBCA).

Note:- using ASM in RAC does not require I/O tuning.

Automatic Storage Management Components in RAC

When you create your database, Oracle creates one ASM instance on each node in your RAC
environment if one does not already exist. Each ASM instance has either an SPFILE or PFILE
type parameter file.

14
The shared disk requirement is the only substantial difference between using ASM in a RAC
database compared to using it in a single-instance Oracle database. ASM automatically re-
balances the storage load after you add or delete a disk or disk group.

In a cluster, each ASM instance manages its node's metadata updates to the disk groups. In
addition, each ASM instance coordinates disk group metadata with other nodes in the
cluster. As in single-instance Oracle databases, you can use Enterprise Manager, DBCA,
SQL*Plus, and the Server Control Utility (SRVCTL) to administer disk groups for ASM in RAC

Automatic Storage Management

ASM automatically optimizes storage to maximize performance by rebalancing the storage

configuration across the disks that ASM manages. ASM spreads the storage load across all
available storage within your cluster database environment for optimal performance. ASM
partitions your total disk space into uniformly sized units across all disks in a disk group.

ASM functionality is controlled by an ASM instance.

The main components of ASM are disk groups, each of which comprise of several physical
disks that are controlled as a single unit. The physical disks are known as ASM disks, while
the files that reside on the disks are know as ASM files.

Cluster File system

What is a Cluster File system (CFS) ?

A cluster file system (CFS) is a file system that may be accessed (read and write) by all
members in a cluster at the same time. This implies that all members of a cluster have the
same view.

If your platform supports an Oracle certified cluster file system, you can store the files that
Real Application Clusters requires directly on the cluster file system.

A clustered file system is a file system which is simultaneously mounted on multiple

servers. There are several approaches to clustering, most of which do not employ a
clustered file system. While many computer clusters don't use clustered file systems, unless
servers are underpinned by a clustered file system the complexity of the underlying storage
environment increases as servers are added.

Distributed file system - the generic term for a client/server or "network" file system
where the data isn't locally attached to a host.

Global file system - this refers to the namespace, so that all files have the same name and
path name when viewed from all hosts. This obviously makes it easy to share data across
machines and users in different parts of the organization.

15
OCFS2 (Oracle Cluster File System 2) is a free, open source, general-purpose, extent-based
clustered file system which Oracle developed and contributed to the Linux community, and
accepted into Linux kernel 2.6.16.

OCFS2 provides an open source, enterprise-class alternative to proprietary cluster file

systems, and provides both high performance and high availability. OCFS2 provides local file
system semantics and it can be used with any application. Cluster-aware applications can
leverage parallel I/O for higher performance, and other applications can make use of the file
system to provide a fail-over setup to increase availability.

Cluster file system - a distributed file system that is not a single server with a set of
clients, but instead a cluster of servers that all work together to provide high performance
service to their clients. To the clients the cluster is transparent - it is just "the file system",
but the file system software deals with distributing requests to elements of the storage
cluster.

Shared-disk cluster file system

The most common type of clustered file system is the shared disk file system, in which two
or more servers are connected to a single shared storage subsystem, such as a stand-alone
RAID array or SAN.

Symmetric file system - A symmetric file system is one in which the clients also run the
metadata manager code; that is, all nodes understand the disk structures.

Asymmetric file system - An asymmetric file system is one in which there are one or more
dedicated metadata managers that maintain the file system and its associated disk
structures.

Shared-nothing clustered file system

Another clustered file system approach is to have each node use its own local storage, and
communicate data changes to other nodes via some network or bus. In this case disks are
not shared amongst nodes, but are instead dedicated to a single node and made readable
and writable to other servers

Parallel file system - file systems with support for parallel applications, all nodes may be
accessing the same files at the same time, concurrently reading and writing. Data for a
single file is striped across multiple storage nodes to provide scalable performance to
individual files.

SAN file system - These provide a way for hosts to share Fibre Channel storage, which is
traditionally carved into private chunks bound to different hosts. To provide sharing, a block-
level metadata manager controls access to different SAN devices. A SAN File system mounts
storage natively in only one node, but connects all nodes to that storage and distributes
block addresses to other nodes. Scalability is often an issue because blocks are a low-level
way to share data placing a big burden on the metadata managers and requiring large
network transactions in order to access data.

16
Want to know more about Real Application Cluster Click Oracle Real Application Cluster

The Practical Handbook on Oracle Real Application Cluster for Oracle

DBA’s.

The “Oracle Real Application Cluster Field DBA Admin Handbook” is a practical guide
for Oracle DBA’s who manage and administer the Oracle Real Application Cluster High
availability solution.

This book provides practical steps to administer Oracle Real Application Cluster (RAC). It’s
like a field guide rather than an in depth manual. It’s written for the DBA’s to handle the day
to day challenges faced by DBA’s managing RAC on the job.

Oracle RAC Background Processes

The Background Processes

Figure. 4.6 shows various background processes (as listed following the figure) that
are spawned to execute the database processing.

Figure 4.6 The Background Processes

SMON - System Monitor process recovers after instance failure and monitors
temporary segments and extents. SMON in a non-failed instance can also perform
failed instance recovery for other failed RAC instance.

PMON - Process Monitor process recovers failed process resources. If MTS (also
called Shared Server Architecture) is being utilized. PMON monitors and restarts any
failed dispatcher or server processes. In RAC, PMON’s role as service registration
agent is particularly important.

DBWR - Database Writer or Dirty Buffer Writer process is responsible for writing
dirty buffers from the database block cache to the database data files. Generally,
DBWR only writes blocks back to the data files on commit, or when the cache is full
and space has to be made for more blocks. The possible multiple DBWR processes in
RAC must be coordinated through the locking and global cache processes to ensure
efficient processing is accomplished.

17
LGWR - Log Writer process is responsible for writing the log buffers out to the redo
logs. I n RAC, each RAC instance has its own LGWR process that maintains that
instance’s thread of redo logs.

ARCH - (Optional) Archive process writes filled redo logs to the archive log
location(s). In RAC, the various ARCH processes can be utilized to ensure that copies
of the archived redo logs for each instance are available to the other instances in the
RAC setup should they be needed for recovery.

CKPT - Checkpoint process writes checkpoint information to control files and data
file headers.

Pnnn - (Optional) Parallel Query Slaves are started and stopped as needed to
participate in parallel query operations.

CQJ0 - Job queue controller process wakes up periodically and checks the job log. If
a job is due, it spawns Jnnn processes to handle jobs.

Jnnn - (Optional) Job processes used by the Oracle9i job queues to process internal
Oracle9i jobs. The CQJ0 process controls it automatically.

QMN - (Optional) Advanced Queuing process is used to control the advanced

queuing jobs.

Snnn - (Optional) Pre-spawned shared server processes are used by the multi-
threaded server (MTS) process to handle connection requests from users, and act as
connection pools for user processes. These user processes also handle disk reads from
database datafiles into the database block buffers.

Dnnn - (Optional) Dispatcher process for shared server (MTS) - It accepts connection
requests and portions them out to the pre-spawned server processes.

MMON – This process performs various manageability-related background tasks, for

example:

 Issuing alerts whenever a given metrics violates its threshold value

 Capturing statistics value for SQL objects which have been recently modified

MMNL - This process performs frequent and lightweight manageability-related tasks,

such as session history capture and metrics computation.

18
MMAN - is used for internal database tasks that manage the automatic shared
memory. MMAN serves as the SGA Memory Broker and coordinates the sizing of the
memory components.

RBAL - This process coordinates rebalance activity for disk groups in an Automatic
Storage Management instance.

ORBn - performs the actual rebalance data extent movements in an Automatic

Storage Management instance. There can be many of these at a time, called ORB0,
ORB1, and so forth.

OSMB - is present in a database instance using an Automatic Storage Management

disk group. It communicates with the Automatic Storage Management instance.

FMON - The database communicates with the mapping libraries provided by storage
vendors through an external non-Oracle Database process that is spawned by a
background process called FMON. FMON is responsible for managing the mapping
information. When you specify the FILE_MAPPING initialization parameter for
mapping data files to physical devices on a storage subsystem, then the FMON
process is spawned.

LMON - The Global Enqueue Service Monitor (LMON) monitors the entire cluster to
manage the global enqueues and the resources. LMON manages instance and process
failures and the associated recovery for the Global Cache Service (GCS) and Global
Enqueue Service (GES). In particular, LMON handles the part of recovery associated
with global resources. LMON-provided services are also known as cluster group
services (CGS)

LMDx - The Global Enqueue Service Daemon (LMD) is the lock agent process that
manages enqueue manager service requests for Global Cache Service enqueues to
control access to global enqueues and resources. The LMD process also handles
deadlock detection and remote enqueue requests. Remote resource requests are the
requests originating from another instance.

RAC Specific Background Processes

LMSx - The Global Cache Service Processes (LMSx) are the processes that handle
remote Global Cache Service (GCS) messages. Real Application Clusters software
provides for up to 10 Global Cache Service Processes. The number of LMSx varies
depending on the amount of messaging traffic among nodes in the cluster.

19
The LMSx handles the acquisition interrupt and blocking interrupt requests from the
remote instances for Global Cache Service resources. For cross-instance consistent
read requests, the LMSx will create a consistent read version of the block and send it
to the requesting instance. The LMSx also controls the flow of messages to remote
instances.

The LMSn processes handle the blocking interrupts from the remote instance for the
Global Cache Service resources by:

 Managing the resource requests and cross-instance call operations for the shared
resources.

 Building a list of invalid lock elements and validating the lock elements during recovery.

 Handling the global lock deadlock detection and Monitoring for the lock conversion
timeouts

RAC Specific Processes

LCKx - This process manages the global enqueue requests and the cross-instance
broadcast. Workload is automatically shared and balanced when there are multiple
Global Cache Service Processes (LMSx).

DIAG: Diagnosability Daemon – Monitors the health of the instance and captures the
data for instance process failures.

The following shows typical background processes of the RAC instance named
NYDB1.
$ rac-1a:NYDB1:/app/home/oracle >ps -ef | grep ora_

All nodes in an Oracle RAC environment must connect to a Local Area Network (LAN) to
enable users and applications to access the database. Applications should use the Oracle
Database services feature to connect to an Oracle database. Services enable you to define
rules and characteristics to control how users and applications connect to database
instances. These characteristics include a unique name, workload balancing and failover
options, and high availability characteristics. Oracle Net Services enables the load balancing
of application connections across all of the instances in an Oracle RAC database.

Users can access an Oracle RAC database using a client-server configuration or through one
or more middle tiers, with or without connection pooling. Users can be DBAs, developers,
application users, power users, such as data miners who create their own searches, and so
on.

20
 Most public networks typically use TCP/IP, but you can use any supported hardware and
software combination. Oracle RAC database instances can be accessed through a database's
defined, default IP address and through VIP addresses.

Note:
Do not to use the interconnect or the private network for user communication because Cache
Fusion uses the private interconnect for inter-instance communications.

In addition to the node's host name and IP address, you must also assign a virtual host name
and an IP address to each node. The virtual host name or VIP should be used to connect to
the database instance. For example, you might enter the virtual host name CRM in the
address list of thetnsnames.ora file.

A virtual IP address is an alternate public address that client connections use instead of the
standard public IP address. To configure VIP addresses, you need to reserve a spare IP
address for each node that uses the same subnet as the public network.

If a node fails, then the node's VIP fails over to another node on which the VIP cannot accept
connections. Generally, VIPs fail over when the node on which a VIP runs fails or if all
interfaces for the VIP fail or are disconnected from the network. Clients that attempt to
connect to the VIP receive a rapidconnection refused error instead of waiting for TCP
connect timeout messages. You configure VIP addresses in the address list for your
database connection definition to enable connectivity. The following section describes the
Oracle RAC software components in more detail.

The Oracle Real Application Clusters Software

Components
Oracle RAC databases have two or more database instances that each contain memory
structures and background processes. An Oracle RAC database has the same processes and
memory structures as a single-instance Oracle database as well as additional process
and memory structures that are specific to Oracle RAC. Any one instance's database view is
nearly identical to any other instance's view within the same Oracle RAC database; the view
is a single system image of the environment.

Each instance has a buffer cache in its System Global Area (SGA). Using Cache Fusion,
Oracle RAC environments logically combine each instance'sbuffer cache to enable the
instances to process data as if the data resided on a logically combined, single cache.

Note:

The SGA size requirements for Oracle RAC are greater than the SGA requirements for single-
instance Oracle databases due to Cache Fusion.

21
o ensure that each Oracle RAC database instance obtains the block that it needs to satisfy a
query or transaction, Oracle RAC instances use two processes, the Global Cache Service
(GCS) and the Global Enqueue Service (GES). The GCS and GES maintain records of the
statuses of each data file and each cached block using a Global Resource Directory (GRD).
The GRD contents are distributed across all of the active instances, which effectively
increases the size of the System Global Area for an Oracle RAC instance.

After one instance caches data, any other instance within the same cluster database can
acquire a block image from another instance in the same database faster than by reading
the block from disk. Therefore, Cache Fusion moves current blocks between instances rather
than re-reading the blocks from disk. When a consistent block is needed or a changed block
is required on another instance, Cache Fusion transfers the block image directly between
the affected instances. Oracle RAC uses the private interconnect for inter-instance
communication and block transfers. The Global Enqueue Service Monitor and the Instance
Enqueue Process manages access to Cache Fusion resources as well as enqueue recovery
processing.

22