ORACLE RAC

ORACLE RAC

Definition

Oracle Real Application Cluster (RAC) is a cluster system at the

application level. It uses shared disk architecture that provides
scalability for all kind of applications. Applications without any
modifications can use the RAC database. (Somewhat)
 ORACLE RAC

Goal
Since the requests in a RAC cluster are spread evenly across the
RAC instances, and since all instances access the same shared
storage, addition of server(s) require no architecture changes
etc. And a failure of a single RAC node results only in the loss
of scalability and not in the loss of data since a single database
image is utilized.
 ORACLE RAC
Shared Disks
A requirement for Oracle Database 10g RAC cluster is a set of
servers with shared disk access and interconnect connectivity. Since
each instance in a RAC system must have access to the same
database files, a shared storage is required that can be accessed
from all RAC nodes concurrently.
 ORACLE RAC
Types of Shared Storage
The shared storage space can be used as raw devices, or by using a
cluster file system or ASM. This article will address Oracle's
Cluster File System OCFS and ASM. Note that Oracle 10g RAC
provides it's own locking mechanisms and therefore it does not rely
on other cluster software or on the operating system for handling
locks.
 ORACLE RAC

Upgrading the Kernel or Parameters

It is recommended to use newer Red Hat Enterprise Linux

kernels since newer kernels might fix known database
performance problems and other issues. However, you also
need to make sure that the OCFS and ASM drivers are
compatible with the kernel version!
 ORACLE RAC

Configuring the Network

Each RAC node should have at least one static IP address for
the public network and one static IP address for the private
cluster interconnect.
 ORACLE RAC

Private Networks

The private networks are critical components of a RAC cluster.

The private networks should only be used by Oracle to carry
Cluster Manager and Cache Fusion inter-node connection. A
RAC database does not require a separate private network, but
using a public network can degrade database performance (high
latency, low bandwidth). Therefore the private network should
have high-speed NICs (preferably one gigabit or more) and it
should only be used by Oracle.
 ORACLE RAC

Network Example

Here is an example how the /etc/hosts file could look like:

# Public hostnames for e.g. eth0 interfaces (public network)

216.88.5.1 rac1pub.puschitz.com rac1pub # RAC node 1

216.88.5.2 rac2pub.puschitz.com rac2pub # RAC node 2
216.88.5.3 rac3pub.puschitz.com rac3pub # RAC node 3
# Private hostnames, private network for e.g. eth1 interfaces
192.168.2.1 rac1prv.puschitz.com rac1prv # RAC node 1
192.168.2.2 rac2prv.puschitz.com rac2prv # RAC node 2
192.168.2.3 rac3prv.puschitz.com rac3prv # RAC node 3
 ORACLE RAC

Configuring Shared Storage

fdisk -l

Disk /dev/sda: 255 heads, 63 sectors, 30515 cylinders

Units = cylinders of 16065 * 512 bytes

Device Boot Start End Blocks Id System

/dev/sda1 1 6375 51207156 83 Linux
/dev/sda2 6376 12750 51207187+ 83 Linux
 ORACLE RAC
Creating Oracle User
If you use OCFS, it is important that the UID of "oracle" and GID
of "oinstall" are the same across all RAC nodes. Otherwise the
Oracle files on the OCFS filesystems on some nodes could either be
"unowned", or they could even be owned by another user account.
In my setup the UID and GID of oracle:dba is 700:700.
su - root
groupadd -g 700 dba # group of users to be granted with
SYSDBA system privilege
groupadd -g 701 oinstall # group owner of Oracle files
useradd -c "Oracle software owner" -u 700 -g oinstall -G dba oracle
passwd oracle
 ORACLE RAC
Setting Oracle Accounts
Since the Oracle Universal Installer (OUI) "runInstaller" is
executed from the oracle account, some environment variables
must be configured for the oracle account before OUI is started.

If you use bash which is the default shell on Red Hat Linux (to
verify your shell run: echo $SHELL), execute the following
commands:
# Oracle Environment
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=orcl1 # Each RAC node must have a
unique Oracle SID! E.g. orcl1, orcl2,...
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
 ORACLE RAC
Cluster File System

At the time of this writing, OCFS only supports Oracle Datafiles

and a few other files. Therefore OCFS should not be used for
Shared Oracle Home installs. See Installing and Configuring Oracle
Cluster File Systems (OCFS) for more information.
 ORACLE RAC
Creating Oracle Directories

If you want to comply with Oracle's Optimal Flexible Architecture

(OFA), then you don't want to place the database files in the /u01
directory but in another directory like /u02. This is not a
requirement but if you want to comply with OFA, then you might
want to create the following directories as well:
su - root
mkdir -p /u02/oradata/orcl
chown -R oracle.oinstall /u02
 ORACLE RAC
Creating Partitions
Note that it is important for the Redo Log files to be on the shared
disks as well.

To partition the disks, you can use the fdisk utility:

su - root
fdisk <device_name>
Here is an example how to create a new 50 GB partition on
drive /dev/sda:
su - root
# fdisk /dev/sda
 ORACLE RAC
Installing and Configuring OCFS
The Oracle Cluster File System (OCFS) was developed by Oracle
to overcome the limits of Raw Devices and Partitions. It also eases
administration of database files because it looks and feels just like a
regular file system.

OCFS only supports Oracle Datafiles and a few other files:

- Redo Log files
- Archive log files
- Control files
- Database datafiles
- Shared quorum disk file for the cluster manager
- Shared init file (srv)
 ORACLE RAC
Creating OCS File Systems

To create the OCFS filesystems, you can use the

ocfstool:
su - root
ocfstool
- Select "Task" - Select "Format"

Alternatively, you can execute the "mkfs.ocfs" command

to create the OCFS filesystems:
su - root
mkfs.ocfs -F -b 128 -L /u02/oradata/orcl -m
/u02/oradata/orcl \
-u `id -u oracle` -g `id -g oracle` -p 0775 <device_name>
 ORACLE RAC
Having File System a Boot Time

Add lines to the /etc/fstab file similar to the following

example:
/dev/sda1 /u02/oradata/orcl ocfs _netdev 0 0
The "_netdev" option prevents the OCFS filesystem from being
mounted until the network has first been enabled on the system.
To make sure the ocfs.o kernel module is loaded and the OCFS
file systems are mounted during the boot process, enter:
su - root
# chkconfig --list ocfs
ocfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
 ORACLE RAC
Configue Hangcheck Timer in Kernel

Oracle uses the Linux kernel module hangcheck-timer to

monitor the system health of the cluster and to reset a RAC node in
case of failures. The hangcheck-timer module uses a kernel-based
timer to periodically check the system task scheduler. This timer
resets the node when the system hangs or pauses. This module uses
the Time Stamp Counter (TSC) CPU register which is a counter that
is incremented at each clock signal.

The hangcheck-timer module comes now with the kernel:

find /lib/modules -name "hangcheck-timer.o"
 ORACLE RAC
Remote Access for RAC Nodes

When you run the Oracle Installer on a RAC node, it will use ssh
to copy Oracle software and data to other RAC nodes. Therefore,
the oracle user on the RAC node where Oracle Installer is
launched must be able to login to other RAC nodes without having
to provide a password or passphrase.

The following procedure shows how ssh can be configured that no

password is requested for oracle ssh logins.

To create an authentication key for oracle, enter the following

command on all RAC node:
 ORACLE RAC
More System Setup

Sizing Swap Space

Setting Shared Memory
Checking /tmp Space
Setting Semaphores
Setting File Handles
 ORACLE RAC
Installing Cluster Ready Services

Cluster Ready Services (CRS) contains cluster and database

configuration information for RAC, and it provides many system
management features. CRS accepts registration of Oracle instances
to the cluster and it sends ping messages to other RAC nodes. If the
heartbeat fails, CRS will use shared disk to distinguish between a
node failure and a network failure.

Once CRS is running on all RAC nodes, OUI will automatically

recognice all nodes on the cluster. This means that you can run OUI
on one RAC node to install the Oracle software on all other RAC
nodes.
 ORACLE RAC
Starting The Installer

Make sure that the oracle user can ssh to all RAC nodes without
ssh asking for a passphrase. This is very important because
otherwise OUI won't be able to install the Oracle software on other
RAC nodes. The following example shows how ssh-agent can
do the authentication for you when the oracle account logs in to
other RAC nodes using ssh.

Open a new terminal for the RAC node where you will execute
runInstaller and use this terminal to login from your desktop
using the following command:
$ ssh -X oracle@rac?pub
 ORACLE RAC
Configure ssh-agent

Now configure ssh-agent to handle the authentication for the

oracle account:
oracle$ ssh-agent $SHELL
oracle$ ssh-add
Enter passphrase for /home/oracle/.ssh/id_dsa: Enter your
passphrase
Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)
oracle$
 ORACLE RAC
Install Cluster Ready Services
In order to install the Cluster Ready Services (CRS) R1 (10.1.0.2)
on all RAC nodes, OUI has to be launched on only one RAC node.
To install CRS, insert the "Cluster Ready Services (CRS) R1
(10.1.0.2)" CD (downloadedd image name:
"ship.crs.cpio.gz"), and mount it on e.g. rac1pub:
su - root
mount /mnt/cdrom
Use the oracle terminal that you prepared for ssh at Automating
Authentication for oracle ssh Logins and execute runInstaller:
oracle$ /mnt/cdrom/runInstaller
 ORACLE RAC
Installing ORACLE with RAC
The following procedure shows the installation of the software for
Oracle Database 10g Software R1 (10.1.0.2) with Real
Application Clusters (RAC).

Note that Oracle Database 10g R1 (10.1) OUI will not be able to
discover disks that are marked as Linux ASMLib. Therefore it is
recommended to complete the software installation and then to
use dbca to create the database, see
http://otn.oracle.com/tech/linux/asmlib/install.html#10gr1 for
more information.
 ORACLE RAC
Setting Up the Database with dbca
The following steps show how to use dbca to create the database
and its instances. Oracle recommends to use dbca to create the
RAC database since the preconfigured databases are optimized for
ASM, server parameter file, and automatic undo management.
dbca also makes it much more easier to create new ASM disk
groups etc.
 ORACLE RAC
DBCA concluded (creating database)
Database File Locations:
Select "Use Oracle-Managed Files"
Database Area: +ORCL_DATA1
Click Next
I selected TAF Policy "Basic".
Click Next
- Initialization Parameters:
Change settings as needed.
Click Next
- Database Storage: Change settings as needed.
Click Next
- Creation Options: Check "Create Database"
Click Finish
 ORACLE RAC
Summary
RAC requires a CFS
RAC is complex
RAC provides transactions (OLTP)