VirtualBox Setup

Perform either the Automatic Setup

VirtualBox Network Setup

We need to make sure a host-only network is configured and check/modify the IP range for that network. This will be the public network for our
RAC installation.

 Start VirtualBox from the menu.

 Select the "File > Preferences" menu option.

 Click "Network" in the left pane and click the "Host-only Networks" tab.

 Click the "Adds new host-only network" button on the right size of the screen. Depending on the host OS, a network called "vboxnet0" or
"VirtualBox Host-Only Ethernet Adapter" will be created.

Click the "Edits selected host-only network." button on the right size of the screen.

 If you want to use a different subnet for your public addresses, you can change the network details here. Just make sure the subnet you
choose doesn't match any real subnets on your network. I've decided to stick with the default, which for me is "192.168.56.X".
  Use the "OK" buttons to exit out of this screen and the previous one.

Virtual Machine Setup

It is probably worth
Network Configuration
It is probably worth
Guest Operating System Installation
We need to make sure
12cc
Oracle Installation Prerequisites
Perform either the Automatic Setup or the Manual Setup to complete the basic prerequisites. The Additional Setup is required for all installations.

Automatic Setup
It is probably worth doing a full update of yum repository, but this is not strictly speaking necessary.
# yum update -y

If you plan to use the "oracle-rdbms-server-12cR1-preinstall" package to perform all your prerequisite setup, issue the following command.
# yum install oracle-rdbms-server-12cR1-preinstall -y
# yum install ntp -y

 Earlier versions of Oracle Linux required manual setup of the Yum repository by following the instructions at http://public-yum.oracle.com.

Manual Setup
If you have not used the "oracle-rdbms-server-12cR1-preinstall" package to perform all prerequisites, you will need to manually perform the
following setup tasks.

Add the following lines to the "/etc/sysctl.conf" file, or in a file called "/etc/sysctl.d/98-oracle.conf".
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Run the following command to change the current kernel parameters.

 /sbin/sysctl -p

Add the following lines to a file called "/etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf" file.

oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle hard memlock 134217728
oracle soft memlock 134217728

In addition to the basic OS installation, the following packages must be installed whilst logged in as the root user. This includes the 64-bit and 32-bit
versions of some packages.
# From Public Yum or ULN

yum install binutils -y

yum install compat-libstdc++-33 -y
yum install compat-libstdc++-33.i686 -y
yum install gcc -y
yum install gcc-c++ -y
yum install glibc -y
yum install glibc.i686 -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y
yum install ksh -y
yum install libgcc -y
yum install libgcc.i686 -y
yum install libstdc++ -y
yum install libstdc++.i686 -y
yum install libstdc++-devel -y
yum install libstdc++-devel.i686 -y
yum install libaio -y
yum install libaio.i686 -y
yum install libaio-devel -y
yum install libaio-devel.i686 -y
yum install libXext -y
yum install libXext.i686 –y
 yum install libXtst -y
yum install libXtst.i686 -y
yum install libX11 -y
yum install libX11.i686 -y
yum install libXau -y
yum install libXau.i686 -y
yum install libxcb -y
yum install libxcb.i686 -y
yum install libXi -y
yum install libXi.i686 -y
yum install make -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel -y
yum install zlib-devel -y
yum install zlib-devel.i686 -y

Create the new groups and users.

groupadd -g 54321 oinstall
groupadd -g 54322 dba
groupadd -g 54323 oper
#groupadd -g 54324 backupdba

#groupadd -g 54325 dgdba

#groupadd -g 54326 kmdba
#groupadd -g 54327 asmdba
#groupadd -g 54328 asmoper
#groupadd -g 54329 asmadmin

useradd -u 54321 -g oinstall -G dba,oper oracle

Uncomment the extra groups you require.

Additional Setup
The following steps must be performed, whether you did the manual or automatic setup.

Perform the following steps whilst logged into the "ol7-121-rac1" virtual machine as the root user.

Set the password for the "oracle" user.

passwd oracle

Apart from the localhost address, the "/etc/hosts" file can be left blank, but I prefer to put the addresses in for reference.

# Public enp0s8
192.168.180.101 ol7-121-rac1.localdomain ol7-121-rac1
192.168.180.102 ol7-121-rac2.localdomain ol7-121-rac2

# Private enp0s9
192.168.190.101 ol7-121-rac1-priv.localdomain ol7-121-rac1-priv
192.168.190.102 ol7-121-rac2-priv.localdomain ol7-121-rac2-priv

# Virtual
192.168.180.103 ol7-121-rac1-vip.localdomain ol7-121-rac1-vip
192.168.180.104 ol7-121-rac2-vip.localdomain ol7-121-rac2-vip

# SCAN
192.168.180.105 ol7-121-scan.localdomain ol7-121-scan
192.168.180.106 ol7-121-scan.localdomain ol7-121-scan
192.168.180.107 ol7-121-scan.localdomain ol7-121-scan

The SCAN address is commented out of the hosts file because it must be resolved using a DNS, so it can round-robin between 3 addresses on the
same subnet as the public IPs. The DNS can be configured on the host machine using BIND or Dnsmasq, which is much simpler. If you are using
Dnsmasq, put the RAC-specific entries in the hosts machines "/etc/hosts" file, with the SCAN entries uncommented, and restart Dnsmasq.

Make sure the "/etc/resolv.conf" file includes a nameserver entry that points to the correct nameserver. Also, if the "domain" and "search" entries are
both present, comment out one of them. For this installation my "/etc/resolv.conf" looked like this.
#domain localdomain
search localdomain
 nameserver 192.168.180.1

The changes to the "resolv.conf" will be overwritten by the network manager, due to the presence of the NAT interface. For this reason, this interface
should now be disabled on startup. You can enable it manually if you need to access the internet from the VMs. Edit the "/etc/sysconfig/network-
scripts/ifcfg-enp0s3" (eth0) file, making the following change. This will take effect after the next restart.
ONBOOT=no

There is no need to do the restart now. You can just run the following command. Remember to amend the adapter name if yours are named
differently.
# ifdown enp0s3
# #ifdown eth0

At this point, the networking for the first node should look something like the following. Notice that enp0s3 (eth0) has no associated IP address
because it is disabled.
[root@ol7-121-rac1 ~]# ifconfig -a
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 08:00:27:71:2e:00 txqueuelen 1000 (Ethernet)
RX packets 622892 bytes 550205625 (524.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 259290 bytes 15588726 (14.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.180.101 netmask 255.255.255.0 broadcast 192.168.180.255
inet6 fe80::5a0a:b24b:4924:4f69 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:d4:dc:ce txqueuelen 1000 (Ethernet)
RX packets 4338 bytes 293374 (286.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5431 bytes 561330 (548.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.190.101 netmask 255.255.255.0 broadcast 192.168.190.255
inet6 fe80::d7ba:f0c0:2c44:2cf3 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:cd:ce:5b txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 29 bytes 4257 (4.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 40 bytes 2870 (2.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40 bytes 2870 (2.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:5f:55:00 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0-nic: flags=4098<BROADCAST,MULTICAST> mtu 1500

ether 52:54:00:5f:55:00 txqueuelen 500 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

With this in place and the DNS configured the SCAN address is being resolved to all three IP addresses.
# nslookup ol7-121-scan
Server: 192.168.56.1
Address: 192.168.56.1#53

Name: ol7-121-scan.localdomain
Address: 192.168.56.105
Name: ol7-121-scan.localdomain
Address: 192.168.56.106
Name: ol7-121-scan.localdomain
Address: 192.168.56.107

#
Amend the "/etc/security/limits.d/20-nproc.conf" file as described below. See MOS Note [ID 1487773.1]
# Change this
* soft nproc 4096

# To this
* - nproc 16384

Change the setting of SELinux to permissive by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follows.
SELINUX=permissive

If you have the Linux firewall enabled, you will need to disable or configure it, as shown here or here. The following is an example of disabling the
firewall.
# systemctl stop firewalld
# systemctl disable firewalld

Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the
RAC nodes. If you want to deconfigure NTP do the following, which is what I did for this installation.
# systemctl stop ntpd
Shutting down ntpd: [ OK ]
# systemctl disable ntpd
# mv /etc/ntp.conf /etc/ntp.conf.orig
# rm /var/run/ntpd.pid

If your RAC is going to be permanently connected to your main network and you want to use NTP, you must add the "-x" option into the following
line in the "/etc/sysconfig/ntpd" file.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

Then restart NTP.

# systemctl enable ntpd
 # systemctl restart ntpd

Create the directories in which the Oracle software will be installed.

mkdir -p /u01/app/12.1.0.2/grid
mkdir -p /u01/app/oracle/product/12.1.0.2/db_extranet
chown -R oracle:oinstall /u01
chmod -R 775 /u01/

Log in as the "oracle" user and add the following lines at the end of the "/home/oracle/.bash_profile" file.
# Oracle Settings
export TMP=/tmp
export TMPDIR=$TMP

export ORACLE_HOSTNAME=ol7-121-rac1.localdomain
export ORACLE_UNQNAME=CDBRAC
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/12.1.0.2/grid
export DB_HOME=$ORACLE_BASE/product/12.1.0.2/db_extranet
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=cdbrac1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

alias grid_env='. /home/oracle/grid_env'

alias db_env='. /home/oracle/db_env'

Create a file called "/home/oracle/grid_env" with the following contents.

export ORACLE_SID=+ASM1
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
Create a file called "/home/oracle/db_env" with the following contents.
export ORACLE_SID=cdbrac1
export ORACLE_HOME=$DB_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

Once the "/home/oracle/.bash_profile" has been run, you will be able to switch between environments as follows.
$ grid_env
$ echo $ORACLE_HOME
/u01/app/12.1.0.2/grid
$ db_env
$ echo $ORACLE_HOME
/u01/app/oracle/product/12.1.0.2/db_1
$

We've made a lot of changes, so it's worth doing a reboot of the VM at this point to make sure all the changes have taken effect.
# shutdown -r now
Install Guest Additions
Click on the "Devices > Install Guest Additions" menu option at the top of the VM screen. If you get the option to auto-run take it. If not, then run
the following commands.
cd /media/VBOXADDITIONS_5.0.20_106931
sh ./VBoxLinuxAdditions.run
Add the "oracle" user into the "vboxsf" group so it has access to shared drives.
# usermod -G vboxsf,dba,oinstall oracle
# usermod -g dba oracle

# id oracle
uid=54321(oracle) gid=54322(dba) groups=54322(dba),54321(oinstall),983(vboxsf)

# chown oracle:dba /u01 -R

Bidirectional clipboard and shared folders

Click
Set Machine’s Shared Memory
Now we will set shared memtory for the machine

vi /etc/fstab
# and add the following line
shmfs /dev/shm tmpfs size 12g 0 0

Restart the machine

Set the system to boot to the right kernel version

From terminal log in as root, check the kernel you have been loggin into and worked on, and then execute the following command

[root@ol7-121-rac1 ~]# uname -a

Linux ol7-121-rac1.localdomain 4.1.12-61.1.27.el7uek.x86_64 #2 SMP Fri Feb 3 12:31:56 PST 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@ol7-121-rac1 ~]# grep "submenu\|^\menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2

The output of this command should be similar to that:

[root@ol7-121-rac1 ~]# grep "submenu\|^\menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2

Oracle Linux Server (4.1.12-61.1.27.el7uek.x86_64 with Unbreakable Enterprise Kernel) 7.3
Oracle Linux Server (3.10.0-514.6.1.el7.x86_64 with Linux) 7.3
Oracle Linux Server (3.10.0-327.el7.x86_64 with Linux) 7.3
Oracle Linux Server (3.8.13-118.16.3.el7uek.x86_64 with Unbreakable Enterprise Kernel) 7.3
Oracle Linux Server (3.8.13-98.7.1.el7uek.x86_64 with Unbreakable Enterprise Kernel) 7.3
Oracle Linux Server (0-rescue-4924daa11c674f2ba813fda32a70ebce with Linux) 7.3

Mark with the mouse the line of the kernel version you want the system to boot to (make sure you’re choosing the right kernel version), and attach it by pasting
it between the quotes of the following command:
 grub2-set-default ""

The result should be similar to:

grub2-set-default "Oracle Linux Server (4.1.12-61.1.27.el7uek.x86_64 with Unbreakable Enterprise Kernel) 7.3"

Restart the machine and validate it boots to the right kernel.

Create Shared Disks
Shut down the "ol7-121-rac1" virtual machine using the following command.
# shutdown -h now

On the host server, create 4 sharable virtual disks and associate them as virtual media using the following commands. You can pick a different
location, but make sure they are outside the existing VM directory.
$ mkdir -p /u04/VirtualBox/ol7-121-rac
$ cd /u04/VirtualBox/ol7-121-rac

$
$ # Create the disks and associate them with VirtualBox as virtual media.
$ VBoxManage createhd --filename asm1.vdi --size 5120 --format VDI --variant Fixed
$ VBoxManage createhd --filename asm2.vdi --size 5120 --format VDI --variant Fixed
$ VBoxManage createhd --filename asm3.vdi --size 5120 --format VDI --variant Fixed
$ VBoxManage createhd --filename asm4.vdi --size 5120 --format VDI --variant Fixed
$
$ # Connect them to the VM.
$ VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
$ VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable
$ VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable
$ VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable
$
$ # Make shareable.
$ VBoxManage modifyhd asm1.vdi --type shareable
$ VBoxManage modifyhd asm2.vdi --type shareable
$ VBoxManage modifyhd asm3.vdi --type shareable
$ VBoxManage modifyhd asm4.vdi --type shareable

If you are using a Windows host, you will have to modify the paths, but the process is the same.
F:
cd "F:\~VirtualBoxVMs\Oracle12cR1RAC-001"

set PATH=%PATH%;C:\Program Files\Oracle\VirtualBox

 VBoxManage createhd --filename "asm1.vdi" --size 10240 --format VDI --variant Fixed
VBoxManage createhd --filename "asm2.vdi" --size 10240 --format VDI --variant Fixed
VBoxManage createhd --filename "asm3.vdi" --size 10240 --format VDI --variant Fixed
VBoxManage createhd --filename "asm4.vdi" --size 10240 --format VDI --variant Fixed

VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 1 --device 0 --type hdd --medium "asm1.vdi"
VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 2 --device 0 --type hdd --medium "asm2.vdi"
VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 3 --device 0 --type hdd --medium "asm3.vdi"
VBoxManage storageattach ol7-121-rac1 --storagectl "SATA" --port 4 --device 0 --type hdd --medium "asm4.vdi"

VBoxManage modifyhd asm1.vdi --type shareable

VBoxManage modifyhd asm2.vdi --type shareable
VBoxManage modifyhd asm3.vdi --type shareable
VBoxManage modifyhd asm4.vdi --type shareable

Start the "ol7-121-rac1" virtual machine by clicking the "Start" button on the toolbar. When the server has started, log in as the root user so you can
configure the shared disks. The current disks can be seen by issuing the following commands.
# cd /dev
# ls sd*
sda sda1 sda2 sdb sdc sdd sde
#

Use the "fdisk" command to partition the disks sdb to sde. The following output shows the expected fdisk output for the sdb disk.
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x62be91cf.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): n

 Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-652, default 652):
Using default value 652

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.
#

In each case, the sequence of answers is "n", "p", "1", "Return", "Return" and "w".

Once all the disks are partitioned, the results can be seen by repeating the previous "ls" command.
# cd /dev
# ls sd*
sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1 sde sde1
#

Configure your UDEV rules, as shown here.

Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist.
options=-g

The SCSI ID of my disks are displayed below.

[root@ol7-121-rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VB618a70ee-e639a49c
[root@ol7-121-rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdc
1ATA_VBOX_HARDDISK_VBd0761660-ade06f21
[root@ol7-121-rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdd
1ATA_VBOX_HARDDISK_VB06b8543a-a65c0699
[root@ol7-121-rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sde
1ATA_VBOX_HARDDISK_VBb7229166-917ab3ea
Using these values, edit the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file adding the following 4 entries. All parameters for a single entry must
be on the same line.
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",
RESULT=="1ATA_VBOX_HARDDISK_VB618a70ee-e639a49c", SYMLINK+="oracleasm/asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",
RESULT=="1ATA_VBOX_HARDDISK_VBd0761660-ade06f21", SYMLINK+="oracleasm/asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",
RESULT=="1ATA_VBOX_HARDDISK_VB06b8543a-a65c0699", SYMLINK+="oracleasm/asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",
RESULT=="1ATA_VBOX_HARDDISK_VBb7229166-917ab3ea", SYMLINK+="oracleasm/asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"

Load updated block device partition tables.

# /sbin/partprobe /dev/sdb1
# /sbin/partprobe /dev/sdc1
# /sbin/partprobe /dev/sdd1
# /sbin/partprobe /dev/sde1

Test the rules are working as expected.

# /sbin/udevadm test /block/sdb/sdb1

Reload the UDEV rules.

# /sbin/udevadm control --reload-rules

The disks should now be visible and have the correct ownership using the following command. If they are not visible, your UDEV configuration is
incorrect and must be fixed before you proceed.
[root@ol7-121-rac1 dev]# ls -al /dev/oracleasm/*
lrwxrwxrwx. 1 root root 7 Feb 20 19:04 /dev/oracleasm/asm-disk1 -> ../sdb1
lrwxrwxrwx. 1 root root 7 Feb 20 19:05 /dev/oracleasm/asm-disk2 -> ../sdc1
lrwxrwxrwx. 1 root root 7 Feb 20 19:05 /dev/oracleasm/asm-disk3 -> ../sdd1
 lrwxrwxrwx. 1 root root 7 Feb 20 19:05 /dev/oracleasm/asm-disk4 -> ../sde1

The shared disks are now configured for the grid infrastructure.

Clone the Virtual Machine

Later versions of VirtualBox allow you to clone VMs, but these also attempt to clone the shared disks, which is not what we want. Instead we must
manually clone the VM.

Shut down the "ol7-121-rac1" virtual machine using the following command.
# shutdown -h now

Manually clone the "ol7-121-rac1.vdi" disk using the following commands on the host server.
$ # Linux
$ mkdir -p /u03/VirtualBox/ol7-121-rac2
$ VBoxManage clonehd /u01/VirtualBox/ol7-121-rac1/ol7-121-rac1.vdi /u03/VirtualBox/ol7-121-rac2/ol7-121-rac2.vdi

Rem Windows

set PATH=%PATH%;C:\Program Files\Oracle\VirtualBox

VBoxManage clonehd "F:\~VirtualBoxVMs\Oracle12cR1RAC-001\ol7-121-rac1.vdi" "F:\~VirtualBoxVMs\Oracle12cR1RAC-001\ol7-121-

rac2.vdi"

Create the "ol7-121-rac2" virtual machine in VirtualBox in the same way as you did for "ol7-121-rac1", with the exception of using an existing "ol7-
121-rac2.vdi" virtual hard drive.
Remember to add the three network adaptor as you did on the "ol7-121-rac1" VM. When the VM is created, attach the shared disks to this VM.
$ # Linux : Switch to the shared storage location and attach them.
$ cd /u04/VirtualBox/ol7-121-rac
$
$ VBoxManage storageattach ol7-121-rac2 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
$ VBoxManage storageattach ol7-121-rac2 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable
$ VBoxManage storageattach ol7-121-rac2 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable
$ VBoxManage storageattach ol7-121-rac2 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable

Rem Windows : Switch to the shared storage location and attach them.
F:
cd "F:\~VirtualBoxVMs\Oracle12cR1RAC-001"
 "C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-121-rac2 --storagectl "SATA" --port 1 --device 0 --type hdd
--medium "asm1.vdi"
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-121-rac2 --storagectl "SATA" --port 2 --device 0 --type hdd
--medium "asm2.vdi"
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-121-rac2 --storagectl "SATA" --port 3 --device 0 --type hdd
--medium "asm3.vdi"
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ol7-121-rac2 --storagectl "SATA" --port 4 --device 0 --type hdd
--medium "asm4.vdi"

Start the "ol7-121-rac2" virtual machine by clicking the "Start" button on the toolbar. Ignore any network errors during the startup.

Log in to the "ol7-121-rac2" virtual machine as the "root" user so we can reconfigure the network settings to match the following.

 hostname: ol7-121-rac2.localdomain
 enp0s3 (eth0): DHCP (*Not* Connect Automatically)
 enp0s8 (eth1): IP=192.168.180.102, Subnet=255.255.255.0, Gateway=0.0.0.0 (Connect Automatically)
 enp0s9 (eth2): IP=192.168.190.102, Subnet=255.255.255.0, Gateway=0.0.0.0 (Connect Automatically)

Amend the hostname in the "/etc/hostname" file.

ol7-121-rac2.localdomain

Unlike previous Linux versions, we shouldn't have to edit the MAC address associated with the network adapters, but we will have to alter their IP
addresses.

Edit the "/etc/sysconfig/network-scripts/ifcfg-eth1" (enp0s8), amending only the IPADDR settings as follows and deleting the UUID entry.
####### just pasted here for reference just in case network fails after this change UUID=991ca8b0-1109-453b-84b5-a67dd4aea1a4
IPADDR=192.168.180.102

Edit the "/etc/sysconfig/network-scripts/ifcfg-eth2" (enp0s9), amending only the IPADDR settings as follows and deleting the UUID entry.
####### just pasted here for reference just in case network fails after this change UUID=f71acb1a-0bb5-4f16-baa8-fa997a9d0227
IPADDR=192.168.190.102

Restart the virtual machines.

# shutdown -r now
At this point, the networking for the second node should look something like the following. Notice that enp0s3 (eth0) has no associated IP address
because it is disabled.
# ifconfig
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 08:00:27:17:88:dc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.180.102 netmask 255.255.255.0 broadcast 192.168.180.255
inet6 fe80::40eb:87b2:9a6a:53b0 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:3f:d2:fd txqueuelen 1000 (Ethernet)
RX packets 85 bytes 11317 (11.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 98 bytes 27586 (26.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.190.102 netmask 255.255.255.0 broadcast 192.168.190.255
inet6 fe80::376d:bde7:eed2:ecb4 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:99:87:b3 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 29 bytes 4262 (4.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 212 bytes 17948 (17.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 212 bytes 17948 (17.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:c3:e2:c6 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
#

Edit the "/home/oracle/.bash_profile" file on the "ol7-121-rac2" node to correct the ORACLE_SID and ORACLE_HOSTNAME values.
export ORACLE_SID=cdbrac2
export ORACLE_HOSTNAME=ol7-121-rac2.localdomain

Also, amend the ORACLE_SID setting in the "/home/oracle/db_env" and "/home/oracle/grid_env" files.

Restart the "ol7-121-rac2" virtual machine and start the "ol7-121-rac1" virtual machine. When both nodes have started, check they can both ping all
the public and private IP addresses using the following commands.
ping -c 3 ol7-121-rac1
ping -c 3 ol7-121-rac1-priv
ping -c 3 ol7-121-rac2
ping -c 3 ol7-121-rac2-priv

Check the SCAN address is still being resolved properly on both nodes.
# nslookup ol7-121-scan
Server: 192.168.56.1
Address: 192.168.56.1#53

Name: ol7-121-scan.localdomain
Address: 192.168.56.105
Name: ol7-121-scan.localdomain
Address: 192.168.56.106
Name: ol7-121-scan.localdomain
Address: 192.168.56.107

#
At this point the virtual IP addresses defined in the "/etc/hosts" file will not work, so don't bother testing them.

Check the UDEV rules are working on both machines. In previous versions of OL7 the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file copied
between servers during the clone without any issues. For some reason, this doesn't seem to happen on my OL7.1 installations, so you may need to
repeat the UDEV configuration on the second node if the output of the following command is not consistent on both nodes.
# ls -al /dev/oracleasm/*
lrwxrwxrwx. 1 root root 7 Sep 18 08:19 /dev/oracleasm/asm-disk1 -> ../sdb1
lrwxrwxrwx. 1 root root 7 Sep 18 08:19 /dev/oracleasm/asm-disk2 -> ../sdc1
lrwxrwxrwx. 1 root root 7 Sep 18 08:19 /dev/oracleasm/asm-disk3 -> ../sdd1
lrwxrwxrwx. 1 root root 7 Sep 18 08:19 /dev/oracleasm/asm-disk4 -> ../sde1
#

Prior to 11gR2 we would probably use the "runcluvfy.sh" utility in the clusterware root directory to check the prerequisites have been met. If you are
intending to configure SSH connectivity using the installer this check should be omitted as it will always fail. If you want to setup SSH connectivity
manually, then once it is done you can run the "runcluvfy.sh" with the following command.
/mountpoint/clusterware/runcluvfy.sh stage -pre crsinst -n ol7-121-rac1,ol7-121-rac2 -verbose

If you get any failures be sure to correct them before proceeding.

The virtual machine setup is now complete.

Before moving forward, you should probably shut down your VMs and take snapshots of them. If any failures happen beyond this point it is
probably better to switch back to those snapshots, clean up the shared drives and start the grid installation again. An alternative to cleaning up the
shared disks is to back them up now using zip and just replace them in the event of a failure.
$ # Linux
$ cd /u04/VirtualBox/ol7-121-rac
$ zip PreGrid.zip *.vdi

Rem Windows
E:
cd “E:\VirtualBox_VM_HDDs\OEL72-12cR1RAC”
zip PreGrid.zip *.vdi
--WIP--

Install the Grid Infrastructure

Make sure both virtual machines are started. Install the following package from the Oracle grid media as the "root" user.
# cd /media/sf_12.1.0.2/grid/rpm
# rpm -Uvh cvuqdisk*

Login to "ol7-121-rac1" as the "oracle" user and start the Oracle installer.

$ cd /media/sf_12.1.0.2/grid

$ ./runInstaller

Select the "Install and Configure Oracle Grid Infrastructure for a Cluster" option, then click the "Next" button.
Accept the "Configure a Standard cluster" option by clicking the "Next" button.
Select the "Typical Installation" option, then click the "Next" button.
On the "Specify Cluster Configuration" screen, enter the correct SCAN Name and click the "Add" button.
Enter the details of the second node in the cluster, then click the "OK" button.
Click the "SSH Connectivity..." button and enter the password for the "oracle" user. Click the "Setup" button to configure SSH connectivity, and the
"Test" button to test it once it is complete. Once the test is complete, click the "Next" button.
If you are doing a 12.1.0.1 installation, you will have to click the "Identify network interfaces" button, but in 12.1.0.2 this is on the following screen.

Check the public and private networks are specified correctly. If the NAT interface is displayed, remember to mark it as "Do Not Use". Click the
"Next" button.
Enter "/u01/app/12.1.0.2/grid" as the software location and "Automatic Storage Manager" as the cluster registry storage type. Enter the ASM
password, select "dba" as the group and click the "Next" button.
Set the redundancy to "External", click the "Change Discovery Path" button and set the path to "/dev/oracleasm/*". Return the main screen and
select all 4 disks and click the "Next" button.
Accept the default inventory directory by clicking the "Next" button.
If you want the root scripts to run automatically, enter the relevant credentials. I prefer to run them manually. Click the "Next" button.
Wait while the prerequisite checks complete. If you have any issues use the "Fix & Check Again" button. Once possible fixes are complete, check the
"Ignore All" checkbox and click the "Next" button. It is likely the "Physical Memory" and "Device Checks for ASM" tests will fail for this type of
installation. This is OK.
If you are happy with the summary information, click the "Install" button.
Wait while the installation takes place.
When prompted, run the configuration scripts on each node.
The output from the "orainstRoot.sh" file should look something like that listed below.

# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

The output of the "root.sh" will vary a little depending on the node it is run on. Example output can be seen here (Node1, Node2).

Once the scripts have completed, return to the "Execute Configuration Scripts" screen on "ol7-121-rac1" and click the "OK" button.

Wait for the configuration assistants to complete.

If any of the configuration steps fail you should check the specified log to see if the error is a show-stopper or not. If you are not using a DNS to
resolve the SCAN you can expect the verification phase to fail with an error like the following.

INFO: Checking Single Client Access Name (SCAN)...

INFO: Checking name resolution setup for "rac-scan.localdomain"...

INFO: ERROR:

INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "rac-scan.localdomain"

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for "rac-scan.localdomain" (IP address: 192.168.2.201) failed

INFO: ERROR:

INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "rac-scan.localdomain"

INFO: Verification of SCAN VIP and Listener setup failed

Provided this is the only error, it is safe to ignore this and continue by clicking the "Next" button.

Click the "Close" button to exit the installer.

The grid infrastructure installation is now complete. We can check the status of the installation using the following commands.

$ grid_env
$ crsctl stat res -t

--------------------------------------------------------------------------------

Name Target State Server State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

ONLINE ONLINE ol7-121-rac1 STABLE

ONLINE ONLINE ol7-121-rac2 STABLE

ora.LISTENER.lsnr

ONLINE ONLINE ol7-121-rac1 STABLE

ONLINE ONLINE ol7-121-rac2 STABLE

ora.asm

ONLINE ONLINE ol7-121-rac1 Started,STABLE

ONLINE ONLINE ol7-121-rac2 Started,STABLE

ora.net1.network

ONLINE ONLINE ol7-121-rac1 STABLE

ONLINE ONLINE ol7-121-rac2 STABLE

ora.ons

ONLINE ONLINE ol7-121-rac1 STABLE

ONLINE ONLINE ol7-121-rac2 STABLE

--------------------------------------------------------------------------------
Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE ol7-121-rac2 STABLE

ora.LISTENER_SCAN2.lsnr

1 ONLINE ONLINE ol7-121-rac1 STABLE

ora.LISTENER_SCAN3.lsnr

1 ONLINE ONLINE ol7-121-rac1 STABLE

ora.MGMTLSNR

1 ONLINE ONLINE ol7-121-rac1 169.254.255.49 192.1

68.1.101,STABLE

ora.cvu

1 ONLINE ONLINE ol7-121-rac1 STABLE

ora.mgmtdb

1 ONLINE ONLINE ol7-121-rac1 Open,STABLE

ora.oc4j

1 ONLINE ONLINE ol7-121-rac1 STABLE

ora.ol7-121-rac1.vip

1 ONLINE ONLINE ol7-121-rac1 STABLE

ora.ol7-121-rac2.vip

1 ONLINE ONLINE ol7-121-rac2 STABLE

ora.scan1.vip

1 ONLINE ONLINE ol7-121-rac2 STABLE

ora.scan2.vip

1 ONLINE ONLINE ol7-121-rac1 STABLE

ora.scan3.vip

1 ONLINE ONLINE ol7-121-rac1 STABLE

--------------------------------------------------------------------------------

At this point it is probably a good idea to shutdown both VMs and take snapshots. Remember to make a fresh zip of the ASM disks on the host
machine, which you will need to restore if you revert to the post-grid snapshots.

$ cd /u04/VirtualBox/ol7-121-rac

$ zip PostGrid.zip *.vdi

Install the Database Software

Make sure the "ol7-121-rac1" and "ol7-121-rac2" virtual machines are started, then login to "ol7-121-rac1" as the oracle user and start the Oracle
installer. Check that all services are up using "crsctl stat res -t", as described before.

$ cd /media/sf_12.1.0.2/database

$ ./runInstaller

Uncheck the security updates checkbox and click the "Next" button and "Yes" on the subsequent warning dialog.
Select the "Install database software only" option, then click the "Next" button.
Accept the "Oracle Real Application Clusters database installation" option by clicking the "Next" button.
Make sure both nodes are selected, then click the "Next" button.
Select the required languages, then click the "Next" button.
Select the "Enterprise Edition" option, then click the "Next" button.
Enter "/u01/app/oracle" as the Oracle base and "/u01/app/oracle/product/12.1.0.2/db_1" as the software location, then click the "Next" button.
Select the desired operating system groups, then click the "Next" button.
Wait for the prerequisite check to complete. If there are any problems either click the "Fix & Check Again" button, or check the "Ignore All" checkbox
and click the "Next" button.
If you are happy with the summary information, click the "Install" button.
Wait while the installation takes place.
When prompted, run the configuration script on each node. When the scripts have been run on each node, click the "OK" button.
Click the "Close" button to exit the installer.
Shutdown both VMs and take snapshots. Remember to make a fresh zip of the ASM disks on the host machine, which you will need to restore if you
revert to the post-db snapshots.

$ cd /u04/VirtualBox/ol7-121-rac
$ zip PostDB.zip *.vdi

Create a Database
Make sure the "ol7-121-rac1" and "ol7-121-rac2" virtual machines are started, then login to "ol7-121-rac1" as the oracle user and start the Database
Creation Asistant (DBCA).

$ dbca

Select the "Create Database" option and click the "Next" button.
Select the "Create a database with default configuration" option. Enter the container database name (cdbrac), pluggable database name (pdb1) and
administrator password. Click the "Next" button.
Wait for the prerequisite checks to complete. If there are any problems either fix them, or check the "Ignore All" checkbox and click the "Next"
button.
If you are happy with the summary information, click the "Finish" button.
Wait while the database creation takes place.
If you want to modify passwords, click the "Password Management" button. When finished, click the "Close" button.
The RAC database creation is now complete.

Check the Status of the RAC

There are several ways to check the status of the RAC. The srvctl utility shows the current configuration and status of the RAC database.
$ srvctl config database -d cdbrac

Database unique name: cdbrac

Database name: cdbrac

Oracle home: /u01/app/oracle/product/12.1.0.2/db_1

Oracle user: oracle

Spfile: +DATA/CDBRAC/PARAMETERFILE/spfile.296.890818285

Password file: +DATA/CDBRAC/PASSWORD/pwdcdbrac.276.890817261

Domain:

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools:

Disk Groups: DATA

Mount point paths:

Services:

Type: RAC

Start concurrency:

Stop concurrency:

OSDBA group: dba

OSOPER group:

Database instances: cdbrac1,cdbrac2

Configured nodes: ol7-121-rac1,ol7-121-rac2

Database is administrator managed

$ srvctl status database -d cdbrac

Instance cdbrac1 is running on node ol7-121-rac1

Instance cdbrac2 is running on node ol7-121-rac2

The V$ACTIVE_INSTANCES view can also display the current status of the instances.

$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sat Sep 19 09:45:35 2015

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL> SELECT inst_name FROM v$active_instances;

INST_NAME

--------------------------------------------------------------------------------

ol7-121-rac1.localdomain:cdbrac1

ol7-121-rac2.localdomain:cdbrac2

SQL>