Oracle RAC Install

Oracle RAC Install
Software
Oracle Database 12c Real Application Clusters 12.1.0.2.0
Oracle Database Grid Infrastructure 12.1.0.2.0 for Linux x86-64

V46096-01_1of2.zip Oracle Database Grid Infrastructure 12.1.0.2.0 for Linux x86-64, 1.6 GB
V46096-01_2of2.zip Oracle Database Grid Infrastructure 12.1.0.2.0 for Linux x86-64, 617.0 MB
Pre-install
1. Check rpms
2. Disable ip v6 - /sbin/ip -6 addr See this
3. nslookup not working - /etc/resolv.conf commented?
4. Check exec on /tmp
5. Check umask 022 ( OEL 7 defaults to 027)
6. HugePages – 60% of 384GB = 226GB or 115712 ~ 115720
vi /etc/security/limits.conf
# memlock should be 90% of memory when hugepages are enabled
@oinstall soft memlock 355781836
@oinstall hard memlock 355781836
vi /etc/sysctl.conf
vm.nr_hugepages=115720
sysctl -p
7. tuned-adm active  Current active profile: throughput-performance
8. transparent Hugepages
appears disabled in /etc/default/grub.conf, but
cat /sys/kernel/mm/transparent_hugepage/defrag
always defer [madvise] never
cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
9. Check NFS mount options
10. Set Timezone for all users - export TZ=/usr/share/zoneinfo/CET
/etc/hosts
# Public Hostnames
10.254.30.188 ebsdbi01ptmp.wincor-nixdorf.com ebsdbi01ptmp ebsdbi01p.wincor-nixdorf.com
ebsdbi01p ebsdbi05p.wincor-nixdorf.com ebsdbi05p ebsdbi06p.wincor-nixdorf.com ebsdbi06p
ebsdbi02p
ebsdbi03p
ebsdbi04p
# Virtual Hostnames
10.254.30.189 ebsdbi01ptmp-virt.wincor-nixdorf.com ebsdbi01ptmp-virt ebsdbi01p-virt.wincor-
nixdorf.com ebsdbi01p-virt ebsdbi05p-virt.wincor-nixdorf.com ebsdbi05p-virt ebsdbi06p-
virt.wincor-nixdorf.com ebsdbi06p-virt
nixdorf.com ebsdbi02p-virt
# Private Hostnames
192.168.100.8 ebsdbi01ptmp-priv1.wincor-nixdorf.com ebsdbi01ptmp-priv1
#SCAN IPs
10.254.30.190 ebsdbiptmp-scan.wincor-nixdorf.com ebsdbiptmp-scan ebsdbip-scan.wincor-
nixdorf.com ebsdbip-scan
Server Layout
Locatio Memor FI A/B FEX A/B Temp Host Temp Virtual Temp Virtual Temp SCAN Temp SCAN
n Usage CPU Model Cores y Serial# Port Port OS Server Name Server IP Netmask Name Temp IP Virtual Host Virtual IP Name IP Private Host Private IP SCAN Name SCAN Ips Name Ips
DC3 CRM Production Database Servers E5-2667 v4 16 384 FCH2143V1W9 1/11 N/A OEL 7.4 ebsdbi01p 10.254.30.132 ebsdbi01ptmp 10.254.30.188 ebsdbi01p-virt 10.254.30.133 ebsdbi01ptmp-virt 10.254.30.189
10.254.30.134
DC3 CRM Production Database Servers E5-2667 v4 16 384 FCH2143V1V4 1/12 N/A OEL 7.4 ebsdbi02p 10.254.30.135 ebsdbi02ptmp 10.254.30.191 ebsdbi02p-virt 10.254.30.136 ebsdbi02ptmp-virt 10.254.30.192
10.254.30.137 10.254.30.190
DC3 CRM Production Database Servers E5-2667 v4 16 384 FCH2143V1V5 1/13 N/A OEL 7.4 ebsdbi03p 10.254.30.138 ebsdbi03ptmp 10.254.30.194 ebsdbi03p-virt 10.254.30.139 ebsdbi03ptmp-virt 10.254.30.195 10.254.30.140 10.254.30.193
DC3 CRM Production Database Servers E5-2667 v4 16 384 FCH2143V1W8 1/14 N/A OEL 7.4 ebsdbi04p 10.254.30.141 ebsdbi04ptmp 10.254.30.197 ebsdbi04p-virt 10.254.30.142 ebsdbi04ptmp-virt 10.254.30.198 ebsdbip-scan 10.254.30.143 ebsdbiptmp-scan 10.254.30.196
Storage
See “EBSP Filesystem Layout on Fibre.docx”
Pre-requisites
Confirm BIOS settings
VT enabled
Directed IO
RAS Memory – Maximum performance
Jumbo frames
TCP Settings
The following values can be set on the virtual machines as well as dom0. Edit /etc/sysctl.conf for the
below mentioned changes. These can be changed both on virtual machines as well as dom0. (See, Oracle
VM 3: 10GbE Network Performance Tuning whitepaper
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.netdev_max_backlog = 300000
net.ipv4.tcp_moderate_rcvbuf =1
Turn off ip v6
ASM Disk Configuration

Issue#1
cd /etc/sysconfig
ls -l oracleasm*
It should be
oracleasm -> oracleasm-_dev_oracleasm
However there were 2 files.
Fix it manually:
cd /etc/sysconfig
rm -f oracleasm
ln -s oracleasm-_dev_oracleasm oracleasm
Issue#2
These are 4k Disks in Emulation mode
cat /sys/block/sdb/queue/physical_block_size
4096
cat /sys/block/sdb/queue/logical_block_size
512
Make sure that ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true, or else the asm diskgroups will be
created with 4096 sector size instead of 512. This can be done with:
/usr/sbin/oracleasm configure -b
oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracrs

Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
/usr/sbin/oracleasm configure -b
cd /etc/sysconfig/
vi oracleasm
Confirm the following:
ORACLEASM_SCANORDER="dm-"
ORACLEASM_SCANEXCLUDE="sd"
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true
Check above on all nodes
# Partition disks to align

parted -s -a optimal /dev/mapper/arch01 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/arch02 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/data01 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/fra01 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/fra02 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/ocrvote01 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/redo01 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/redo02 mklabel gpt mkpart primary 0% 100%
parted -s -a optimal /dev/mapper/temp01 mklabel gpt mkpart primary 0% 100%
On All nodes
kpartx -a /dev/mapper/arch01
kpartx -a /dev/mapper/arch02
kpartx -a /dev/mapper/data01
kpartx -a /dev/mapper/fra01
kpartx -a /dev/mapper/fra02
kpartx -a /dev/mapper/ocrvote01
kpartx -a /dev/mapper/redo01
kpartx -a /dev/mapper/redo02
kpartx -a /dev/mapper/temp01
Create ASMLib Disks

oracleasm createdisk EBSP_ARCH01 /dev/mapper/arch01p1
oracleasm createdisk EBSP_ARCH02 /dev/mapper/arch02p1
oracleasm createdisk EBSP_DATA01 /dev/mapper/data01p1
oracleasm createdisk EBSP_FRA01 /dev/mapper/fra01p1
oracleasm createdisk EBSP_FRA02 /dev/mapper/fra02p1
oracleasm createdisk EBSP_OCR_VOTE01 /dev/mapper/ocrvote01p1
oracleasm createdisk EBSP_REDO01 /dev/mapper/redo01p1
oracleasm createdisk EBSP_REDO02 /dev/mapper/redo02p1
oracleasm createdisk EBSP_TEMP01 /dev/mapper/temp01p1
oracleasm exit
oracleasm init
oracleasm scandisks
oracleasm listdisks
oracleasm listdisks | xargs oracleasm querydisk -d
Confirm the disks are using dm- and not sd devices

oracleasm querydisk -d EBSP_ARCH01
Disk "EBSP_ARCH01" is a valid ASM disk on device [249,31]
ls -l /dev/|grep 249|grep 31
brw-rw----. 1 root disk 249, 31 Mar 12 18:30 dm-31
ls -l /dev/mapper/|grep dm-31
lrwxrwxrwx. 1 root root 8 Mar 13 15:58 arch01p1 -> ../dm-31
Grid Install
CRS_HOME = /oracle/app/oracrs/oracrsdb/grid
ORACLE_BASE = /oracle/app/oracrs/base
Unix id = uid=501(oracrs) gid=501(oinstall) groups=501(oinstall),500(dba)
set ssh manually - https://docs.oracle.com/database/121/CWLIN/manpreins.htm#CWLIN515
cd /oracle/app/oracrs/software/grid
./runInstaller
Install and Configure Oracle Grid Infrastructure for a Cluster

Configure a Standard Cluster
Advanced Installation
English
Cluster Name: ebsp-dc3 (ebsp-dc4 in DR)
SCAN Name: ebsdbip-scan.wincor-nixdorf.com
SCAN Port: 1521
Public Hostname Virtual Hostname
ebsdbi01p.wincor-nixdorf.com ebsdbi01p-virt.wincor-nixdorf.com
DR ( Public hostname is different, but virtual hostname is same)

Public Hostname Virtual Hostname
ebsdbi01dr.wincor-nixdorf.com ebsdbi01p-virt.wincor-nixdorf.com
Check SSH Connectivity

Test
Next
Interface Name Subnet Use for
enp9s0 192.168.100.0 Private
enp10s0 192.168.100.0 Private
enp11s0 10.254.150.0 Do Not Use
bond0:520 10.254.30.128 Public
bond0:1905 10.173.124.0 Do Not Use
Use Standard ASM for storage

Disk group Name: EBSP_GRID_DG
Redundancy:External
AU Size: 1MB
ORCL: EBSP_OCR_VOTE01
Next
Use Same Password
Next
Do not use IPMI
Do not Register with OEM
Next
Oracle ASM Adminsitrator(OSASM) Group: oinstall
Oracle ASM DBA(OSDBA for ASM) Group: oinstall
Oracle ASM Operator( OSOPER for ASM) Group (Optional):
Ignore warning on OSDBA and OSASM are same group
Oracle base = /oracle/app/oracrs/base
Software location= /oracle/app/oracrs/oracrsdb/grid ( Keeping the existing directory name)
Inventory: /oracle/app/oracrs/oraInventory
Step 16: Do not check Automatically run configuration scripts
Pre-requisite checks:
NTP – Port not open. Waiting for F/W rule
Resolv.conf integrity – Port not open. Waiting for F/W rule
DNS/NIS name service – Port not open. Waiting for F/W rule
/dev/shm mounted as temporary filesystem – This is a bug and can be ignored
Ignore
Save Response File
Install
/oracle/app/oracrs/oraInventory/orainstRoot.sh
/oracle/app/oracrs/oracrsdb/grid/root.sh
Install
root.sh
[root@ebsdbi01p ~]# /oracle/app/oracrs/oracrsdb/grid/root.sh
Performing root user operation.
The following environment variables are set as:

ORACLE_OWNER= oracrs
ORACLE_HOME= /oracle/app/oracrs/oracrsdb/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/oracrs/oracrsdb/grid/crs/install/crsconfig_params
2018/03/20 23:33:10 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2018/03/20 23:33:31 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2018/03/20 23:33:32 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful

root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2018/03/20 23:34:04 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'ebsdbi01p'
CRS-2672: Attempting to start 'ora.mdnsd' on 'ebsdbi01p'
CRS-2676: Start of 'ora.mdnsd' on 'ebsdbi01p' succeeded
CRS-2676: Start of 'ora.evmd' on 'ebsdbi01p' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ebsdbi01p'
CRS-2676: Start of 'ora.gpnpd' on 'ebsdbi01p' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ebsdbi01p'
CRS-2672: Attempting to start 'ora.gipcd' on 'ebsdbi01p'
CRS-2676: Start of 'ora.cssdmonitor' on 'ebsdbi01p' succeeded
CRS-2676: Start of 'ora.gipcd' on 'ebsdbi01p' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ebsdbi01p'
CRS-2672: Attempting to start 'ora.diskmon' on 'ebsdbi01p'
CRS-2676: Start of 'ora.diskmon' on 'ebsdbi01p' succeeded
CRS-2676: Start of 'ora.cssd' on 'ebsdbi01p' succeeded
ASM created and started successfully.
Disk Group GRID created successfully.
CRS-2672: Attempting to start 'ora.crf' on 'ebsdbi01p'

CRS-2672: Attempting to start 'ora.storage' on 'ebsdbi01p'
CRS-2676: Start of 'ora.storage' on 'ebsdbi01p' succeeded
CRS-2676: Start of 'ora.crf' on 'ebsdbi01p' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ebsdbi01p'
CRS-2676: Start of 'ora.crsd' on 'ebsdbi01p' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 66759df176834f81bf63288b7d65a203.
Successfully replaced voting disk group with +GRID.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 66759df176834f81bf63288b7d65a203 (ORCL:EBSP_OCR_VOTE01) [GRID]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ebsdbi01p'
CRS-2673: Attempting to stop 'ora.crsd' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.crsd' on 'ebsdbi01p' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'ebsdbi01p'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'ebsdbi01p'
CRS-2673: Attempting to stop 'ora.crf' on 'ebsdbi01p'
CRS-2673: Attempting to stop 'ora.ctssd' on 'ebsdbi01p'
CRS-2673: Attempting to stop 'ora.evmd' on 'ebsdbi01p'
CRS-2673: Attempting to stop 'ora.storage' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.storage' on 'ebsdbi01p' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.crf' on 'ebsdbi01p' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'ebsdbi01p' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'ebsdbi01p' succeeded
CRS-2677: Stop of 'ora.evmd' on 'ebsdbi01p' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'ebsdbi01p' succeeded
CRS-2677: Stop of 'ora.asm' on 'ebsdbi01p' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'ebsdbi01p' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.cssd' on 'ebsdbi01p' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.gipcd' on 'ebsdbi01p' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ebsdbi01p' has completed
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ebsdbi01p'
CRS-2672: Attempting to start 'ora.ctssd' on 'ebsdbi01p'
CRS-2676: Start of 'ora.ctssd' on 'ebsdbi01p' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ebsdbi01p' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ebsdbi01p'
CRS-2676: Start of 'ora.asm' on 'ebsdbi01p' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: ebsdbi01p
CRS-6016: Resource auto-start has completed for server ebsdbi01p
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
2018/03/20 23:38:23 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.GRID.dg' on 'ebsdbi01p'
CRS-2676: Start of 'ora.GRID.dg' on 'ebsdbi01p' succeeded
2018/03/20 23:39:33 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@ebsdbi02p oracrs]# /oracle/app/oracrs/oracrsdb/grid/root.sh





CRS-2672: Attempting to start 'ora.net1.network' on 'ebsdbi02p'
CRS-2676: Start of 'ora.net1.network' on 'ebsdbi02p' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'ebsdbi02p'
CRS-2676: Start of 'ora.ons' on 'ebsdbi02p' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'ebsdbi01p' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'ebsdbi01p'
CRS-2677: Stop of 'ora.scan1.vip' on 'ebsdbi01p' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'ebsdbi02p'
CRS-2676: Start of 'ora.scan1.vip' on 'ebsdbi02p' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'ebsdbi02p'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'ebsdbi02p' succeeded

[root@ebsdbi03ptmp oracrs]# /oracle/app/oracrs/oracrsdb/grid/root.sh





CRS-2672: Attempting to start 'ora.net1.network' on 'ebsdbi03p'
CRS-2676: Start of 'ora.net1.network' on 'ebsdbi03p' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'ebsdbi03p'
CRS-2676: Start of 'ora.ons' on 'ebsdbi03p' succeeded
/oracle/app/oracrs/oraInventory/logs/installActions2018-03-20_10-50-08PM.log
INFO: PRVF-7590 : "ntpd" is not running on node "ebsdbi02p"
INFO: PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
INFO: PRVF-5415 : Check to see if NTP daemon or service is running failed
INFO: Clock synchronization check using Network Time Protocol(NTP) failed
INFO: PRVF-9652 : Cluster Time Synchronization Services check failed
We are waiting on F/W rules. Run CVU after rules are in place.
You can find the log of this install session at:
/oracle/app/oracrs/oraInventory/logs/installActions2018-03-20_10-50-08PM.log
Grid Patches
Patch 6880880
cd /oracle/app/oracrs/oracrsdb/grid/OPatch
zip -r opatch_old.zip *
cd /oracle/app/oracrs/oracrsdb/grid
unzip -o /dbinst/software/p6880880_122010_Linux-x86-64.zip
Patch 27010930: DATABASE PROACTIVE BUNDLE PATCH 12.1.0.2.180116

p27010930_121020_Linux-x86-64
export PATH=/oracle/app/oracrs/oracrsdb/grid/OPatch:$PATH
export ORACLE_HOME=/oracle/app/oracrs/oracrsdb/grid
cd /dbinst/software/27010930
opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 26925263
vi /tmp/patch_list_gihome.txt
/dbinst/software/27010930/26983807
Check for free space
opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
Apply Patch
As root:
opatchauto apply /dbinst/software/27010930
On 2nd and 3rd nodes

As root:
export PATH=/oracle/app/oracrs/oracrsdb/grid/OPatch:$PATH
export ORACLE_HOME=/oracle/app/oracrs/oracrsdb/grid
opatchauto apply /dbinst/software/27010930
Oracle Database Proactive Patch 12.1.0.2.180116 Known Issues (Doc ID 2326746.1)

Patch 26838953
p26838953_12102180116ProactiveBP_Linux-x86-64.zip
opatch prereq CheckConflictAgainstOHWithDetail -ph ./
As root:
/oracle/app/oracrs/oracrsdb/grid/crs/install/rootcrs.sh -prepatch
As oracrs:
/oracle/app/oracrs/oracrsdb/grid/OPatch/opatch apply
As root: /oracle/app/oracrs/oracrsdb/grid/crs/install/rootcrs.sh -postpatch
Patch 27611612
p27611612_12201180116DBJAN2018RU_Linux-x86-64.zip
opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Conflict with Composite Patch 26925263
Conflict with Sub-Patch 26717470
Conflict with Sub-Patch 24732088
Not Applied
ASM Diskgroups
asmca
Disk group NameRedundanc Disks AU ASM Database
y size Compatibility Compatibility
EBSP_REDO01_DG External ORCL:EBSP_REDO01 1 12.1.0.0 11.2.0.4
EBSP_REDO02_DG External ORCL:EBSP_REDO02 1 12.1.0.0 11.2.0.4
EBSP_ARCH01_DG External ORCL:EBSP_ARCH01 1 12.1.0.0 11.2.0.4
ORCL:EBSP_ARCH02
EBSP_FRA_DG External ORCL:EBSP_FRA01 1 12.1.0.0 11.2.0.4
ORCL:EBSP_FRA02
EBSP_TEMP_DG External ORCL:EBSP_TEMP01 1 12.1.0.0 11.2.0.4
EBSP_DATA_DG External ORCL:EBSP_DATA01 1 12.1.0.0 11.2.0.4
ORCL:EBSP_DATA02
ORCL:EBSP_DATA03
ORCL:EBSP_DATA04
ORCL:EBSP_DATA05
ORCL:EBSP_DATA06
ORCL:EBSP_DATA07
ORCL:EBSP_DATA08
ORCL:EBSP_DATA09
Make sure to use sector_size and compatible=11.2, au =16M
CREATE DISKGROUP data NORMAL REDUNDANCY
FAILGROUP controller1 DISK '/devices/diska1', '/devices/diska2', '/devices/diska3', '/devices/diska4'
FAILGROUP controller2 DISK '/devices/diskb1', '/devices/diskb2', '/devices/diskb3', '/devices/diskb4'
ATTRIBUTE 'compatible.asm' = '11.2', 'compatible.rdbms' = '11.2', 'sector_size'='512';
ASM in init.ora
show parameter spfile
+GRID/ebsp-dc3/ASMPARAMETERFILE/registry.253.971307355
Backup the spfile on disk
create pfile='$ORACLE_HOME/dbs/asm_pfile_orig_bkp' from spfile;
alter system set ASM_DISKSTRING='ORCL:*' scope=both sid='*';
alter system set memory_max_target=2G scope=spfile sid='*';
alter system set memory_target=1G scope=spfile sid='*';
create pfile='$ORACLE_HOME/dbs/asm_pfile_bkp' from spfile;
srvctl setenv asm -T "TZ=Europe/Berlin"
Setup the Database Oracle Home

On target:
mkdir /oracle/app/crm/crmdb/11gRAC
Copy software from source
vi ~/rsync_oh_exclude
- admin/CRM*/**
- log/**
rsync --delete -auvzpoglH -e ssh --exclude-from=$HOME/rsync_oh_exclude

/oracle/app/crm/crmdb/11gRAC/ 10.254.30.188:/oracle/app/crm/crmdb/11gRAC/
scp /oracle/dat/crm/share/orapwCRM_DC1
10.254.30.188:/oracle/app/crm/crmdb/11gRAC/dbs/orapwCRM1
All soft links
find . -type l -ls
./lib/cx_Oracle.so -> /nsr/SAPbackup/cx_Oracle/REDHAT_6.6_Linux64_Python2.6.6/Oracle11gRAC_64bit/cx_Oracle.so
./lib/libclntsh.so -> libclntsh.so.11.1
./lib/libocci.so -> libocci.so.11.1
./lib/libodm11.so -> libnfsodm11.so
./lib/libagtsh.so -> libagtsh.so.1.0
./lib/libclntsh.so.10.1 -> /oracle/app/crm/crmdb/11gRAC/lib/libclntsh.so
./lib/liborasdkbase.so -> liborasdkbase.so.11.1
./lib/liborasdk.so -> liborasdk.so.11.1
./dbs/orapwCRM1 -> /oracle/dat/crm/share/orapwCRM_DC1
./jdk/jre/lib/amd64/server/libjsig.so -> ../libjsig.so
./precomp/public/SQLDA.H -> sqlda.h
./precomp/public/SQLCA.COB -> sqlca.cob
./precomp/public/SQLCA.H -> sqlca.h
./precomp/public/ORACA.H -> oraca.h
./precomp/public/ORACA.COB -> oraca.cob
./network/admin/CRM1_ebsdbi01p_ifile.ora -> /oracle/app/admin/CRM/net/admin/CRM_db_ifile_DC1.ora
./network/admin/sqlnet_ifile.ora -> /oracle/app/admin/CRM/net/admin/db_CRM_sqlnet_ifile.ora
./network/admin/listener_ifile.ora -> /oracle/app/admin/CRM/net/admin/db_CRM1_ebsdbi01p_listener_ifile.ora
./appsutil/clone/jre/lib/amd64/server/libjsig.so -> ../libjsig.so
./appsutil/clone/jre/javaws/javaws -> ../bin/javaws
./appsutil/clone/jre/bin/ControlPanel -> ./jcontrol
./appsutil/etcc -> /dbinst/stage/Rci/ETCC
./appsutil/jre/lib/amd64/server/libjsig.so -> ../libjsig.so
./appsutil/jdk1.6.0_29/jre/lib/amd64/server/libjsig.so -> ../libjsig.so
./appsutil/jdk1.6.0_29/jre/javaws/javaws -> ../bin/javaws
./appsutil/jdk1.6.0_29/jre/bin/ControlPanel -> ./jcontrol
./appsutil/jdk1.6.0_29/man/ja -> ja_JP.eucJP
./appsutil/jdk1.6.0_29/bin/ControlPanel -> ./jcontrol
./appsutil/jre.old/lib/amd64/server/libjsig.so -> ../libjsig.so
./appsutil/jre.old/javaws/javaws -> ../bin/javaws
./appsutil/jre.old/bin/ControlPanel -> ./jcontrol
./bin/lbuilder -> /oracle/app/crm/crmdb/11gRAC/nls/lbuilder/lbuilder
alter system set recovery_parallelism=32 scope=spfile sid='*';
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dnfs_off
chown root oradism
chmod u+s oradism
attachome
Backup the inventory
As root:
cp -pr /oracle/app/oracrs/oraInventory
/oracle/app/oracrs/oraInventory_20180320
As crm:
vi /oracle/app/crm/crmdb/11gRAC/oraInst.loc
cd $ORACLE_HOME/oui/bin
/oracle/app/crm/crmdb/11gRAC/oui/bin/runInstaller -attachHome -
noClusterEnabled ORACLE_HOME=/oracle/app/crm/crmdb/11gRAC
ORACLE_HOME_NAME=OraDb11g_home1
CLUSTER_NODES=ebsdbi01p,ebsdbi02p,ebsdbi03p
"INVENTORY_LOCATION=/oracle/app/oracrs/oraInventory"
LOCAL_NODE=ebsdbi01p
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 51199 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/app/oracrs/oraInventory
'AttachHome' was successful.
Repeat on nodes 2 and 3. Change the LOCAL_NODE as needed
$ORACLE_HOME/OPatch/opatch lsinventory
Setup the Applications

On Target
mkdir -p /oracle/shr/crm/share_r12/tmp/
mkdir -p /oracle/shr/crm/share_r12/tools/
mkdir -p /oracle/shr/crm/share_r12/fs1/EBSapps/appl/
mkdir -p /oracle/shr/crm/share_r12/fs1/EBSapps/comn/
mkdir -p /oracle/shr/crm/share_r12/fs1/EBSapps/10.1.2/
mkdir -p /oracle/shr/crm/share_r12/fs1/FMW_Home
mkdir -p /oracle/shr/crm/share_r12/fs1/inst/
mkdir -p /oracle/shr/crm/share_r12/fs2/EBSapps/appl/
mkdir -p /oracle/shr/crm/share_r12/fs2/EBSapps/comn/
mkdir -p /oracle/shr/crm/share_r12/fs2/EBSapps/10.1.2/
mkdir -p /oracle/shr/crm/share_r12/fs2/FMW_Home
mkdir -p /oracle/shr/crm/share_r12/fs1/inst/
mkdir -p /oracle/shr/crm/share_r12/fs_ne/EBSapps/appl/
mkdir -p /oracle/shr/crm/share_r12/fs_ne/EBSapps/temp/
mkdir -p /oracle/shr/crm/share_r12/fs_ne/EBSapps/patch/
mkdir -p /oracle/shr/crm/share_r12/fs_ne/EBSapps/log/
Setup APPL_TOP
vi $HOME/rsync_appl_top_exclude
- admin/log/**
- admin/out/**
- *.log
- *.out
- *.req
- *.mgr
- *.tmp
- *.rti
- *.PDF
- *.tar.gz
- *.EXCEL
function tmp {
rsync --delete -auvzpoglH -e ssh --exclude-from=$HOME/rsync_appl_top_exclude ${1} 10.254.30.203:${1}
}
tmp /oracle/shr/crm/share_r12/EBSapps.env
tmp /oracle/shr/crm/share_r12/tools/
tmp /oracle/shr/crm/share_r12/fs1/EBSapps/appl/
tmp /oracle/shr/crm/share_r12/fs1/EBSapps/comn/
tmp /oracle/shr/crm/share_r12/fs1/EBSapps/10.1.2/
tmp /oracle/shr/crm/share_r12/fs1/FMW_Home
tmp /oracle/shr/crm/share_r12/fs1/inst/
tmp /oracle/shr/crm/share_r12/fs2/EBSapps/appl/
tmp /oracle/shr/crm/share_r12/fs2/EBSapps/comn/
tmp /oracle/shr/crm/share_r12/fs2/EBSapps/10.1.2/
tmp /oracle/shr/crm/share_r12/fs2/FMW_Home
tmp /oracle/shr/crm/share_r12/fs2/inst/
tmp /oracle/shr/crm/share_r12/fs_ne/inst/
tmp /oracle/shr/crm/share_r12/fs_ne/EBSapps/appl/
/oracle/shr/crm/share_r12/fs2/EBSapps/
appl – 23GB
comn – 33GB
10.1.2 – 2.1GB
cp -p /oracle/app/admin/CRM/net/admin/db_CRM1_ebsdbi01p_listener_ifile.ora
/oracle/app/admin/CRM/net/admin/db_CRM1_ebsdbi01p_listener_ifile.ora.DC3
vi /oracle/app/admin/CRM/net/admin/db_CRM1_ebsdbi01p_listener_ifile.ora
Change
(SID_DESC = (ORACLE_HOME = /oracle/app/crm/crmdb/11gRAC)(SID_NAME = CRM1)
(GLOBAL_DBNAME=CRM_DC1_DGMGRL))
To
(SID_DESC = (ORACLE_HOME = /oracle/app/crm/crmdb/11gRAC)(SID_NAME = CRM1)
(GLOBAL_DBNAME=CRM_DC3_DGMGRL))
Register Database with RAC

srvctl add database
srvctl modify database -d CRM_DC3 -a

"EBSP_ARCH01_DG,EBSP_DATA_DG,EBSP_FRA_DG,EBSP_GRID_DG,EBSP_REDO01_DG,
EBSP_REDO02_DG,EBSP_TEMP_VG"
srvctl add listener -l ebsi -p "TCP:51201" -o /oracle/app/ebsi/ebsidb/11gRAC

srvctl setenv listener -l ebsi -T "TZ=Europe/Berlin"
srvctl setenv database -d ebsi -T "TZ=Europe/Berlin"

Pending
To run I/O calibration and assess the I/O capability of the storage subsystem used by Oracle Database,
use the DBMS_RESOURCE_MANAGER.CALIBRATE_IO procedure:
SET SERVEROUTPUT ON
DECLARE
lat INTEGER;
iops INTEGER;
mbps INTEGER;
BEGIN
DBMS_RESOURCE_MANAGER.CALIBRATE_IO (2, 10, iops, mbps, lat);
DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);
DBMS_OUTPUT.PUT_LINE ('latency = ' || lat);
dbms_output.put_line('max_mbps = ' || mbps);
end;
/
max_iops = 307483
latency = 0
max_mbps = 3485
Login as oracrs user.
cd /oracle/app/oracrs/oracrsdb/grid/crs/install
vi s_crsconfig_ebsdbi02d_env.txt
TZ=Etc/UTC
Change To
TZ=Europe/Berlin
References:
FAQ: Flash Storage with ASM (Doc ID 1626228.1)
How To Align Partitions on Large HDD (Doc ID 1523947.1)
Supporting 4K Sector Disks [Video] (Doc ID 1133713.1)
Supporting ASM on 4K/4096 Sector Size (SECTOR_SIZE) Disks (Doc ID 1630790.1)
https://kb.netapp.com/app/answers/answer_view/a_id/1001578
Business Continuity for Oracle E-Business Suite Release 12.2 using Virtual Hosts with Oracle 12c Physical
Standby Database (Doc ID 2088692.1)
The Cisco whitepaper recommends adding the following to GRUB_CMDLINE_LINUX:

intel_idle.max_cstate=0 processor.max_cstate=0
And GRUB_DEFAULT=saved
what do you think?
cat /etc/multipath.conf – From Cisco
defaults {
polling_interval 10
}
devices {
device {
vendor "NETAPP"
path_grouping_policy multibus
path_checker tur
path_selector "queue-length 0"
fast_io_fail_tmo 10
dev_loss_tmo 60
no_path_retry 0
}
}
From Netapp - https://library.netapp.com/ecm/ecm_download_file/ECMLP2547957

https://library.netapp.com/ecm/ecm_download_file/ECMLP2547958
flush_on_last_del yes
max_fds max
pg_prio_calc avg
queue_without_daemon no
user_friendly_names no  will this impact current names?
dev_loss_tmo infinity
fast_io_fail_tmo 5
failback immediate
features 3 queue_if_no_path pg_init_retries 50
getuid_callout uid_attribute "ID_SERIAL"
hardware_handle 1 alua
path_checker tur
path_grouping_policy group_by_prio
path_selector service-time 0
prio ALUA
product LUN.*
rr_min_io 1000
rr_weight uniform
vendor NETAPP
The Netapp doc has recommendations for both Veritas and Oracle VM
Hi Bob,
Here is another recommendation
Current:
tuned-adm active
Current active profile: throughput-performance
Change to:
tuned-adm profile enterprise storage
systemctl status tuned.service  Shows below error

Mar 14 14:07:34 ebsdbi01ptmp tuned[3574]: raise exception_from_errno(errno)
Mar 14 14:07:34 ebsdbi01ptmp tuned[3574]: EnvironmentError: [Errno 105] No buffer spa...ne
Hint: Some lines were ellipsized, use -l to show in full.
Also for the VMs like SAP, we have to go through a tuning exercise, tuned-admin should be set to virtual-
guest
Reference: https://library.netapp.com/ecm/ecm_download_file/ECMLP2547958
Thanks,
Aditya
Hi Bob,
From: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
ucs_flashstack_oracleRAC12cr2.html
Below are for Pure storage, but I think it should be same for Netapp too. But I am not sure if these are
for physical or VM:
# Recommended settings for PURE Storage FlashArray
# Use noop scheduler for high-performance solid-state storage
ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE",
ATTR{queue/scheduler}="noop"
# Reduce CPU overhead due to entropy collection
ATTR{queue/add_random}="0"
# Schedule I/O on the core that initiated the process
ATTR{queue/rq_affinity}="2"
# End of Recommended settings for PURE Storage FlashArray
From Oracle: https://docs.oracle.com/en/database/oracle/oracle-database/18/cwlin/setting-the-disk-io-

scheduler-on-linux.html#GUID-B59FCEFB-20F9-4E64-8155-7A61B38D8CDF
For best performance for Oracle ASM, Oracle recommends that you use the Deadline I/O Scheduler.
Disk I/O schedulers reorder, delay, or merge requests for disk I/O to achieve better throughput and
lower latency. Linux has multiple disk I/O schedulers available, including Deadline, Noop, Anticipatory,
and Completely Fair Queuing (CFQ).
On each cluster node, enter the following command to verify that the Deadline disk I/O scheduler is
configured for use:
# cat /sys/block/${ASM_DISK}/queue/scheduler
noop [deadline] cfq
What we have:
cd /sys/block/dm-45/queue
[root@ebsdbi01ptmp queue]# cat iostats
0
[root@ebsdbi01ptmp queue]# cat add_random
0
[root@ebsdbi01ptmp queue]# cat scheduler
none
[root@ebsdbi01ptmp queue]# cat rq_affinity
0
cd /sys/block/sda/queue
cat scheduler
noop [deadline] cfq
So the ASM disks are not choosing the right scheduler. Need to change
More recommedations:
echo 256 > /sys/block/${block}/queue/nr_requests  Currently 128
echo 0 > /sys/block/${block}/queue/rotational  Currently 1
echo 256 > /sys/block/${block}/queue/read_ahead_kb  Currently 128

Oracle RAC Install

Uploaded by

Copyright:

Available Formats

Oracle RAC Install

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oracle RAC Install

Uploaded by

Copyright:

Available Formats

Oracle RAC Install

Oracle Database Grid Infrastructure 12.1.0.2.0 for Linux x86-64

ASM Disk Configuration

Default user to own the driver interface []: oracrs

# Partition disks to align

Create ASMLib Disks

Confirm the disks are using dm- and not sd devices

Install and Configure Oracle Grid Infrastructure for a Cluster

DR ( Public hostname is different, but virtual hostname is same)

Check SSH Connectivity

Use Standard ASM for storage

The following environment variables are set as:

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Creating /etc/oratab file...

2018/03/20 23:33:32 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful

CRS-4133: Oracle High Availability Services has been stopped.

ASM created and started successfully.

Disk Group GRID created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'ebsdbi01p'

CRS-2672: Attempting to start 'ora.asm' on 'ebsdbi01p'

The following environment variables are set as:

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Creating /etc/oratab file...

2018/03/20 23:40:16 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful

CRS-4133: Oracle High Availability Services has been stopped.

[root@ebsdbi03p ~]# /oracle/app/oracrs/oracrsdb/grid/root.sh

The following environment variables are set as:

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Creating /etc/oratab file...

2018/03/20 23:45:27 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful

CRS-4133: Oracle High Availability Services has been stopped.

Patch 27010930: DATABASE PROACTIVE BUNDLE PATCH 12.1.0.2.180116

On 2nd and 3rd nodes

Oracle Database Proactive Patch 12.1.0.2.180116 Known Issues (Doc ID 2326746.1)

srvctl setenv asm -T "TZ=Europe/Berlin"

Setup the Database Oracle Home

rsync --delete -auvzpoglH -e ssh --exclude-from=$HOME/rsync_oh_exclude

alter system set recovery_parallelism=32 scope=spfile sid='*';

Starting Oracle Universal Installer...

Setup the Applications

Register Database with RAC

srvctl modify database -d CRM_DC3 -a

srvctl add listener -l ebsi -p "TCP:51201" -o /oracle/app/ebsi/ebsidb/11gRAC

srvctl setenv database -d ebsi -T "TZ=Europe/Berlin"

Login as oracrs user.

The Cisco whitepaper recommends adding the following to GRUB_CMDLINE_LINUX:

cat /etc/multipath.conf – From Cisco

From Netapp - https://library.netapp.com/ecm/ecm_download_file/ECMLP2547957

systemctl status tuned.service  Shows below error

From Oracle: https://docs.oracle.com/en/database/oracle/oracle-database/18/cwlin/setting-the-disk-io-

You might also like