Building a RAC Test Environment

on VMWare For Free

Dan Norris
d
[email protected]
i @ i
http://www.dannorris.com/
Thanks to Tim Hall @ oracle-base.com
Agenda
•Who is Dan?
•VMWare Orientation
•Importance of prerequisite checking
•Configuring node #1
•Cloning (VM) node #1
•Installing Clusterware
•Installing ASM and Database
•Creating Database
•Common
C Ch
Challenges
ll and
d IIssues

2
Who is Dan?
• Virgo
• Scuba Diver (PADI Advanced OW, Nitrox)
• Over 21, under 35
• Oracle DBA & UNIX Admin backgroundg
• Certifiable: OCM, ACE Director, RHCE
• Consultant, mostly fixing things that are broken
• Active community participant: RAC SIG, SIG
Council, DBA Track Manager, blogger, tweeter
• ESA Practice Manager at Piocon Technologies

3
VMWare Orientation
•Desktop Products
–VMWare
VMWare Workstation: “Full”
Full product,
product $
–VMWare Player: Free, no creation, just use
•Server
Server Products
–VMWare Infrastructure (ESX Server): $$$,
robust,, clustering
g capable,
p , OS replacement
p
–VMWare VirtualCenter: $$, management
tool for whole environment, Windows
–VMWare Server: free, limited, on top of OS

4
VMWare Server Hardware
•Almost any server will do, mine is:
–Dell SC440: 1 x dual-core 3.0 Ghz
–2 x 500Gb internal SATA drives, using LVM
–4 Gb RAM
•I use Linux (OEL 5) as host OS
•Let Linux use LVM (default), allows disk
space to be pooled for 900+
900 Gb on /u01
•VMWare Server 1.0.5
•Two Virtual Networks: 1 bridged
g & 1 host-
only

5
RAC: System Architecture

Node1 Cluster
Cl t Node2
Interconnect
rac1 rac2

Local Shared Local

Storage Storage Storage

6
First: Build Node1

Node1 Node2

Local Shared Local

Storage Storage Storage

7
Node1: Create New VM

8
Node1: Create New VM

9
Node1: Create New VM

10
Node1: Create New VM

11
Node1: Create New VM

12
Node1: Create New VM

13
Node1: Create New VM

14
Node1: Create New VM

15
Node1: Create New VM

(Pretend it says RAC1)

16
Node1: Create New VM

17
Node1: Create New VM

18
Node1: Create New VM

19
Node1: Create New VM

20
Node1: Create New VM

21
Node1: Create New VM

22
Node1: Create New VM

23
Install Node1 OS
•Plan the network addresses for public and
private networks
•Start the VM and the CD image should boot
•Install the OS
OS, ensuring enough swap and
/tmp
•Assign
A i statict ti addresses
dd tto each
h iinterface
t f
•See whitepaper for package and setup
d t il
details

24
Post-Install Steps on Node1
•Modify /etc/hosts
•Set
Set kernel parameters
•Set limits in /etc/security/limits.conf
•Add groups (oinstall, dba), oracle user
•Set
Set up oracle user’s
user s profile (shell limits,
ORACLE_BASE)
•Set up SSH user equivalency

25
Prerequisite Checking
•Most issues arise from misconfiguration
of OS,
OS packages
packages, networks
networks, or storage
•Tools exist to help check prerequisites
–CVU
CVU
–RDA
•Some basic networking knowledge helps
•Recipes
p can help,
p, be sure yyou
understand why, not just follow steps

26
Prerequisite Checking - RDA
•Metalink Note 314422.1 will orient you to
the RDA and has links for download
•Download the .zip file, unzip and run (as
root or oracle):
–./rda.sh –T hcve
–provide the release you want to check
–answer additional questions
–obtain results in ./output/RDA* files

27
Prerequisite Checking - RDA

28
Prerequisite Checking - RDA

29
Prerequisite Checking - CVU
•http://www.oracle.com/technology/produ
cts/database/clustering/cvu/cvu downloa
cts/database/clustering/cvu/cvu_downloa
d_homepage.html
•Download
Download a .zip
zip file
file, unzip itit, run it:
–cluvfy stage -list

30
Prerequisite Checking - CVU

On AMD64 platforms,
some packages will be
listed twice, once for
i386 and once for
AMD64

31
On AMD64 platforms,
some packages will be
listed twice, once for
i386 and once for
AMD64

32
Done: Build Node1

Node1 Node2

Local Shared Local

Storage Storage Storage

33
Next: Add Shared Storage

Node1 Node2

Local Shared Local

Storage Storage Storage

34
Node1: Add Shared Storage
•Shut down guest OS in Node1
•On
On the VMWare server
ser er host OS
OS, create
a new directory for shared disk files:
mkdir –p
p /u01/vmware/vms/rac
/u01/vmware/vms/rac-disks
disks
•Back at the VMWare Server Console,
add
dd new disks
di k to
t the
th Node1
N d 1 (RAC1) VM
as shown in the following example.

35
Node1: Add Shared Storage

36
Node1: Add Shared Storage

37
Node1: Add Shared Storage

38
Node1: Add Shared Storage

39
Node1: Add Shared Storage

40
Node1: Add Shared Storage

41
Node1: Add Shared Storage
•Repeat to create the following:
Virtual
Vi t l Device
D i
Path Size
Node
/u01/vmware/vms/rac-disks/ocr1.vmdk 10 Gb SCSI 1:0
/u01/vmware/vms/rac-disks/voting1.vmdk 10 Gb SCSI 1:1
/u01/vmware/vms/rac-disks/asm1.vmdk 10 Gb SCSI 1:2
/u01/vmware/vms/rac-disks/asm2.vmdk 10 Gb SCSI 1:3
/u01/vmware/vms/rac-disks/asm3.vmdk 10 Gb SCSI 1:4

42
Node1: Add Shared Storage
•Now for the tricky part: hand-editing
VMWare configuration filefile.
•The changes we make enable the disks
to be shared with other VMs
VMs.
•Add or update the following parameters
i th
in the “/
“/u01/vmware/vms/RAC1/Red
/ / / /
Hat Enterprise Linux 4.vmx” file

43
Node1: Add Shared Storage
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
scsi1.sharedBus
i1 h dB = "VIRTUAL"

44
Node1: Add Shared Storage
scsi1:0.deviceType = "disk"
scsi1:0.redo = ""
scsi1:1.deviceType = "disk"
scsi1:1.redo = ""
scsi1:2 deviceType
scsi1:2.deviceType = "disk"
scsi1:2.redo = ""
scsi1:3.deviceType
yp = "disk"
scsi1:3.redo = ""
scsi1:4.deviceType = "disk"
scsi1:4.redo = ""

45
Node1: Add Shared Storage
•Boot Node1 again and partition the disks
to create partition 1 as the whole disk
•“Quick and Dirty” setup: add this to
/etc/rc local:
/etc/rc.local:
chown oracle:oinstall /dev/sd[b-f]1
chmod 600 /dev/sd[b-f]1
/ /

46
Done: Add Shared Storage

Node1 Node2

Local Shared Local

Storage Storage Storage

47
Next: Clone Node1 to Make Node2

Node1 Node2

Local Shared Local

Storage Storage Storage

48
Create Node2: Clone Node1
•Node2 is a VMWare clone of Node1
•Shut
Sh t do
downn Node1 nicel
nicely
•On host OS, do
cp –Rp /u01/vmware/vms/RAC1 \
/u01/vmware/vms/RAC2

mv /u01/vmware/vms/RAC2/RAC1.vmx \
/u01/vmware/vms/RAC2/RAC2 vmx
/u01/vmware/vms/RAC2/RAC2.vmx

49
Create Node2: Clone Node1
•Boot Node2 (leave Node1 down)
•During power on
on, choose Create when
prompted about handling the identifer.
•Modify
–/etc/sysconfig/network
–/etc/sysconfig/network-scripts/ifcfg-eth0
–/etc/sysconfig/network-scripts/ifcfg-eth1
/ / fi / k i /if f h1
•From the console, run “service network
restart”

50
Create Node2: Clone Node1
•Boot Node2 (leave Node1 down)
•During
During power on
on, choose Create when
prompted about handling the identifer.
•Modifyy
– /etc/sysconfig/network
– /etc/sysconfig/network-scripts/ifcfg-eth0
– /etc/sysconfig/network-scripts/ifcfg-eth1
/ / y g/ p / g

•From the virtual machine’s console, run

“service network restart”
•You can now power up Node1

51
Prerequisite Checking - CVU
•Finally able to complete more
prerequisite checking
•As the oracle user, run CVU to confirm
private networking
networking, and user
equivalence:
./cluvfy
/ l f stage
t –post
t hwos
h –n rac1,rac2
1 2

52
53
Prerequisite Checking - CVU
•Shared storage checks don’t work on
VMWare since the disks do not support
SMART (used by CVU to verify disk
accessibility)
•Next, check the pre-crsinst stage like this:
./cluvfy
/ l f –pre crsinst
i t –n rac1,rac2
1 2 \
–r 11gR1

54
Prerequisite Checking - CVU

55
Prerequisite Checking - CVU

56
Prerequisite Checking - CVU
•With all CVU checks complete and
successful we can move ahead to
successful,
installing clusterware!

57
Done: Node2 Created

Node1 Node2

Local Shared Local

Storage Storage Storage

58
Next: Install Clusterware

Node1 Cluster
Cl t Node2
Interconnect

Local Shared Local

Storage Storage Storage

59
Installing Clusterware
•Clusterware installation performed from
one node (Node1)
(Node1).
•Ensure /etc/hosts is configured with host
addresses VIP addresses
addresses, addresses, and private
addresses

60
Installing Clusterware

61
Installing Clusterware

62
Installing Clusterware

63
Installing Clusterware

64
Installing Clusterware

65
Installing Clusterware

66
Installing Clusterware

67
Installing Clusterware

68
Installing Clusterware

69
Installing Clusterware

70
Installing Clusterware

71
Installing Clusterware

72
Installing Clusterware
Note: Do not
run these
scripts in
parallel. They
should be run
sequentially.

73
Installing Clusterware

74
Installing Clusterware

Now go back
and click OK
on the installer
prompt
p p box.

75
Installing Clusterware

76
Installing Clusterware

77
Installing Clusterware

78
Installing Clusterware
•Once the installation is complete, you
can check the status with crs_stat
crs stat -t t

79
Finished: Install Clusterware

Node1 Cluster
Cl t Node2
Interconnect

Local Shared Local

Storage Storage Storage

80
Next: Install ASM and Database

Node1 Cluster
Cl t Node2
Interconnect
+ASM1 +ASM2

Local Shared Local

Storage Storage Storage

81
Installing ASM & Database

82
Installing ASM & Database

83
Installing ASM & Database

84
Installing ASM & Database

85
Installing ASM & Database

86
Installing ASM & Database

87
Installing ASM & Database

88
Installing ASM & Database

89
Installing ASM & Database

90
Installing ASM & Database

91
Installing ASM & Database

92
Installing ASM & Database

93
Installing ASM & Database

94
Installing ASM & Database

95
Installing ASM & Database

96
Installing ASM & Database

97
Installing ASM & Database

98
Finished: Install ASM and Database

Node1 Cluster
Cl t Node2
Interconnect
+ASM1 +ASM2

Local Shared Local

Storage Storage Storage

99
Next: Create Database

rac1 Cluster
Cl t
rac2
Node1 Interconnect Node2
+ASM1 +ASM2

Local Shared Local

Storage Storage Storage

100
Create Database

101
Create Database

102
Create Database

103
Create Database

104
Create Database

105
Create Database

106
Create Database

107
Create Database

108
Create Database

109
Create Database

110
Create Database

111
Create Database

112
Create Database

113
Create Database

114
Create Database

115
Create Database

116
Create Database

117
Create Database

118
Create Database

119
Create Database

120
Create Database

121
Create Database

…after a rather long pause while

i t
instances are started…(drumroll,
t t d (d ll please)
l )

122
Done: Create Database

rac1 Cluster
Cl t
rac2
Node1 Interconnect Node2
+ASM1 +ASM2

Local Shared Local

Storage Storage Storage

123
Common Issues #1: Firewalls
•Firewall and SELinux should be disabled
•Can
Can be done d during
ring install with
ith some
versions of Linux. If not, then…
•Set
S t SELINUX=disabled in i
/etc/sysconfig/selinux
•chkconfig iptables off
•Reboot
Reboot

124
Common Issues #2: Storage
•Shared Storage misconfiguration –
different names for disks on each node
•For RHEL or OEL 5, use udev for
privileges and device mapping
•Note Linux v4 and v5 handle things
differently
•Ensure you use proper device in
multipath configurations (not applicable
in VMWare)

125
Common Issues #2: Storage, cont’d
Note Title

443996.1 How to map raw device on RHEL5 and OEL5

How to setup /etc/udev/rules.d for OCR and Voting for RAC on

414897.1
SLES10/RH5/EL5

371814.1 Using udev with Oracle Architecture (RAC & ASM) – Red Hat 4.0

456239.1 Understanding device-mapper in Linux 2.6 kernel

357472.1 Configuring device-mapper for CRS/ASM

Configuring raw devices for Oracle 10gR2 (10.2.0) Clusterware on

465001 1
465001.1
RH/EL 5

126
Common Issues #3: Networking
•Networking, especially private network,
must be configured properly
•NO CROSSOVER CABLES ALLOWED!
•In VMWare,
VMWare ensure you use correct
interface for public, private networks
•Use
U h host-only
t l network
t k ffor private,
i t bridged
b id d
for public
•Set
S t up bbonding
di b before
f iinstalling
t lli clusterware
l t

127
Common Issues #4: Filesystems
•Don’t “go nuts” with Linux filesystem
layout
•Avoid too many filesystems (especially
on a test system)
•Ensure you create sufficient swap space

128
Common Issues #5: Host equivalency

•Don’t hate SSH, just set it up properly!

ssh keygen –t
ssh-keygen t dsa
cat ~/.ssh/id_dsa.pub >> \
~/.ssh/authorized_keys
/ /
scp –rp ~/.ssh othernode:.
ssh <remote_node> date
•Metalink Note 300548
300548.1
1 is helpful

129
References
•Much of this article originated from Tim
Hall’s
Hall s article at oracle-base
oracle-base.com
com (with
permission--Thanks, Tim!):
http://tinyurl com/39ywx2
http://tinyurl.com/39ywx2
•Oracle 11g Documentation
•Oracle Metalink, multiple notes and
articles

130
131
RAC SIG Events
• See www.oracleracsig.org for details
–Webcasts: Average 2x per month, live
–Conference
C f Events:
• Customer Panel, Monday @ 10:30a
• Experts Panel, Tuesday @ 12:15p
• Birds
Bi d off a F
Feather,
th W Wednesday
d d @1 1:30p
30
–Forums (via OTN): Lots of participation from RAC
SIG as well as Oracle gurus
• Join
J i th
the RAC SIG att www.oracleracsig.org!
l i !

132
Building a RAC Test Environment on
VMWare For Free

S
Session
i #341

Dan Norris,
Norris Twitter: dannorris
Piocon booth #1619
dnorris@piocon com
[email protected]
http://www.dannorris.com/
Legal
The information contained herein should be deemed reliable
but not guaranteed. The author has made every attempt to
provide current and accurate information. If yyou have any
p y
comments or suggestions, please contact the author at:

[email protected]

You may request redistribution permission from

[email protected].
Copyright © 2008, Piocon Technologies

134