ASM Configuration Using ASMLIB On RHEL 4 and Above 6.X
ASM Configuration Using ASMLIB On RHEL 4 and Above 6.X
8)
This Tutorial was done on RHEL-AS (4.8) installed on VMWare for learning purpose.
ASM can be setup using the ASM Library Driver or RAW devices. For older Oracle database
and Red-Hat Enterprise Linux versions RAW devices are still a recommended option for ASM
and data files, and remain supported. RAW Partitions allow Oracle to bypass the OS cache.
It creates all Oracle database files on RAW BLOCK DEVICES managed by ASM using ASMLib
calls. RAW devices are NOT required with this method as ASMLib works with block devices.
This method creates all Oracle database files on RAW CHARACTER DEVICES managed by ASM using
Standard Linux I/O system calls. You will be required to create RAW devices for all disk
partitions used by ASM.
For Oracle 10g Release 2 in Red Hat Enterprise Linux 4 and 5 it is NOT recommended to use
raw devices but to use block devices instead. Raw I/O is still available in Red Hat
Enterprise Linux 4 and 5, but it is now a deprecated interface.
By default reading and writing to block devices are buffered I/Os. In fact, Raw I/O was
briefly deprecated by the Linux community, why because it has been replaced by the O_DIRECT
flag which can be used to open block devices to bypass the OS cache. So Oracle 10g R2 no
longer requires raw devices for the database. Oracle database 10g R2 automatically opens
all block devices such as SCSI disks using the O_DIRECT flag, thus bypasses the OS cache;
so starting with Oracle 10g under Linux, RAW devices support does NOT matter anymore.
Since Oracle uses the O_DIRECT kernel flag to open database files and thereby bypass file
caching so RAW driver is DEPRECATED does NOT mean removed; it means they suggest you
DO NOT USE it. The deprecation was because they intended to remove it at some future point
but never actually did.
As I said above In LINUX kernel, RAW devices were deprecated and scheduled for removal at
one point, because the O_DIRECT flag can be used instead. However, later the decision was
made to keep RAW devices support since some software cannot use the O_DIRECT flag. Support
for raw devices was reinstated in Red Hat Enterprise Linux 5.4 and later.
WHAT IS ASMLIB ?
Oracle has developed a storage management interface called the ASMLIB API.
ASMLib is the support library for the ASM. Although ASMLib is NOT mandatory to run ASM.
ASMLIB allows an Oracle database using ASM more efficient and capable access to disk groups.
ASMLIB is an alternative interface for ASM to access block devices. ASM is STILL able to
access block devices without ASMLib, by simply using the block device drivers that have
always been supplied by the OS. On Linux, ASM is capable of referencing disks as raw devices
or by using the ASMLib software. Oracle ASMLIB is explained at Oracle Linux: Oracle ASMLib.
PURPOSE OF ASMLIB
The ASMLIB API provides 2 major feature enhancements over standard interfaces.
Linux does NOT guarantee persistent device naming i.e. Linux may change the device names
on reboot. The name of the devices are assigned by the Linux and is determined by the scan
order of the bus. Therefore, the device names are not guaranteed to persist across reboots.
For example, SCSI device /dev/sdb can change to /dev/sda if the scan order of the controllers
is not configured.
But ASM would be able to find its disks by scanning the device header because ASMLib labels
the disks and ensures that ASM can see the same disk names when trying to mount a diskgroup.
But still you have to configure the correct ownership and permissions for the device file
for ASM to be able to work with this device. Device Persistence with Oracle Linux ASMLib.
When using ASMLib no need to create raw devices & to point the Disk discovery path to it.
SYSTEM CONFIGURATION
# uname -a
Linux RAC-SERVER1 2.6.9-89.EL #1 Mon Apr 20 10:23:08 EDT 2009 i686 i686 i386 GNU/Linux
The 32 bit system will show i686 and i386 after the install date and time.
Oracle ASMLib is available for Linux; as a kernel module is needed only the supported Linux
distributions, the required RPM packages are downloadable from OTN. You need to pick exact
version that matches to your distribution based on kernel and architecture.
# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
There are 3 packages that one would normally install as part of ASM:
oracleasm + oracleasm-support + oracleasmlib
# uname –r
2.6.9-89.EL
All ASMLib installations require the oracleasmlib & oracleasm-support packages appropriate
for their machine. Library and Tools common for all.
# uname –p –i
i686 i386
Once you download above packages using ftp upload above all in Linux Server. You need #
(root) privilege to install all downloaded rpms.
POINTS TO NOTE
The oracleasm-2.6 package is a kernel driver that ships for various different kind of Linux
kernels that existed under EL 4. But we only need the driver for the kernel we are using
and cannot install the driver for a kernel we do not have installed.
oracleasmlib oracleasmlib-2.0.4-1.el4.i386.rpm
oracleasm-support oracleasm-support-2.1.7-1.el4.i386.rpm
oracleasm kernel driver oracleasm-2.6.9-89.EL-2.0.5-1.el4.i686.rpm
5. Stamp/Label the partitions as ASM disks. Configure ENV files for asm & oracle.
I have added 1st disk. Follow same procedure for other 5 disks. Each disk size is 10G and
all on different SCSI disk controller. Now you can create remaining all disks in the same
way as shown in figure above steps.
Above screenshot states we have added 6 Virtual disks for our ASM configuration. Once we
created required disks, then power on your Virtual machine just above left side of the
VMWARE WORK STATION and login as root # user.
# ls /dev/sd*
...
Since we added SCSI disk we can use /dev/sd* command. sda is our main disk where OS and
other apps are installed and sda1 to sda8 are its partitions. The following output from
'cat /proc/partitions' shows [sdb to sdg] are not partitioned yet.
# cat /proc/partitions
8 0 85983232 sda
8 1 112423 sda1
8 2 32764567 sda2
8 3 22531162 sda3
8 4 1 sda4
8 5 10546641 sda5
8 6 6144831 sda6
8 7 3068383 sda7
8 8 10811713 sda8
8 16 10485760 sdb
8 32 10485760 sdc
8 48 10485760 sdd
8 64 10485760 sde
8 80 10485760 sdf
8 96 10485760 sdg
6 disks (sdb to sdg) are the disks that Just I have added. ASMLib requires attached disks
to be partitioned. In this example, three 10Gg VMware virtual disks are to be used.
As of now no partitions, I would like to create at-least one partition one each disk. We
can create more than one partition on each disk and it’s possible to use each partition as
a different ASM disk.
Creating single partition on the whole device is really good because the reason is you have
one controller per disk in this case so as faster IO (If each disks are separated).
For production environment it is recommended to spread the data over different Physical
disks, to keep IO low for a Single disk. Just spreading data over different mount points
(on the same Physical disk) has no effect. At the end all IO gets to the same disk multiple
partitions on the same Physical spindle will NOT increase your I/O.
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)
Repeat the fdisk operation for all remaining disks in the same way as demonstrated above.
# fdisk /dev/sdb
# fdisk /dev/sdc
..
...
Once you created new partition using fdisk then you need to reboot LINUX based systems to
get partition recognized. The new table will be used at the next reboot. However you can
use partition table changes, by requesting that the operating system re-read the partition
table without reboot by partprobe command.
# partprobe /dev/sd[b-g]
# ls -l /dev/sd[b-g]1
brw-rw---- 1 root disk 8, 17 Feb 15 22:47 /dev/sdb1
brw-rw---- 1 root disk 8, 33 Feb 15 22:47 /dev/sdc1
brw-rw---- 1 root disk 8, 49 Feb 15 22:47 /dev/sdd1
brw-rw---- 1 root disk 8, 65 Feb 15 22:47 /dev/sde1
brw-rw---- 1 root disk 8, 81 Feb 15 22:47 /dev/sdf1
brw-rw---- 1 root disk 8, 97 Feb 15 22:47 /dev/sdg1
..
...
Following output shows that i have one partition for every disk.
# cat /proc/partitions
8 0 85983232 sda
8 1 112423 sda1
8 2 32764567 sda2
8 3 22531162 sda3
8 4 1 sda4
8 5 10546641 sda5
8 6 6144831 sda6
8 7 3068383 sda7
8 8 10811713 sda8
8 16 10485760 sdb
8 17 10482381 sdb1
8 32 10485760 sdc
8 33 10482381 sdc1
8 48 10485760 sdd
8 49 10482381 sdd1
8 64 10485760 sde
8 65 10482381 sde1
8 80 10485760 sdf
8 81 10482381 sdf1
8 96 10485760 sdg
8 97 10482381 sdg1
ASMLIB INSTALLATION
Install ASMLibs in following order to avoid package dependency errors make sure you are
logged in as root. Install using this command rpm -ivh package-name
i - Install
h - Hash to display
v – Verbose
Install ASMLibs in following order to avoid package dependency errors make sure you are
logged in as # (root) user.
The oracleasm-support package provides the utilities used to get the ASM driver up and
running. The oracleasm kernel driver (for the kernel you use. The oracleasmlib package
provides the actual ASM library; which provides the ASM discover utility, is not GPL and
therefore a separate download and not included in the Linux OS release distribution
oracleasm-2.6.9-89.EL-debuginfo-2.0.5-1.el4
oracleasm-support-2.1.7-1.el4
oracleasm-2.6.9-89.EL-2.0.5-1.el4
oracleasmlib-2.0.4-1.el4
# oracleasm -h
Usage: oracleasm [--exec-path=<exec_path>] <command> [ <args> ]
oracleasm --exec-path
oracleasm -h
oracleasm –V
You must be logged on as the # user to run the oracleasm script. The script calls a number
of library functions which are declared in /usr/lib/oracleasm/oracleasm.init.functions.
The /etc/init.d/oracleasm script has following options
# /etc/init.d/oracleasm
# /etc/init.d/oracleasm configure
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
ENABLING ASM
# /etc/init.d/
# ./oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
Above command will load the ASM driver and mount the ASM driver. By selecting “y” during
the configuration, the system will always load the module and mount it on system boot.
# ls -l /dev/sd[b-g]1
brw-rw---- 1 root disk 8, 17 Feb 10 05:44 /dev/sdb1
brw-rw---- 1 root disk 8, 33 Feb 10 05:44 /dev/sdc1
brw-rw---- 1 root disk 8, 49 Feb 10 05:44 /dev/sdd1
brw-rw---- 1 root disk 8, 65 Feb 10 05:44 /dev/sde1
brw-rw---- 1 root disk 8, 81 Feb 10 05:44 /dev/sdf1
brw-rw---- 1 root disk 8, 97 Feb 10 05:44 /dev/sdg1
# cd /etc/init.d/
[root@RAC-SERVER1 init.d]# ./oracleasm createdisk VOL1 /dev/sdb1
Marking disk "VOL1" as an ASM disk: [ OK ]
# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
..
...
# ls -l /dev/oracleasm/disks/
total 0
brw-rw---- 1 oracle dba 8, 17 Feb 16 19:40 VOL1
brw-rw---- 1 oracle dba 8, 33 Feb 16 19:41 VOL2
brw-rw---- 1 oracle dba 8, 49 Feb 16 19:41 VOL3
brw-rw---- 1 oracle dba 8, 65 Feb 16 19:41 VOL4
brw-rw---- 1 oracle dba 8, 81 Feb 16 19:41 VOL5
brw-rw---- 1 oracle dba 8, 97 Feb 16 19:41 VOL6
$ cat ora10.env
export ORACLE_BASE=/u02/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_home
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
export ORACLE_SID=orcl
$ cat asm.env
export ORACLE_BASE=/u02/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/asm_home
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
export ORACLE_SID=+ASM
START LISTENER
$ . ./asm.env
$ dbca
If the Oracle Cluster Synchronization Service (CSS) is not currently running, a warning
screen will be displayed. Follow the instructions and click the "OK" button. Once you've
returned to the previous screen, click the "Next" button again.
# /u02/app/oracle/product/10.2.0/db_home/bin/localconfig add
Adding to inittab
Startup will be queued to init within 30 seconds.
Enter Disk Group Name - and select required disks and click on OK.
Enter Disk Group Name - and select remaining disks and click on OK.
REDUNDANCY
In our example, the "High" redundancy is used. So I choose 3 disks for ASM_DG_DATA disk group &
another 3 disks for ASM_DG_FRA disk group. By default, each disk is a failure group.
POINTS TO NOTE
High redundancy means file extents are mirrored among 3 failure groups. When we go to normal
or high redundancy multiple fail groups and each group has to have at least one disk. The
level of redundancy cannot be change once a disk group has been created. ASM does not
mirror devices and provides file based redundancy only between failure groups.
POINTS TO NOTE
A minimum of four LUNs (Oracle ASM disks) of equal size and performance is recommended for
each disk group. ASM is NOT RAID; it provides Stripping and Mirroring capabilities which
are similar to RAID 10.
RAID uses a fixed stripe size and mirrors data at the device block level. i.e. (RAID Stripes
at the block-level and doesn’t aware of the different Oracle file types). ASM does not
behave exactly as RAID and it can be even smarter. It is aware of the different Oracle file
types so it can Stripe data differently.
ASM Stripes the data in small chunks of 128KB for lower I/O latency for small I/O operations
such as writing redo log entries to the redo log files (Fine-Grained Striping). For Data
Files for example, ASM will stripe the data in bigger chunks which are equal to the
Allocation Unit Size (Coarse-Grained Striping). Get more detail from here
. ./db10.env
Connected to:
With the Partitioning, OLAP, Data Mining and Real Application Testing options
NAME
--------------------------------------------------------------------------------
+ASM_DG_DATA/orcl/datafile/system.256.904166909
+ASM_DG_DATA/orcl/datafile/undotbs1.258.904166911
+ASM_DG_DATA/orcl/datafile/sysaux.257.904166909
+ASM_DG_DATA/orcl/datafile/users.259.904166911
+ASM_DG_DATA/orcl/datafile/example.265.904167107
NAME
--------------------------------------------------------------------------------
+ASM_DG_DATA/orcl/controlfile/current.260.904167019
+ASM_DG_FRA/orcl/controlfile/current.256.904167019
MEMBER
--------------------------------------------------------------------------------
+ASM_DG_DATA/orcl/onlinelog/group_3.263.904167043
+ASM_DG_FRA/orcl/onlinelog/group_3.259.904167049
+ASM_DG_DATA/orcl/onlinelog/group_2.262.904167033
+ASM_DG_FRA/orcl/onlinelog/group_2.258.904167037
+ASM_DG_DATA/orcl/onlinelog/group_1.261.904167023
+ASM_DG_FRA/orcl/onlinelog/group_1.257.904167027
6 rows selected.
$ . ./asm.env
ORACLE_SID = [orcl] ? +ASM
$ sqlplus / as sysdba
STATUS
------------
STARTED
INSTANCE_NAME
----------------
+ASM
NAME
------------------------------
ASM_DG_DATA
ASM_DG_FRA
6 rows selected.
$ cat /tmp/init+ASM.ora
+ASM.asm_diskgroups='ASM_DG_DATA','ASM_DG_FRA'#Manual Mount
*.asm_diskgroups='ASM_DG_DATA','ASM_DG_FRA'
*.background_dump_dest='/u02/app/oracle/admin/+ASM/bdump'
*.core_dump_dest='/u02/app/oracle/admin/+ASM/cdump'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='SHARED'
*.user_dump_dest='/u02/app/oracle/admin/+ASM/udump'
Oracle ASM Stripes and spreads data evenly across all of the disks in a disk group.
FILE TEMPLATES
ASM mirrors files using a variable Stripe size according to file templates. Templates are
collections of attribute values that are used to specify disk regions, file mirroring, and
striping attributes for an Oracle ASM file when it is created. When creating a file, you
can include a template name and assign desired attributes based on an individual file
rather than the file type.
Coarse-grained striping provides load balancing for disk groups while fine-grained striping
reduces latency for certain file types by spreading the load more widely.
ASMCMD COMMANDS
$ export ORACLE_SID=+ASM
$ asmcmd
ASMCMD> help
asmcmd [-p] [command]
Specifying the -p option allows the current directory to be displayed in the command
prompt, like so:
[command] specifies one of the following commands, along with its parameters.
commands:
--------
cd
du
find
help
ls
lsct
lsdg
mkalias
mkdir
pwd
rm
rmalias
ASMCMD> ls
ASM_DG_DATA/
ASM_DG_FRA/
ASMCMD> ls -lrt
State Type Rebal Unbal Name
MOUNTED HIGH N N ASM_DG_FRA/
MOUNTED HIGH N N ASM_DG_DATA/
ASMCMD> lsdg
ASMCMD> ls
ASM_DG_DATA/
ASM_DG_FRA/
ASMCMD> cd ASM_DG_DATA
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y ORCL/
ASMCMD> cd ORCL
ASMCMD> pwd
+ASM_DG_DATA/ORCL
ASMCMD> ls –l
Type Redund Striped Time Sys Name
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
N spfileorcl.ora =>
+ASM_DG_DATA/ORCL/PARAMETERFILE/spfile.266.904167283
ASMCMD> cd DATAFILE
ASMCMD> ls -l
Type Redund Striped Time Sys Name
DATAFILE HIGH COARSE FEB 19 01:00:00 Y EXAMPLE.265.904167107
DATAFILE HIGH COARSE FEB 19 01:00:00 Y SYSAUX.257.904166909
DATAFILE HIGH COARSE FEB 19 01:00:00 Y SYSTEM.256.904166909
DATAFILE HIGH COARSE FEB 19 01:00:00 Y UNDOTBS1.258.904166911
DATAFILE HIGH COARSE FEB 19 01:00:00 Y USERS.259.904166911
ASM is not a regular file system which can be accessed using standard FTP and HTTP service.
Oracle 10g R2 has introduced a new feature to access the ASM disk groups outside the
database via FTP and through a web browser using HTTP. The use of FTP and HTTP provide a
flexibility to manage the ASM environment.
Any FTP client can be used to move/copy the files stored on ASM DISKGROUP. To make this
enable we need to setup the Oracle XML DB access which is a onetime setup. Oracle XML DB
provides the file mapping functionality.
This can be done by executing the below script from SYSDBA and pass the FTP and HTTP port.
SYS> @?/rdbms/admin/catxdbdbca
Function dropped.
$ lsnrctl status
Now we are ready to access ASM disk groups using FTP client or HTTP browser.
For access through web browser: http://devserver.com:8080/
# cd Desktop/
[root@devserver Desktop]# ls -l
total 246016
-rw-r--r-- 1 root root 7 Oct 15 05:03 em
-rw-r--r-- 1 root root 251666432 Feb 19 03:06 SYSAUX.257.904166909
Connected to devserver.com.
220- devserver.com
Unauthorised use of this FTP server is prohibited and may be subject to civil and criminal
prosecution.
220 devserver.com FTP Server (Oracle XML DB/Oracle Database) ready.
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (devserver.com:root): : system
331 pass required for SYSTEM
Password:
230 SYSTEM logged in
Remote system type is Unix.
ftp> dir
ftp> cd sys
ftp> dir
ftp> cd asm
ftp> dir
ftp> cd ASM_DG_DATA
250 CWD Command successful
ftp> dir
ftp> cd ORCL
250 CWD Command successful
ftp> dir
ftp> cd DATAFILE
250 CWD Command successful
ftp> dir
ftp> bin
200 Type set to I.
ftp> hash
Hash mark printing on (1024 bytes/hash mark).
#############################################################################################
#############################################################################################
#############################################################################################
#############################################################################################
#############################################################################################
#############################################################################################
#############################################################################################
#############################################################################################
226 BIN Transfer Complete
ftp> bye
221 QUIT Goodbye.
[root@devserver u03]# ls -l
total 102528
-rw-r--r-- 1 root root 104865792 Feb 19 03:35 EXAMPLE.265.904167107