VXVM Lab
VXVM Lab
VXVM Lab
Objective:
Learn how to integrate, operate, VERITAS Storage Foundation in a Solaris environment. This training
provides instruction on operational management procedures for VERITAS Volume Manager (VxVM) and
you will learn how to install and configure VERITAS Volume Manager and how to manage disks, disk
groups, and volumes by using from the command line.For your kind information In this training , I am
not going to explain the theory in depth.
Instead of spending money on training courses , you can learn vxvm yourself by reading this blog and if
you have any doubt ,please leave a comment. I will get back to you as soon as possible.
Prerequisites:
1.Skills:Knowledge of UNIX system administration
2.Lab: VM Solaris 10 or physical machine with solaris 10,vxvm 5.0 and above version software.
VXVM objects:
There are several Volume Manager objects that must be understood before you can use the Volume
Manager to perform disk management tasks:
Physical objects:
1.Physical disks – Its a physical disk or LUN from storage.
Virtual objects:
VM disks
A VM disk is a contiguous area of disk space from which the Volume Manager allocates storage.Its
nothing but a public region of disk.
Disk groups
A disk group is a collection of VM disks that share a common configuration.
Subdisks
A subdisk is a set of contiguous disk blocks.A VM disk can be divided into one or more subdisks.
Plexes
The Volume Manager uses subdisks to build virtual entities called plexes.A plex consists of one or more
subdisks located on one or more disks
Volumes
A volume is a virtual disk device that appears to applications, databases, and file systems like a
physical disk partition, but does not have the physical limitations of a physical disk partition.
In this tutorial,we are going to cover the below topics.In the end of these tutorials you will
very familiar with veritas volume manager and its operation.Your suggestions are always welcome
to improve these tutorials.
Download the packages from Symantec and keep the package under /var/tmp.
bash-3.00# ls -lrt
bash-3.00# cd dvd2-sol_x64
total 95
Task Menu:
Do you agree with the terms of the End User License Agreement as specified in
the
storage_foundation_high_availability/EULA/en/EULA_SFHA_Ux_6.0.pdf file
present on media? [y,n,q,?] y
Enter the Solaris x64 system names separated by spaces: [q,?] node1
node1
1) SF Standard HA
2) SF Enterprise HA
VxVM vxdctl ERROR V-5-1-1589 enable failed: Volboot file not loaded
bash-3.00# vxconfigd
Now you can see that init command created volboot for SF.
hostid node1
hostguid {b6208f08-0d8c-11e2-8046-000c2985ec00}.
Once the VXVM installation is completed ,you good to start work on the below things.
VxVM presents the disks in a disk array as volumes to the operating system
in below manner.
To avoid confusion with OS based names(i.e format),please check it,which is set in your system.
bash-3.00# vxddladm get namingscheme
NAMING_SCHEME PERSISTENCE LOWERCASE USE_AVID
======================================================
Enclosure Based Yes Yes Yes
bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
disk_2 auto:ZFS - - ZFS
disk_3 auto:ZFS - - ZFS
As per the above output,the system was set to use enclosure based naming scheme. Anytime you can
change the naming scheme on the fly.There will be no impact by doing this.
You can match the OS native names(i.e format) with VXVM enclose based names using the below
command.
After scanning the disk in OS,Scan the disks in veritas level by using the below command.
Note:In older version we may used “vxdctl enable” or “vxconfigd -k” to scan new disks by restarting it
vxvm daemon.Symantec recommending to use the above command to scan new disks.
vxdisksetup command brings the disks in to veritas volume manager control.By default this will format
the disk in CDS(Cross Data platform Sharing ) .Other formats are sliced and simple. sliced will be used
for boot disks.
bash-3.00# /etc/vx/bin/vxdisksetup -i disk_0
bash-3.00# /etc/vx/bin/vxdisksetup -i disk_1
bash-3.00# /etc/vx/bin/vxdisksetup -i disk_4
bash-3.00# /etc/vx/bin/vxdisksetup -i disk_5
bash-3.00# /etc/vx/bin/vxdisksetup -i disk_6
If you want to setup the disk in specific format,you can use the below syntax.
By Checking the disk status,you will come to know how its formated and disk
status.
Diskgroup Operations
The vxdg utility performs various diskgroup operations that includes the creation of disk
groups, addition of disks in to diskgroup,removing disk from diskgroup.This command performs disk
group imports and deports as well.Here we are going to see how create a new diskgroup and adding
disks in existing diskgroup.In the end of the article we will see how to backup the diskgroup
configuration.
In the below output you can see, we have five disks which are in VXVM control. We have brought the
disk in to vxvm control using vxdisksetup.
uarena#vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk - - online
disk_1 auto:cdsdisk - - online
disk_2 auto:ZFS - - ZFS
disk_3 auto:ZFS - - ZFS
disk_4 auto:cdsdisk - - online
disk_5 auto:cdsdisk - - online
disk_6 auto:cdsdisk - - online
Now you can see we have created new diskgroup UXDG using disk_0 & disk_1 and we have assigned the
meaning full name to disks.
Task:2-Addition of disk:
If you want to add the new disk in to existing diskgroup, bring the disk in to vxvm control using
vxdisksetup and add it using below command.
uarena#vxdg -g UXDG adddisk UXDISK3=disk_4
uarena#vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk UXDISK1 UXDG online
disk_1 auto:cdsdisk UXDISK2 UXDG online
disk_2 auto:ZFS - - ZFS
disk_3 auto:ZFS - - ZFS
disk_4 auto:cdsdisk UXDISK3 UXDG online
disk_5 auto:cdsdisk - - online
disk_6 auto:cdsdisk - - online
Task:3-Removing the disk:
You can remove the disk any time from DG if it’s not used in any volume.
uarena#vxdg -g UXDG rmdisk UXDISK3
uarena#vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk UXDISK1 UXDG online
disk_1 auto:cdsdisk UXDISK2 UXDG online
disk_2 auto:ZFS - - ZFS
disk_3 auto:ZFS - - ZFS
disk_4 auto:cdsdisk - - online
disk_5 auto:cdsdisk - - online
disk_6 auto:cdsdisk - - online
Task:5-Deporting diskgroup:
After un-mounting the volumes , you can deport the diskgroup.To see the imported diskgroup
uarena#vxdg list
NAME STATE ID
UXDG enabled,cds 1364022395.37.sfos
uarena#vxdg deport UXDG
By default vxdisk output will not show the deported diskgroup tag.
uarena#vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk - - online
disk_1 auto:cdsdisk - - online
disk_2 auto:ZFS - - ZFS
disk_3 auto:ZFS - - ZFS
disk_4 auto:cdsdisk - - online
disk_5 auto:cdsdisk - - online
disk_6 auto:cdsdisk - - online
Sometimes you may need to use -C flag to import the cluster diskgroup to clear the VCS lock.
Since there is no configuration chnages VXVM refuse to take backup.Here i am doing small
changes.
# /etc/vx/bin/vxresize -g UXDG oravol1 +20M
# /etc/vx/bin/vxconfigbackup
Start backing up diskgroup UXDG to
/etc/vx/cbr/bk/UXDG.1364474089.57.sfos ...
VxVM NOTICE V-5-2-3100 Backup complete for diskgroup: UXDG
Volume Operations
VxVM builds volume using virtual objects of VM disks, disk groups, subdisks, and plexes.These virtual
objects can organized easily using vxassist command to create new volume.Here we are going to see
about how to create a new volumes with different layout and volume redundancy.In the end of the
article we see how to destroy the volume in details. Here the assumtion is we have configured
diskgroup with the name of UXDG.
bash-3.00# vxdg list
NAME STATE ID
UXDG enabled,cds 1364022395.37.sfos
Syntax:
#vxassist -g diskgroup make vol_name size layout=format disk-name
v - volume - uxoravol1
pl - Plex - uxoravol1-01
sd - Subdisk - UXDISK1-01
As per the above output,subdisk(UXDISK1-01) has been created using UXDISK1 and plex(uxoravol1-01) is
sitting on top of subdisk. On top of the plex virtual layer, volume is placed. These virtual layer provides
more flexible on volume level operation.
Once you have created the volume,you can use mkfs to create vxfs filesystem.To create VXFS
filesystem
# mkdir /uxoravol1
# mount -F vxfs /dev/vx/dsk/UXDG/uxoravol1 /uxoravol1
# df -h /uxoravol1
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/UXDG/uxoravol1 50M 3.1M 44M 7% /uxoravol1
In mirror volume, data is written identically in to two plexs which were formed with two or more
subdisks.
Note:
# vxprint -hvt
Disk group: UXDG
2.Removing volume.
Volume Resize:
VXVM is very flexible and reliable volume manager to resize volumes without un-mounting
it. we can re-size the volumes using vxassist and resize filesystem using fsadm.vxresize
command will perform both volume and filesystem re-size operation simultaneously.
Here we are going to see how to re-size the volume smvol in two different ways without un-
mounting it .
# df -h /smvol
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/UXDG/smvol 50M 3.1M 44M 7% /smvol
Step:1
Determining how much space we can increase the volume.
Step:2
Re-size the volume using vxassist.
Step:3
# df -h /smvol
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/UXDG/smvol 60M 3.1M 53M 6% /smvol
Step:2
Resize the volume & filesystem:
You can see the changes
# /etc/vx/bin/vxresize -g UXDG smvol +10M
# df -h /smvol
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/UXDG/smvol 70M 3.1M 63M 5% /smvol
Step:2
Reduce the volume using vxassist.You will lose data if you resize the volume less than
filesystem size. So better to shrink the volume with some extra space.Here instead
of shrink to 50M , i am shrinking to 51MB.
# vxassist -g UXDG shrinkto smvol 51M
VxVM vxassist ERROR V-5-1-7236 Shrinking a FSGEN or RAID5 usage type volume
can result in loss of data. It is recommended to use the "vxresize"command or
specify "-f" option to force the operation.
Note:You will above warning messages if you used without -f option.Handle with care.
# vxassist -g UXDG -f shrinkto smvol 51M
or
# vxassist -g UXDG -f shrinkby smvol 10M
# vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
170 - ATCOPY/R 60.00% 0/102400/61440 PLXATT uxoravol1 uxoravol1-02 UXDG auto-
throttled
bash-3.00# vxtask monitor
175 - ATCOPY/R 92.00% 0/102400/94208 PLXATT uxoravol1 uxoravol1-02 UXDG auto-
throttled
175 - ATCOPY/R 96.00% 0/102400/98304 PLXATT uxoravol1 uxoravol1-02 UXDG auto-
throttled
175 - ATCOPY/R 100.00% 0/102400/102400 PLXATT uxoravol1 uxoravol1-02 UXDG
auto-throttled
Step:2– Once the convert has been done,then use vxassist convert to convert the volume
to mirror-stripe.
# vxassist -g UXDG convert concatv1 layout=mirror-stripe
bash-3.00# vxprint -hvt
Disk group: UXDG
v concatv1 - ENABLED ACTIVE 102400 SELECT - fsgen
pl concatv1-02 concatv1 ENABLED ACTIVE 102400 STRIPE 2/128 RW
sd UXDISK1-01 concatv1-02 UXDISK1 0 51200 0/0 disk_0 ENA
sd UXDISK2-02 concatv1-02 UXDISK2 0 51200 1/0 disk_1 ENA
pl concatv1-03 concatv1 ENABLED ACTIVE 102400 STRIPE 2/128 RW
sd UXDISK3-02 concatv1-03 UXDISK3 0 51200 0/0 disk_4 ENA
sd UXDISK4-02 concatv1-03 UXDISK4 0 51200 1/0 disk_5 ENA
# vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
182 - RELAYOUT/R 49.97% 0/204800/102336 RELAYOUT concatv1 UXDG
# vxtask monitor
TASKID PTID TYPE/STATE PCT PROGRESS
182 - RELAYOUT/R 57.97% 0/204800/118720 RELAYOUT concatv1 UXDG
182 - RELAYOUT/R 58.97% 0/204800/120768 RELAYOUT concatv1 UXDG
182 - RELAYOUT/R 59.97% 0/204800/122816 RELAYOUT concatv1 UXDG
Method:1
Since the plex is attached to the volume, you can’t delete it.First disassociate from the
volume using the below command.
Method:2
In other way you can delete the plex using single command,
# vxplex -g UXDG -o rm dis uxvol2-02
# vxprint -hvt
Disk group: UXDG
v uxoravol1 - ENABLED ACTIVE 102400 SELECT - fsgen
pl uxoravol1-01 uxoravol1 ENABLED ACTIVE 102400 CONCAT - RW
sd UXDISK1-01 uxoravol1-01 UXDISK1 0 102400 0 disk_0 ENA
bash-3.00# df -h /uxvol2
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/UXDG/uxvol2 50M 3.1M 44M 7% /uxvol2
Command Syntax:
In similar way you can covert other existing volume layouts to new volume layout. Refer
storage foundation admin guide to find Permitted relayout transformations.
Veritas Dynamic Multi-Pathing provides greater availability, reliability to SAN paths and it
increase the SAN I/O using load balancing .SAN path redundancy ensured by path failover.
Active/Active (A/A)
Active/Passive (A/P)
As per the above output,we have two controllers for emc0 enclosure.
Here you can see ,we have two enclosure are available. emc0 (SAN) & disk(local disks)
To find what are Disk are using controller c1,
bash-3.00# vxdmpadm getsubpaths ctlr=c1
NAME STATE[A] PATH-TYPE[M] DMPNODENAME ENCLR-TYPE ENCLR-NAMEATTRS
======================================================
c1t3d0 ENABLED(A) - disk_0 Disk disk -
c1t5d0 ENABLED(A) - disk_1 Disk disk -
c1t4d0 ENABLED(A) - disk_2 Disk disk -
To find who controls the path,i.e to find the enclosure based name.
# vxdmpadm getdmpnode nodename=c2t3000034578233D18d57s2
NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME
================================================================
emc0_0e790 ENABLED EMC 2 4 0 emc0
To see what are the disks are coming from emc0 enclosure,
# vxdmpadm getdmpnode enclosure=emc0
NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME
===============================================================
emc0_0e790 ENABLED EMC 2 4 0 emc0
To disable controller,
# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
===========================================================
c1 Disk ENABLED disk
c2 EMC ENABLED emc0
c4 EMC ENABLED emc0
# vxdmpadm disable ctlr=c2
To enable controller,
adaptive
adaptiveminq
balanced
minimumq
priority
round-robin
singleactive
To stop DMP,
# vxdmpadm stop restore
To start DMP,
# vxdmpadm start restore
A snapshot is the state of a volume at a particular point in time. Veritas Volume Manager
snapshot capability for taking an image of a volume at a given point in time.It provides
various snapshot depends on the environments and product cost.
Comparison of snapshot.
table.tableizer-table { border: 1px solid #CCC; font-family: Verdana, Verdana, Geneva, sans-serif; font-size: 12px;
} .tableizer-table td { padding: 4px; margin: 3px; border: 1px solid #ccc; } .tableizer-table th { background-color:
#104E8B; color: #FFF; font-weight: bold; }
Full-sized Space-optimized Mirror Break-
Snapshot feature
instant instant off
Thanks to www.symantec.com
3.Prepare snap-volume:
You will get the below error if you didn’t prepare the snap volume.
VxVM vxassist ERROR V-5-1-7061 Volume oravol1-snap is not instant ready
# vxsnap -g UXDG -b prepare oravol1-snap
v oravol1-snap - ENABLED ACTIVE 20480 SELECT - fsgen
pl oravol1-snap-01 oravol1-snap ENABLED ACTIVE 20480 CONCAT - RW
sd UXDISK3-01 oravol1-snap-01 UXDISK3 0 20480 0 disk_4 ENA
dc oravol1-snap_dco oravol1-snap oravol1-snap_dcl
v oravol1-snap_dcl - ENABLED ACTIVE 67840 SELECT - gen
pl oravol1-snap_dcl-01 oravol1-snap_dcl ENABLED ACTIVE 67840 CONCAT - RW
sd UXDISK3-02 oravol1-snap_dcl-01 UXDISK3 20480 67840 0 disk_4 ENA
You can see still,snapshot is holding the data with last refresh state.
# df -h /snaporavol1/
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/UXDG/oravol1-snap 100M 43M 53M 45% /snaporavol1
Now you can see snapshot has been updated with last volume update.
# df -h /snaporavol1/
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/UXDG/oravol1-snap 100M 63M 35M 65% /snaporavol1
# ls -lrt /snaporavol1/snapshot-test1
-rw------T 1root root 20971520 Apr1 15:09/snaporavol1/snapshot-test1
In this way you can mount the snap volume and take a backup without impact the volume
performance for database filesystems if you are performing the backup on production
server.
Once you split the snapshot volume to new diskgroup,you can’t update the snapshot.
# vxsnap -g UXDG refresh oravol1-snap source=oravol1
VxVM vxsnap ERROR V-5-1-7015 Volume oravol1-snap doesn't exist
# cd /apporavol1/
# rm *
# ls -lrt
#
Trying to restore snapshot:
# vxsnap -g UXDG restore oravol1 source=oravol1-snap
VxVM vxsnap ERROR V-5-1-7067 Volume oravol1 is open, cannot restore
# umount /apporavol1/
bash-3.00# vxsnap -g UXDG restore oravol1 source=oravol1-snap
Ex:Volume size is increasing 100M/per hour,then you need minimum 100MB cache object.If
the backJob needs two hours to complete,then you need 100×2=200MB cache volume.
Note:Since we have created the cache volume with size of 10M,if the volume changes
become more than 10M,snapshot become invalid and you need to perform the snapshot
again (Step:2)
5.Detach/Attach snapshot:
Detach the snapshot from volume:
# vxsnap -g UXDG dis snap-oravol1
Attach the space optimized snapshot to volume.You can not use the reattach command for
space optimized snapshot.
# vxsnap -g UXDG reattach snap-oravol1 source=oravol1
VxVM vxplex ERROR V-5-1-6390 Cannot reattach space optimized snapshot to a
volume
If you get the below error ,possible cache object has been stopped,
# vxedit -g UXDG -rf rm snap-oravol1
VxVM vxedit ERROR V-5-1-10128 Cache object meta-data update error
Start the cache object again to remove the snapshot.
# vxcache -g UXDG start cacheobj
we are going to see traditional mirror break-off snapshot.To perform this snapshot ,we need
free space equal to volume size on the diskgroup.It typically perform the volume mirror and
it will make the new plex as snapshot volume for backup operations.
High Level plan for backup of database volume using third-mirror break-off:
1.Prepare the volume for snapshot.
2.Add a mirror using the vxassist or vxsnap
3.Make the snapshot and break it from volume.
4.Mount the volume and perform the backup.
USING VXASSIST
Note:If you have additional mirror configured with volume,you can use that
plex as snapshot using below command.
# vxplex -g DG_NAME convert state=SNAPDONE plex_name
2.To take the snapshot
This step will break the mirror plex in to separate volume.So that we can mount it in
different mount point and backup can be performed without touching the actual database
volume. (oravol1)
This process will overwrite the volume using the snapshot data.
USING VXSNAP:
The above process can be done by using vxsnap command as well.
1.Prepare the volume for snapshot
# vxsnap -g UXDG prepare oravol1 ndcomirs=2 drl=off
bash-3.00# vxprint -hvt
Disk group: UXDG
v oravol1 - ENABLED ACTIVE 204800 SELECT -
fsgen
pl oravol1-01 oravol1 ENABLED ACTIVE 204800 CONCAT - RW
sd UXDISK1-01 oravol1-01 UXDISK1 0 143056 0 disk_0
ENA
sd UXDISK2-01 oravol1-01 UXDISK2 0 61744 143056 disk_1
ENA
dc oravol1_dco oravol1 oravol1_dcl
v oravol1_dcl - ENABLED ACTIVE 67840 SELECT -
gen
pl oravol1_dcl-01 oravol1_dcl ENABLED ACTIVE 67840 CONCAT - RW
sd UXDISK3-01 oravol1_dcl-01 UXDISK3 0 67840 0 disk_4
ENA
pl oravol1_dcl-02 oravol1_dcl ENABLED ACTIVE 67840 CONCAT - RW
sd UXDISK4-01 oravol1_dcl-02 UXDISK4 0 67840 0 disk_5
ENA