SAN Persistent Binding and Multipathing in the 2.

6 Kernel
Michelle Butler, Technical Program Manager Andy Loftus, System Engineer Storage Enabling Technologies NCSA [email protected] or [email protected]

Slides available at http://dims.ncsa.uiuc.edu/set/san/

LCI Conference 2007

National Center for Supercomputing Applications

Who?
NCSA
a unit of the University of Illinois at Urbana-Champaign a federal, state, university, and industry funded center

Academic Users
NSF peer review

Large amount of applications/user needs

3rd party codes, user written All running on same environment

Many research areas

LCI Conference 2007
National Center for Supercomputing Applications

NCSAs 1st Dell Cluster

Tungsten: 1750 server cluster
3.2 GHz Xeon
2,560 processors (compute only) 16.4 TF; 3.8 TB RAM;122 TB disk Dell OpenManage

st fir ale e Th e-sc er!!! rg lust la l c l De

Production date: April 2004 User Environment
Platform Computing LSF Softenv Intel Compilers ChaMPIon Pro, MPICH, VMI-2

Myrinet
Full bi-section

Lustre over Gig-E

13 DataDirect 8500 104 OSTs, 2 MDS w/separate disk 11.1 GB/sec sustained

Power/Cooling
593 KW / 193 tons
LCI Conference 2007

National Center for Supercomputing Applications

NCSAs 3rd Dell Cluster

T2 retired into: Tungsten-3 1955 blade cluster
2.6 GHz Woodcrest Dual Core
1,040 processors/2080 cores 22 TF; 4.1 TB RAM; 20 TB disk Warewulf

Cisco InfiniBand
3 to 1 over-subscribed OFED-1.1 w/ HPSM subnet manager

Lustre over IB
4 FasT controllers direct FC 1.2GB/s sustained 8 OSTs and 2 MDS w/complete auto failovers

Production date: March 2007 User Environment

Torque/Moab Softenv Intel Compilers VMI-2

Power/Cooling
148 KW / 42 tons

LCI Conference 2007

National Center for Supercomputing Applications

NCSAs 4th Dell Cluster

Abe: 1955 blade cluster
2.33 GHz Cloverton Quad-Core
1,200 blades/9,600 cores 89.5 TF; 9.6 TB RAM; 120 TB disk Perceus management; diskless boot

Cisco Infiniband
2 to 1 oversubscribed OFED-1.1 w/ HPSM subnet manager

t ges ! lar ter!! e Th l clus l De

Production date: May 2007 (anticipated) User Environment
Torque/Moab Sofenv Intel Compiler MPI: evaluating Intel MPI, MPICH, MVAPICH, VMI-2, etc.

Lustre over IB
22 OSTs 2 9500 DDN controllers direct FC 10 FasT controllers on SAN fabric 8.4GB/s sustained 22 OSTs and 2 MDS w/complete auto failovers

Power/Cooling
500 KW / 140 tons

LCI Conference 2007

National Center for Supercomputing Applications

NCSA Facility - ACB

Advanced Computation Building
Three rooms, totals:
16,400 sqft raised floor 4.5 MW power capacity 250 kW UPS 1,500 tons cooling capacity

Room 200:
7,000 sqft no columns 70 raised floor 2.3 MW power capacity 750 tons cooling capacity

LCI Conference 2007

National Center for Supercomputing Applications

NCSAs Other Systems

Distributed Memory Clusters
Mercury (IBM, 1.3/1.5 GHz Itanium2):
1,846 processors 10 TF; 4.6 TB RAM; 90 TB disk

Shared Memory Clusters

Copper (IBM p690,1.3 GHz Power4): 12 x 32 processors
2 TF; 64 or 256 GB RAM each; 35 TB disk

Cobalt (SGI Altix, 1.5 GHz Itanium2): 2 x 512 processors

6.6 TF; 1 TB or 3 TB RAM; 250 TB disk
LCI Conference 2007
National Center for Supercomputing Applications

NCSA Storage Systems

Archival: SGI/Unitree (5 PB total capacity)
72TB disk cache; 50 tape drives currently 2.8PB of data in MSS
>1PB ingested in last 6 months project ~3.2PB by end of CY2006 licensed to support 5PB resident data

~30 data collections hosted

Infrastructure: 394TB Fiberchannel SAN connected

Fiberchannel SAN connected; FC and SATA environments Lustre, IBRIX, NFS filesystems

Databases:
8 processor 12GB memory SGI Altix
30TB of SAN storage Oracle 10G, mysql, Postgres

Oracle RAC cluster Single-system Oracle deployments for focused projects

LCI Conference 2007
National Center for Supercomputing Applications

Visualization Resources
30M-pixel Tiled Display Wall
8192 x 3840 pixels composite display 40 NEC VT540 projectors, arranged in a 5H x 8W matrix driven by 40-node Linux cluster
dual-processor 2.4GHz Intel Xeons with NVIDIA FX 5800 Ultra graphics accelerator cards Myrinet interconnect to be upgrade by early CY2007
funded by State of Illinois

SGI Prisms
8 x 8 processor (1.6 GHz Itanium2) 4 graphics pipes each; 1 GB RAM each InfiniBand connection to Altix machines
LCI Conference 2007
National Center for Supercomputing Applications

SAN at NCSA
1.3PB spinning disk
895TB SAN attached

1392 Brocade switch ports 7 SAN fabrics 2 data centers

LCI Conference 2007

National Center for Supercomputing Applications

10

Persistent Binding
Device naming problems Udev solution Examples Interactive Demo

LCI Conference 2007

National Center for Supercomputing Applications

11

Device Naming Problem

Before After

Add hardware SAN zoning New SAN luns Modify config

LCI Conference 2007

National Center for Supercomputing Applications

Device node mapping can change with changes to - hardware - software - SAN Devices assigned random names (based on next available major/minor pair for device type) CLUSTER - Multiple hosts that see the same disk will assign the disk to different device nodes - may be /dev/sda on system1 but /dev/sdc on system2 - Can change with hardware changes; what used to be /dev/sda is not /dev/sdc Devfs helps only a little: - Fixes device naming; on a single host, disk will always have the same device node - But different hosts may have different device names for the same physical disk

12

What needs to happen

Storage target always maps to same local device (ie. /dev/) Local device name should be meaningful
/dev/sda conveys no information about the storage device

LCI Conference 2007

National Center for Supercomputing Applications

13

udev - Persistent Device Naming

Udev is a userspace solution for a dynamic /dev directory, with persistent device naming *
Userspace: not required to remain in memory Dynamic: /dev not filled with unused files Persistent: devices always accessable using the same device node

Provides for custom device names

* Daniel Drake (http://www.reactivated.net/writing_udev_rules.html)

LCI Conference 2007

National Center for Supercomputing Applications

Devfs provides dynamic and persistent naming, but: - kernel based - entire device db stored in kernel memory, never swapped - not possible to customize device names UDEV CUSTOM - custom names for devices - custom scripts can be run when specifice devices attached/removed

14

Setting up udev device mapper

Overview 1. Uniquely identify each lun 2. Assign a meaningful name to each lun

LCI Conference 2007

National Center for Supercomputing Applications

15

1. Uniquely identify each lun

/sbin/scsi_id
device name

scsi_id
Unique id

SCSI INQUIRY

Sample usage:
root# scsi_id -g -u -s /block/sda SSEAGATE_ST318406LC_____3FE27FZP000073302G5W root# scsi_id -g -u -s /block/sdb 3600a0b8000122c6d00000000453174fc
LCI Conference 2007
National Center for Supercomputing Applications

/sbin/scsi_id - INPUT: existing local device name - OUTPUT: string that uniquely identifies the specific device (guaranteed unique among all scsi devices) SAMPLE: - sda: locally installed drive - sdb: SAN attached disk

16

2. Associate a meaningful name

New udev rules le: /etc/udev/rules.d/20-local.rules
BUS="scsi", SYSFS{vendor}="DDN", SYSFS{model}="S2A 8000", PROGRAM="/sbin/scsi_id -g -u -s /block/%k ", RESULT="360001ff020021101092fadc32a450100", NAME="disk/fc/sdd4c1l0"

BUS=scsi
/sys/bus/scsi

SYSFS
<BUS>/devices/H:B:T:L/<filename>

PROGRAM & RESULT

Program to invoke and result to look for

NAME
Device name to create (relative to /dev)
LCI Conference 2007
National Center for Supercomputing Applications

Custom naming controlled by rulesets stored in /etc/udev/rules.d A rule is a lists of keys to match against. When all keys match, the specified action is taken (create a device name or symlink)

17

Example: Customizing for multiple paths

Problem Multiple paths to a single lun results in multiple device nodes. Need to know which path each device uses.

LCI Conference 2007

National Center for Supercomputing Applications

18

Example: Customizing for multiple paths

Custom script : mpio_scsi_id
WWPN device name scsi_id

Disk Ctlr

udev
WWPN + scsi_id

mpio_scsi_id

Sample udev rule:

BUS="scsi", SYSFS{vendor}="DDN", SYSFS{model}="S2A 8000",

PROGRAM="/root/bin/mpio_scsi_id %k", RESULT="23000001ff03092f360001ff020021101092fadc32a450100",

NAME="disk/fc/sdd4c1l0"

LCI Conference 2007

National Center for Supercomputing Applications

Get disk controller WWPN (Emulex) /sys/class/fc_transport/target<H>:<B>:<T>/port_name (QLA) grep + awk to pull value from /proc/scsi/ql2xxx/<host_id>

19

Demo: udev persistent device naming

Single HBA Single disk unit
4 luns Each lun presented through both controllers

Host sees 8 logical luns Use mpio_scsi_id to identify the ctlr-lun

LCI Conference 2007

National Center for Supercomputing Applications

20

Demo: udev persistent device naming

Original Configuation udev config file
/etc/udev/udev.conf

Custom device names Custom rules file

20-local.rules

scsi_id config file

/etc/scsi_id.config

Restart udev
udevstart

Scan fc luns
{sysfs}/hostX/scan /dev/disk/by-id

Custom device names created

/dev/disk/fc

LCI Conference 2007

National Center for Supercomputing Applications

BEGIN - tail -f /var/log/messages 1. 2. 3. 4. 5. 6. 7. 8. Enable udev logging Enable scsi_id for all devices (options -g) /proc/partitions Scan fc luns (echo - - - > /sys/class/scsi_host/hostX/scan) See udev log lines in messages file ; See fc disks in /dev/disk/by-id Enable 20-local rules file Udevstart See udev log lines in messages file ; See fc disks in /dev/disk/fc

DEFAULT CONFIGURATION Local rules file already exists. Disable it. Default behavior for scsi_id is to blacklist everything unknown (-b option). Enable white list everything (g option) so scsi_ids will be returned. Even before custom rules are in place, see default udev rule selection activity in /var/log/messages After running delete_fc_luns, udev removes /dev/sdX devices files (/var/log/messages) CUSTOM CONFIGURATION Udev custom rules are selected (see /var/log/messages) Major/Minor numbers line up for /dev/disk/fc/* and /proc/partition/*

21

Demo: udev persistent device naming

Debugging Not all sysfs files are available immediately
HBA target WWPN Add udevstart to boot scripts

Udev tools can help

udevinfo udevtest

Examples
udevinfo -a -p $(udevinfo -q path -n /dev/sdb) udevtest /block/sdb

LCI Conference 2007

National Center for Supercomputing Applications

Exmaple: multiple paths on Nadir - If luns are removed (delete_fc_luns) - Then added (scan_fc_luns) - No matches are found in 20-local.rules - Add syslog output to mpio_scsi_id + Shows params the script is called with + Shows what the script returns + target_wwpn is not getting set - Run udevstart (luns already attached now), matches found in 20-local.rules and device files created Probably either a driver or udev issue. Easiest solution is to run scan_luns and udevstart at system boot time (/etc/rc.d/rc.local)

22

Custom script: ls_fc_luns

Get HBA list Get HBA type Get target list Get lun list Get lun info sysfs lspci sysfs (emulex) /proc (QLA) sysfs sysfs /sys/class/scsi_host/hostX/targetX:Y:Z /proc/scsi/qla2xxx/X /sys/class/scsi_host/hostX/targetX:Y:Z/X:Y:Z:L /sys/class/scsi_host/hostX/targetX:Y:Z/X:Y:Z:L/* /sys/class/fc_host

0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4

0x200300a0b8122c6e 0x200300a0b8122c6e 0x200200a0b8122c6e 0x200200a0b8122c6e

2:0:0:0 2:0:0:1 2:0:1:0 2:0:1:1

sdb sdc sdi sdj

3600a0b8000122c6d00000000453174fc 3600a0b80000fd6320000000045317563 3600a0b8000122c6d00000000453174fc 3600a0b80000fd6320000000045317563

LCI Conference 2007

National Center for Supercomputing Applications

23

Custom script: lip_fc_hosts

Get host list ls_fc_luns

echo 1 > /sys/class/fc_host/hostX/lip

LCI Conference 2007

National Center for Supercomputing Applications

24

Custom script: scan_fc_luns

Get host list ls_fc_luns

echo - - - > /sys/class/scsi_host/hostX/scan

LCI Conference 2007

National Center for Supercomputing Applications

25

Custom script: delete_fc_luns

Get lun list ls_fc_luns

echo 1 > /sys/class/scsi_host/hostX/targetX:Y:Z/X:Y:Z:L/delete

LCI Conference 2007

National Center for Supercomputing Applications

26

udev - Additional Resources

man udev
http://www.emulex.com/white/hba/wp_linux26udev.pdf
Excellent white paper

http://www.reactivated.net/udevrules.php
How to write udev rules

http://www.us.kernel.org/pub/linux/utils/kernel/hotplug/ udev.html
Information and links

http://dims.ncsa.uiuc.edu/set/san
FC tools : custom tools used in demo

LCI Conference 2007

National Center for Supercomputing Applications

27

Linux Multipath I/O

Overview History Setup Demos
Active / Passive Controller Pair Active / Active Controller Pair

LCI Conference 2007

National Center for Supercomputing Applications

28

Linux Multipath - History

Providers Storage Vendor HBA Vendor Filesystem OS

LCI Conference 2007

National Center for Supercomputing Applications

STORAGE VENDOR - End to end solution (they provide disk, HBA, driver, addl software, sometimes even FC switch) - HBAs (and other parts) come at a markup - One location for support tickets, but no alternate recourse if they cant fix the problem - Proprietary requirements (typically require 2 HBAs, only works with their systems) HBA VENDOR - QLA > Linux support spotty + 2.4 kernel ok, but strict requirements (2 HBAs, exactly 2 paths per lun, active/active controllers) + 2.6 kernel inconsistent behavior > Solaris support spotty (2 months to get 1 machine working, next month stops working, machine was untouched) > Dropped Windows support prematurely (Windows MPIO layer not complete yet, only an API for vendors) > Proprietary solution, only works with their HBAs and configuration software - Emulex (unix philosophy, do one thing and do it well; MPIO doesnt belong in the driver) FILESYSTEM - 3rd party - Veritos, others?? - Parallel Filesystems - Ibrix, Lustre, GPFS, CXFS (enable MPIO via failover hosts) OS - *NEW* Solaris 10 (XPATH, but requires Solaris branded QLA cards) - *NEW* Linux (device mapper multipath) (RedHat4, Suse, others)

29

Device Mapper Multipath

Identify luns by scsi_id Create path groups
Round-robin I/O on all paths in groups

Monitor paths for failure

When no paths left in current group, use next group

Monitor failed paths for recovery

Upon path recovery, recheck group priorities Assign new active group if necessary
LCI Conference 2007
National Center for Supercomputing Applications

30

Linux Device Mapper Multipath

Overview 1. Identify unique luns 2. Monitor active paths for failure 3. Monitor failed paths for recovery

LCI Conference 2007

National Center for Supercomputing Applications

Multipath handles 3 areas. All settings are saved in /etc/multipath.conf

31

1. Identify unique luns

Storage Device vendor product getuid_callout
device { vendor product getuid_callout }

"DDN" "S2A 8000" "/sbin/scsi_id -g -u -s /block/%n"

LCI Conference 2007

National Center for Supercomputing Applications

32

1. Identify unique luns

Multipath Device wwid alias

multipath { wwid 360001ff020021101092fb1152a450900 alias sdd4l0 }

LCI Conference 2007

National Center for Supercomputing Applications

33

2. Monitor Healthy Paths for Failure

Priority group
Collection of paths to the same physical lun I/O is split across all paths in round-robin fashion

path_grouping_policy
multibus failover group_by_prio group_by_serial group_by_node

LCI Conference 2007

National Center for Supercomputing Applications

Multipath control creates priority groups. Paths are grouped based on path_grouping_policy MULTIBUS - all paths in one priority group (DDN) (no penalty to access luns via alternate controllers) FAILOVER - one path per priority group (Use only 1 path at a time) (typically only 1 usable path, such as IBM fastt with AVT disabled) GROUP_BY_PRIO - Paths with same priority in same priority group, 1 group for each unique priority (Priorities assigned by external program) GROUP_BY_SERIAL - Paths grouped by scsi target serial (controller node WWN) GROUP_BY_NODE - (I have not tested or researched this, never had a need to)

34

2. Monitor Healthy Paths for Failure

Path Grouping Policy = group_by_prio
Path Priority
Integer value assigned to a path Higher value == higher priority Directly controls priority group selection

prio_callout
3rd party pgm to assign priority values to each path

multipath
Integer value Device name

prio_callout

LCI Conference 2007

National Center for Supercomputing Applications

Only matters if using group_by_prio grouping policy DIRECTLY CONTROLS PRIORITY GROUP SELECTION - Priority group with highest value is active group PREVIOUS SLIDE - When all paths in a group are failed, next group becomes active. That would be the priority group with the next highest priority value that has an active path. PRIO_CALLOUT - Provided by vendor or (more typically) custom script written by admin for specific setup - If not using group_by_prio, then set this to /bin/true

35

2. Monitor Healthy Paths for Failure

path_checker
tur readsector0 directio (Custom)
emc_clarion hp_sw

no_path_retry
queue (N > 0) fail

LCI Conference 2007

National Center for Supercomputing Applications

TUR - SCSI Test Unit Ready - Preferred if lun supports it (OK on DDN, IBM fastt) - Does not cause AVT on IBM fastt - Does not fill up /var/log/messages on failures READSECTOR0 - physical lun access via /dev/sdX (IS THIS CORRECT???) DIRECTIO - physical lun access via /dev/sgY (IS THIS CORRECT???) Both readsector0 and directio cause AVT on IBM fastt, resulting in lun thrashing Both readsector0 and directio log fail messages in /var/log/messages (could be useful if you want to monitor logs for these events) NO_PATH_RETRY - # of retries before failing path - queue: queue I/O forever - (N > 0): queue I/O for N retries, then fail - fail: fail immediately

36

3. Monitor failed paths for recovery

Failback
Immediate (same as n=0) (n > 0) manual

LCI Conference 2007

National Center for Supercomputing Applications

FAILBACK - When a path recovers, wait # seconds before enabling the path - Recovered path is added back into multipath enabled path list - multipath re-evaluates priority groups, changes active priority group if needed MANUAL RECOVERY - User runs /sbin/multipath to update enabled paths and priority groups

37

Putting it all togehter

multipaths { multipath { wwid 3600a0b8000122c6d00000000453174fc alias fastt21l0 } multipath { wwid 3600a0b80000fd6320000000045317563 alias fastt21l1 } } devices { device { vendor "IBM" product "1742-900" getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_grouping_policy prio_callout path_checker no_path_retry failback } } group_by_prio "/usr/local/sbin/path_prio.sh %n"

tur fail immediate

LCI Conference 2007

National Center for Supercomputing Applications

38

Putting it all together

path_prio.sh
sdb matching line

multipath

50

path_prio.sh

Primary-paths

/usr/local/etc/primary-paths
0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4 0x10000000c95ebeb4 0x200200a0b8122c6e 0x200200a0b8122c6e 0x200200a0b8122c6e 0x200200a0b8122c6e 0x200300a0b8122c6e 0x200300a0b8122c6e 0x200300a0b8122c6e 0x200300a0b8122c6e 2:0:0:0 2:0:0:1 2:0:0:2 2:0:0:3 2:0:1:0 2:0:1:1 2:0:1:2 2:0:1:3 sdb sdc sdd sde sdi sdj sdk sdl 3600a0b8000122c6d00000000453174fc 3600a0b80000fd6320000000045317563 3600a0b8000122c6d0000000345317524 3600a0b80000fd6320000000245317593 3600a0b8000122c6d00000000453174fc 3600a0b80000fd6320000000045317563 3600a0b8000122c6d0000000345317524 3600a0b80000fd6320000000245317593 50 2 50 2 5 51 5 51

LCI Conference 2007

National Center for Supercomputing Applications

PATH_PRIO.SH - grep device from primary-paths file - return value from last column

39

Demo: Active/Passive Disk

Host
One Emulex LP11000

Disk
IBM DS4500 Luns presented through both controllers Luns accessible via 1 controller only at a time AVT enabled

LCI Conference 2007

National Center for Supercomputing Applications

AVT - Lun will migrate to alternate controller if requested there - Tolerance of cable/switch failure - AVT penalty - lun inaccessible for 5-10 secs while controller ownership changing SCREENS: /var/log/messages , multi-port-mon , command , script host 1. 2. No luns (ls_fc_luns) /etc/multipath.conf 1. 2. 3. 4. 5. 6. 7. 1. Multipaths (fastt) Devices (fastt) Identify controller A, controller B

/usr/local/sbin/path_prio.sh /usr/local/etc/primary-paths Add luns (scan_fc_luns) 1. 1. 1. 2. 3. See multipath bindings & path_prio.sh output in /var/log/messages Multipath -v2 -l Script-host: disable disk port A See multipathd reconfig in /var/log/messages See I/O path change in multi-port-mon Script-host: enable disk port A View current multipath configuration Failover test

8.

Recover test 1.

40

Demo: Active/Active Disk

Host
One Emulex LP11000

Disk
DDN 8500 Luns accessible via both controllers (no penalty)

LCI Conference 2007

National Center for Supercomputing Applications

SCREENS: multi-port-mon , /var/log/messages , command , script-host 1. /etc/multipath.conf 1. 2. 2. 3. 4. 1. 1. 1. 2. 3. 5. 1. 2. 3. Devices (DDN) (path_prio = /bin/true ; path_grouping_policy = multibus) Multipath (DDN) See multipath bindings in /var/log/messages Multipath -v2 -l Expected changes in multi-port-mon Disable switch port for disk ctlr 1 See failover in /var/log/messages and multi-port-mon Expected changes in multi-port-mon Enable switch port for disk ctlr 1 See failback in /var/log/messages and multi-port-mon

Luns present? (ls_fc_luns) Add luns if needed (scan_fc_luns) View multipath configuration Failover test

Restore ctlr access

41

Path Grouping Policy Matrix

1 HBA Active/Active Active/Passive with AVT Active/Passive w/o AVT
LCI Conference 2007

2 HBAs multibus path_prio failover

(demo1) multibus (demo2) path_prio *multiple points of failure

National Center for Supercomputing Applications

ACTIVE/ACTIVE 2 HBAs - trivial, same as demo1 - Each HBA sees 1 ctlr - Can let both HBAs see both ctlrs (4 paths to each lun) + Use path_prio if need to control path usage ACTIVE/PASSIVE (AVT) 2 HBAs - trivial, similar to demo2 ACTIVE/PASSIVE (no AVT) 1 HBA - Tolerant of ctlr failure only. - If anything else fails, luns will not AVT to alternate ctlr, host will lose access ACTIVE/PASSIVE (no AVT) 2 HBAs - Non-preferred paths will be failed - Each HBA must have full access to both controllers

42

Linux Multipath Errata

Making changes to multipath.conf
Stop multipathd service Clear multipath bindings
/sbin/multipath -F

Create new multipath bindings

/sbin/multipath -v2 -l

Start multipathd service

Cannot multipath root or boot device user_friendly_names

Not really, just random names dm-1, dm-2
LCI Conference 2007
National Center for Supercomputing Applications

CANNOT MULTIPATH ROOT OR BOOT DEVICE - per ap-rhcs-dm-multipath-usagetxt.html (see references section)

43

Linux Multipath Resources

multipath.conf.annotated man multipath http://christophe.varoqui.free.fr/wiki/wakka.php?wiki=H ome
Multipath tools official home

http://www.redaht.com/docs/manuals/csgfs/browse/rhcs-en/ap-rhcs-dm-multipath-usagetxt.html
Description of output (multipath -v2 -l)

http://kbase.redhat.com/faq/FAQ_85_7170.shtm
Setup device-mapper multipathing in Red Hat Enterprise Linux 4?

http://dims.ncsa.uiuc.edu/set/san
Multi-port-mon Set switchport state : (en/dis)able switch port via SNMP
LCI Conference 2007
National Center for Supercomputing Applications

MULTIPATH.CONF.ANNOTATED (RedHat) - /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.annotated

44