Welcome to:

Unit 8
Block Devices,
RAID, and LVM

© Copyright IBM Corporation 2006

Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 3.3.3
Unit Objectives
After completing this unit, you should be able to:
• Name the most important characteristic of a block device
• List various block devices
• List the device naming scheme for IDE and SCSI hard disks
• Partition a hard disk and list the device naming for partitions
• Use RAM disks
• Configure and use LVM
• Configure and use RAID

© Copyright IBM Corporation 2006

Block Devices
• A "Block Device" is any device which allows random access
("seeks") and which is divided into "blocks" of a given size.

block
1 2 3 4 5 6 7 8

0 512 4096
byte

• Typical block devices:

– Hard disks (and partitions)
– Floppy disks
– Virtual block devices (RAID and LVM)

© Copyright IBM Corporation 2006

Traditional Block Device Naming
• All block devices have a special file representation in /dev
# ls -l /dev
total 1
drwxr-xr-x 10 root root 7800 Apr 4 03:37 .
drwxr-xr-x 23 root root 568 Apr 4 03:36 ..
drwxr-xr-x 3 root root 60 Apr 4 04:27 .udev
crw------- 1 root video 10, 175 Apr 4 03:36 agpgart
prw------- 1 root root 0 Apr 4 03:37 blog
drwxr-xr-x 3 root root 60 Apr 4 03:36 bus
lrwxrwxrwx 1 root root 3 Apr 4 03:36 cdrom -> hdc
crw------- 1 root root 5, 1 Jan 30 05:52 console
. . .

• fd0, fd1, ...: floppy disk (max 8)

• hda, hdb, ...: IDE hard disk (max 8)
• sda, sdb, ...: SCSI hard disk (max 128)

© Copyright IBM Corporation 2006

Dynamic Device Naming with udev
• Traditionally stored in /dev directory on Linux
– Node(file) for every possible type of device, regardless of
whether it actually exists in the system
• The /dev directory can take up a lot of space
•udev introduces a new way of creating device nodes
– Compares the information made available by sysfs with
data provided by the user in the form of rules
•udev rule driven
– /etc/udev/udev.rules
• Identify device path and serial number
– udevinfo command
• udevinfo –a –p `udevinfo –q path –n /dev/hda`

© Copyright IBM Corporation 2006

Floppy Disks
• Floppy disks may need to be initialized first with correct size
information (low level format) with the fdformat command

# fdformat -n /dev/fd0h1440
Double-sided, 80 tracks, 18 sec/track. Total
capacity 1440 kB.
Formatting ... done
Verifying ... done

Note: To format a floppy disk with a different capacity, use

another device file (for example /dev/fd0u2880)

© Copyright IBM Corporation 2006

Hard Disks
• Most common device for persistent storage
• Two common types: IDE and SCSI
• IDE (Integrated Drive Electronics)
– Max 2 disks (master/slave) on one bus
– Max 2 buses on adapter
– Max 2 adapters in system
– Also supports CD-ROM (ATAPI)
– Device naming /dev/hda, hdb, ..., hdh
• SCSI (Small Computer System Interface)
– Different subtypes: fast, wide, fast wide, ultra-wide, ...
– Max 7 or 15 disks on one bus (depends on subtype)
– Needs correct termination at both ends of bus
– Generally more expensive than IDE
– Also supports CD-ROM, tapes, zip drives, ...
– Device naming /dev/sda, ..., sdz, sdaa, ..., sddx

© Copyright IBM Corporation 2006

Monitoring Hard Disk Health
• Self Monitoring And Reporting Technology (SMART)
– Technology included in most modern IDE/SCSI disks
– Reports various disk parameters (errors, temperature,
various counts, ...) to OS
• smartctl: Tool for accessing SMART data
– smartctl -a /dev/hda
reports all /dev/hda parameters
– smartctl -X /dev/hda initialized long self-test
– man smartctl for more options
• smartd: Monitoring daemon
– Monitors attributes every 30 minutes
– Reports changes to syslog
– Newer versions can send mail too if attributes change
(use /etc/smartd.conf for configuration)

© Copyright IBM Corporation 2006

Hard Disk Partitions
• IDE and SCSI hard disks can be partitioned
• Maximum of four primary partitions
• One primary partition may be an extended partition
• An extended partition can hold an unlimited amount of
logical partitions (Linux: max 59 for IDE, 11 for SCSI)

hda: The first sector of the disk contains the MBR

master boot record and Partition Table
partition table
hda1: First primary partition holds a Windows
Windows filesystem
hda2: Second primary partition is an extended
partition and holds three logical partitions
hda5: First logical partition holds a Linux
Linux /
filesystem that will be mounted as /
hda6: Second logical partition holds a Linux
Linux /home filesystem that will be mounted as /home
hda7: Third logical partition holds a Linux swap
Linux swap space

© Copyright IBM Corporation 2006

Partitioning Tools
• PartitionMagic
– Commercial DOS/Windows program from PowerQuest
– Can create/resize/move/delete partitions
• fips
– Free DOS program, comes with all Linux distributions
– Can split an existing Windows partition in two parts
• fdisk
– Virtually every PC OS comes with a tool fdisk to create
partitions for that OS
• Linux, Windows, and so forth
• parted
– GPL'ed Linux program, available at www.gnu.org
– Can create/resize/move/delete partitions
• Disk Druid and others
– Partitioning program integrated in Linux install program

© Copyright IBM Corporation 2006

RAM Disks
• A RAM disk is a block device created in memory
– Automatically created when used
– Size is compiled into the kernel
– Disappears after reboot
• Linux supports up to 16 RAM disks by default (255 max)
• To create a RAM disk: write data to it
• To delete a RAM disk: use freeramdisk

# tar -cvf /dev/ram0 /etc

# tar -tvf /dev/ram0
-rw-r--r-- root/root 854 2003-04-19 20:25:41 etc/passwd
-rw-r--r-- root/root 440 2003-04-19 20:25:41 etc/group
...
# freeramdisk /dev/ram0

© Copyright IBM Corporation 2006

The "loop" Device
• The "loop" device is used to access files as block devices
• Linux supports a maximum of 16 loop devices by default

# mount -o loop bootnet.img /mnt/floppy

# mount -o loop,ro SuSE-8.2-CD1.iso /media/cdrom

– Use losetup to initially set up the loop device

– Can then mount and umount the device transparently

# dd if=/dev/zero of=secrets.enc bs=1M count=32

# losetup -e blowfish /dev/loop0 secrets.enc
... asks for password to be used as encryption key ...
# mke2fs /dev/loop0
# losetup -d /dev/loop0
# mount -o loop,encryption=blowfish secrets.enc /mnt/secrets

© Copyright IBM Corporation 2006

Logical Volume Management (1 of 2)
• Traditional disk partitioning scheme has several
disadvantages:
– Virtually impossible to resize or move a partition
– Partition size is limited by disk size
• Logical Volume Management solves these disadvantages:
– One or more Physical Volumes (hard disks, partitions)
are assigned to a Volume Group (VG)
– All Physical Volumes (PV) are split into Physical Extents
(PE) of identical size (default 4 MB)
– PEs in a VG can be combined into Logical Volumes (LV),
which can be used like any block device
• An LV can span multiple disks
• To increase the size of an LV, add PEs
• To increase the size of a VG, add PVs

© Copyright IBM Corporation 2006

Logical Volume Management (2 of 2)

physical volume physical volume

(hard disk or partition) (hard disk or partition)

PE PE PE PE PE PE

PE PE PE PE PE PE

PE PE PE PE PE PE
logical volume

volume group
© Copyright IBM Corporation 2006
LVM Implementation Overview
• Add hard disks and/or create partitions (type 0x8e) on
existing hard disks
• Initialize physical volumes (disks or partitions)

# pvcreate /dev/hda3
# pvcreate /dev/hdb

• Create volume group "vg00" with physical volumes

# vgcreate vg00 /dev/hda3 /dev/hdb

• Create logical volume "lv00" in volume group

# lvcreate -L 50M -n lv00 vg00

• Can now use /dev/vg00/lv00 as block device

© Copyright IBM Corporation 2006

Physical Volume Commands
• pvcreate <pv>
– Initializes a physical volume by putting an (empty) Volume Group
Descriptor Area at the start of the PV
VGDA ( Volume Group Descriptor Area)

• pvmove [-n <lv>] <source pv> [<destination pv>]

– Move PEs from one PV to another PV in the volume group
• pvdisplay <pv>
– List information about a PV

© Copyright IBM Corporation 2006

Volume Group Commands
• vgcreate [-s <pe size>] <vg name> <pv> [<pv>...]
– Create a volume group

Volume Group (VG)

Physical Physical Physical

Volume (PV) Volume (PV) Volume (PV)

• vgdisplay [<vg>]
– Display information about a volume group
• vgremove <vg>
– Delete a volume group

© Copyright IBM Corporation 2006

Logical Volume Commands
• lvcreate -L <size> [-n <lv name>] <vg> [<pv>...]
– Create a logical volume in a volume group

Logical Volume (LV) LV LV LV

Volume Group (VG)

Physical Physical Physical

Volume (PV) Volume (PV) Volume (PV)

• lvdisplay <lv> [<lv>...]

– Display information about a logical volume
• lvremove <lv> [<lv>...]
– Remove a logical volume

© Copyright IBM Corporation 2006

Striping Logical Volumes
• A Logical Volume may be striped across two or more Physical
Volumes during creation
• For large data transfers, this increases performance

# lvcreate -L 300M -i 2 -I 8 -n mystripedlv vg00

physical volume physical volume

(hard disk or partition) (hard disk or partition)

PE PE PE PE PE PE

PE PE PE PE PE PE

PE PE PE PE PE PE

volume group

/dev/vg00/mystripedlv
© Copyright IBM Corporation 2006
Extending/Reducing a Volume Group
• To add or remove a Physical Volume to or from a Volume
Group, use the vgextend and vgreduce commands
• To move Physical Extents from one Physical Volume to
another use pvmove

# vgextend vg00 /dev/hdc6

# vgreduce vg00 /dev/sda5
ERROR: can't reduce volume group "vg00" by used
physical volume "/dev/sda5"
# pvmove /dev/sda5 /dev/hdc6
# vgreduce vg00 /dev/sda5

© Copyright IBM Corporation 2006

Extending/Reducing a Logical Volume
• To extend/shrink a Logical Volume use the
lvextend/lvreduce commands
Use -L option to specify size in bytes
Use -l option to specify size in PEs
•lvextend/lvreduce do NOT extend/shrink a filesystem in
the LV automatically!!!
(Extending/shrinking a filesystem will be covered later)
# lvextend -L +300M /dev/vg00/mylv
lvextend -- rounding relative size up to physical extent boundary
lvextend -- extending logical volume "/dev/vg00/mylv" to 380 MB
lvextend -- doing automatic backup of volume group "system"
lvextend -- logical volume "/dev/vg00/mylv" successfully extended
# lvreduce -l -12 /dev/system/mystripedlv
...

© Copyright IBM Corporation 2006

LVM Backup and Recovery
• It is very important to save the LVM metadata stored in the VGDA
for recovering reasons.

1. vgcfgbackup
2. vgcfgrestore -n vg_name PV
VGDA VGDA VGDA

/etc/lvmconf/vg_name.conf

VGDA VGDA VGDA

© Copyright IBM Corporation 2006

Additional LVM Considerations
• Linux LVM implementation has "snapshot" capability
– Can be useful for fast backups
• LVM information can be obtained from /proc/lvm tree
• If LVM-based filesystems are listed in /etc/fstab, then LVM
support needs to be included in the initrd
• Mirroring is handled outside of the LVM structure

© Copyright IBM Corporation 2006

LVM Version 2
• LVM 2 is almost completely backward compatible with
volumes created with LVM 1
• LVM 2 uses the device mapper kernel driver
• LVM 2 snapshots are read/write by default

© Copyright IBM Corporation 2006

RAID
• "Redundant Array of Inexpensive Disks"
• Typical PC hard disks, compared to expensive mainframe-
quality hard disks are
– Slower
– Less reliable
– Smaller
– ...But less expensive
• Idea: Use multiple hard disks in an array to create a larger,
logical device that is
– Faster
– More reliable
– Or larger
– ...And still relatively inexpensive

© Copyright IBM Corporation 2006

 RAID Levels (1 of 2)

RAID-Linear RAID-0: striping RAID-1: mirroring

1 6 1 2 1 1
2 7 3 4 2 2
3 8 5 6 3 3
4 9 7 8 4 4
5 10 9 10 5 5

RAID-4: striping with parity disk RAID-5: striping with parity

1 2 p 1 2 p
3 4 p 3 p 4
5 6 p p 5 6
7 8 p 7 8 p
9 10 p 9 p 10

© Copyright IBM Corporation 2006

RAID Levels (2 of 2)
• RAID Levels have different characteristics
– RAID-5 is not "better" than RAID-1
• Use RAID level according to needs
RAID Min # Read Write Redundancy Data capacity Other remarks
level disks performance performance with 3x1GB
disk
Linear 2 Equal Equal No 3 GB Can be used if disks are not
equal

0 2 Fast Fast No 3 GB

1 2 fast Somewhat Yes 1 GB Can sustain N-1 disk

slower crash(s)

4 3 Somewhat Slow Yes 2 GB Can sustain 1 disk crash

faster Parity disk is bottleneck

5 3 Somewhat Somewhat Yes 2 GB Can sustain 1 disk crash

faster faster CPU intensive

(*) Performance compared to a single disk, for data transfers greater than block size
© Copyright IBM Corporation 2006
Linux RAID Support
• Software RAID
– Implemented in Linux kernel
– Needs raidtools package
– Uses disk partitions to create RAID devices
– Logical device name: /dev/mdn
• Hardware RAID
– Implemented in special adapter cards
– Adapter needs to be supported by Linux kernel
– Generally specific software needed to configure adapter
correctly (might not be available under Linux)
– RAID devices show up as regular SCSI disk

© Copyright IBM Corporation 2006

Linux Software RAID
Implementation (1 of 2)
• Create RAID partitions
– Partition type 0xfd (Linux RAID autodetect)
• Create /etc/raidtab file
# cat /etc/raidtab
raiddev /dev/md0
raid-level 0
nr-raid-disks 2
persistent-superblock 0
chunk-size 8
device /dev/hda1
raid-disk 0
device /dev/hdb1
raid-disk 1

© Copyright IBM Corporation 2006

Linux Software RAID
Implementation (2 of 2)
• Initialize RAID devices with mkraid /dev/md0
• /dev/md0 may be used as a normal block device

# mkraid /dev/md0
# mkfs /dev/md0
...
# mount /dev/md0 /mnt/raid0

© Copyright IBM Corporation 2006

Multiple Devices Admin - mdadm
• raidtools replacement
• Extremely useful tool for running RAID systems
• Common syntax for every RAID management
command
• No configuration file

© Copyright IBM Corporation 2006

mdadm Modes
• mdadm major modes of operation:
– Assemble
– Build
– Create
– Manage
– Misc
– Follow or Monitor

© Copyright IBM Corporation 2006

mdadm Implementation
• Create
# mdadm --create -v /dev/md0 -l \
linear -n 2 /dev/hda6 /dev/hda7
• Manage
# mdadm --detail /dev/md0
• Misc
# cat /proc/mdstat

© Copyright IBM Corporation 2006

Watching a Running RAID
• To see the current state of your RAID view /proc/mdstat

# cat /proc/mdstat
Personalities:[linear] [raid5]
read_ahead 1024 sectors
md0: active raid5 sdc1[2] sdb1[1] sdd1[3] sde1[0]
633849 blocks level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>

–"U" means all devices are up

– If a device fails, "F" is shown Very
Important

• To simulate a failed disk use raidsetfaulty

© Copyright IBM Corporation 2006
Spare Disks
• To make RAID1/RAID5 more failsafe in case of a disk failure, use
spare disks!
# cat /etc/raidtab
...
nr-spare-disks 1
device /dev/sdd1
spare-disk 0
...

spare disk

The spare disk takes over...

• Remove a failed disk with raidhotremove

• Add a new disk to the array with raidhotadd

© Copyright IBM Corporation 2006

Additional RAID Considerations
• Put RAID partitions on different disks
• Use different SCSI or IDE controllers if possible for different
disks that are part of a RAID volume
• Do not use RAID for /boot partition
• If RAID-based filesystems are listed in /etc/fstab, then RAID
support needs to be included in the initrd
• Software RAID-4 and RAID-5 needs a lot of CPU time
• Do not use RAID-linear or RAID-0 for swap space
– The Linux kernel can stripe across swap spaces more
efficiently

© Copyright IBM Corporation 2006

Checkpoint
1. True / False: RAID volumes can be used as Physical
Volumes in an LVM setup.

2. Mirroring is offered by RAID level:

a. Linear
b. Zero
c. One
d. Four
e. Five

3. What command is used to create a RAM disk?

______________________________________________

© Copyright IBM Corporation 2006

Checkpoint Solutions
1. True / False: RAID volumes can be used as Physical
Volumes in an LVM setup.

2. Mirroring is offered by RAID level:

a. Linear
b. Zero
c. One
d. Four
e. Five

3. What command is used to create a RAM disk?

There is no specific command. A RAM disk is created
automatically as soon as you start using it.

© Copyright IBM Corporation 2006

Exercise 8: Block Devices, RAID, and LVM

What you will do in this exercise:

• Create and manipulate LVM
resources
• Use RAM disk resources
• Configure RAID devices

© Copyright IBM Corporation 2006

Unit Summary

• Block devices are devices that offer random access.

• Block devices are: hard disks, hard disk partitions, floppy
disks, RAM disks, files, LVM logical volumes and RAID
volumes.
• Block devices can be used to store a filesystem, as swap
space, or "raw".
• Logical Volume Management allows you to go beyond the
limits of regular partitioning, since it allows you to create
logical volumes that are larger than the disk size, and which
can be resized.
• RAID is a technology to use inexpensive, less reliable,
relatively slow and small IDE or SCSI disks in such a
fashion that the virtual volume is larger, more reliable or
faster than the individual disks.
© Copyright IBM Corporation 2006