The Linux Logical Volume Manager

by Heinz Mauelshagen and Matthew O'Keefe Introduction Basic LVM commands Differences between LVM1 and LVM2 Summary About the authors Storage technology plays a critical role in increasing the performance, availability, and manageability of Linux servers. One of the most important new developments in the Linux 2.6 kernelon which the Red Hat Enterprise Linux 4 kernel is basedis the Linux Logical Volume Manager, version 2 (or LVM 2). It combines a more consistent and robust internal design with important new features including volume mirroring and clustering, yet it is upwardly compatible with the original Logical Volume Manager 1 (LVM 1) commands and metadata. This article summarizes the basic principles behind the LVM and provide examples of basic operations to be performed with it.

Introduction
Logical volume management is a widely-used technique for deploying logical rather than physical storage. With LVM, "logical" partitions can span across physical hard drives and can be resized (unlike traditional ext3 "raw" partitions). A physical disk is divided into one or more physical volumes (Pvs), and logical volume groups (VGs) are created by combining PVs as shown in Figure 1. LVM internal organization. Notice the VGs can be an aggregate of PVs from multiple physical disks.

Figure 1. LVM internal organization Figure 2. Mapping logical extents to physical extents shows how the logical volumes are mapped onto physical volumes. Each PV consists of a number of fixed-size physical extents (PEs); similarly, each LV consists of a number of fixed-size logical extents (LEs). (LEs and PEs are always the same size, the default in LVM 2 is 4 MB.) An LV is created by mapping logical extents to physical extents, so that references to logical block numbers are resolved to physical block numbers. These mappings can be constructed to achieve particular performance, scalability, or availability goals.

Figure 2. Mapping logical extents to physical extents For example, multiple PVs can be connected together to create a single large logical volume as shown in Figure 3. LVM linear mapping. This approach, known as a linear mapping, allows a file system or database larger than a single volume to be created using two physical disks. An alternative approach is a striped mapping, in which stripes (groups of contiguous physical extents) from alternate PVs are mapped to a single LV, as shown in Figure 4. LVM striped mapping. The striped mapping allows a single logical volume to nearly achieve the combined performance of two PVs and is used quite often to achieve high-bandwidth disk transfers.

Figure 3. LVM linear mapping

Figure 4. LVM striped mapping (4 physical extents per stripe) Through these different types of logical-to-physical mappings, LVM can achieve four important advantages over raw physical partitions: Logical volumes can be resized while they are mounted and accessible by the database or file system, removing the downtime associated with adding or deleting storage from a Linux server Data from one (potentially faulty or damaged) physical device may be relocated to another device that is newer, faster or more resilient, while the original volume remains online and accessible Logical volumes can be constructed by aggregating physical devices to increase performance (via disk striping) or redundancy (via disk mirroring and I/O multipathing) Logical volume snapshots can be created to represent the exact state of the volume at a certain point-in-time, allowing accurate backups to proceed simultaneously with regular system operation

1. 2. 3. 4.

Basic LVM commands

Initializing disks or disk partitions
To use LVM, partitions and whole disks must first be converted into physical volumes (PVs) using the pvcreate command. For example, to convert /dev/hda and /dev/hdb into PVs use the following commands:
pvcreate /dev/hda pvcreate /dev/hdb

If a Linux partition is to be converted make sure that it is given partition type 0x8E using fdisk, then use pvcreate:
pvcreate /dev/hda1

Creating a volume group

Once you have one or more physical volumes created, you can create a volume group from these PVs using the vgcreate command. The following command:
vgcreate volume_group_one /dev/hda /dev/hdb

creates a new VG called volume_group_one with two disks, /dev/hda and /dev/hdb, and 4 MB PEs. If both /dev/hda and /dev/hdb are 128 GB in size, then the VGvolume_group_one will have a total of 2**16 physical extents that can be allocated to logical volumes. Additional PVs can be added to this volume group using the vgextend command. The following commands convert /dev/hdc into a PV and then adds that PV tovolume_group_one:
pvcreate /dev/hdc vgextend volume_group_one /dev/hdc

This same PV can be removed from volume_group_one by the vgreduce command:

vgreduce volume_group_one /dev/hdc

Note that any logical volumes using physical extents from PV /dev/hdc will be removed as well. This raises the issue of how we create an LV within a volume group in the first place.

Creating a logical volume

We use the lvcreate command to create a new logical volume using the free physical extents in the VG pool. Continuing our example using VG volume_group_one (with two PVs /dev/hda and /dev/hdb and a total capacity of 256 GB), we could allocate nearly all the PEs in the volume group to a single linear LV called logical_volume_one with the following LVM command:
lvcreate -n logical_volume_one --size 255G volume_group_one

Instead of specifying the LV size in GB we could also specify it in terms of logical extents. First we use vgdisplay to determine the number of PEs in the volume_group_one:
vgdisplay volume_group_one | grep "Total PE"

which returns
Total PE 65536

Then the following lvcreate command will create a logical volume with 65536 logical extents and fill the volume group completely:
lvcreate -n logical_volume_one -l 65536 volume_group_one

To create a 1500MB linear LV named logical_volume_one and its block device special file /dev/volume_group_one/logical_volume_one use the following command:
lvcreate -L1500 -n logical_volume_one volume_group_one

The lvcreate command uses linear mappings by default. Striped mappings can also be created with lvcreate. For example, to create a 255 GB large logical volume with two stripes and stripe size of 4 KB the following command can be used:
lvcreate -i2 -I4 --size 255G -n logical_volume_one_striped volume_group_one

It is possible to allocate a logical volume from a specific physical volume in the VG by specifying the PV or PVs at the end of the lvcreate command. If you want the logical volume to be allocated from a specific physical volume in the volume group, specify the PV or PVs at the end of the lvcreate command line. For example, this command:
lvcreate -i2 -I4 -L128G -n logical_volume_one_striped volume_group_one /dev/hda /dev/hdb

creates a striped LV named logical_volume_one that is striped across two PVs (/dev/hda and /dev/hdb) with stripe size 4 KB and 128 GB in size. An LV can be removed from a VG through the lvremove command, but first the LV must be unmounted:
umount /dev/volume_group_one/logical_volume_one lvremove /dev/volume_group_one/logical_volume_one

Note that LVM volume groups and underlying logical volumes are included in the device special file directory tree in the /dev directory with the following layout:
/dev/<volume_group_name>/<logical_volume_name>

so that if we had two volume groups myvg1 and myvg2 and each with three logical volumes named lv01, lv02, lv03, six device special files would be created:
/dev/myvg1/lv01 /dev/myvg1/lv02 /dev/myvg1/lv03 /dev/myvg2/lv01 /dev/myvg2/lv02 /dev/myvg2/lv03

Extending a logical volume

An LV can be extended by using the lvextend command. You can specify either an absolute size for the extended LV or how much additional storage you want to add to the LVM. For example:
lvextend -L120G /dev/myvg/homevol

will extend LV /dev/myvg/homevol to 12 GB, while

lvextend -L+10G /dev/myvg/homevol

will extend LV /dev/myvg/homevol by an additional 10 GB. Once a logical volume has been extended, the underlying file system can be expanded to exploit the additional storage now available on the LV. With Red Hat Enterprise Linux 4, it is possible to expand both the ext3fs and GFS file systems online, without bringing the system down. (The ext3 file system can be shrunk or expanded offline using the ext2resize command.) To resize ext3fs, the following command
ext2online /dev/myvg/homevol

will extend the ext3 file system to completely fill the LV, /dev/myvg/homevol, on which it resides. The file system specified by device (partition, loop device, or logical volume) or mount point must currently be mounted, and it will be enlarged to fill the device, by default. If an optional size parameter is specified, then this size will be used instead.

Differences between LVM1 and LVM2

The new release of LVM, LVM 2, is available only on Red Hat Enterprise Linux 4 and later kernels. It is upwardly compatible with LVM 1 and retains the same command line interface structure. However it uses a new, more scalable and resilient metadata structure that allows for transactional metadata updates (that allow quick recovery after server failures), very large numbers of devices, and clustering. For Enterprise Linux servers deployed in mission-critical environments that require high availability, LVM2 is the right choice for Linux volume management. Table 1. A comparison of LVM 1 and LVM 2summarizes the differences between LVM1 and LVM2 in features, kernel support, and other areas.

Features
RHEL AS 2.1 support

LVM1
No

LVM2
No

Features
RHEL 3 support RHEL 4 support Transactional metadata for fast recovery Shared volume mounts with GFS Cluster Suite failover supported Striped volume expansion Max number PVs, LVs Max device size Volume mirroring support

LVM1
Yes No No No Yes No 256 PVs, 256 LVs 2 Terabytes No

LVM2
No Yes Yes Yes Yes Yes 2**32 PVs, 2**32 LVs 8 Exabytes (64-bit CPUs) Yes, in Fall 2005

Table 1. A comparison of LVM 1 and LVM 2

Summary
The Linux Logical Volume Manager provides increased manageability, uptime, and performance for Red Hat Enterprise Linux servers. You can learn more about LVM by visiting to following websites:

To display PVs of an LV #lvdisplay m /dev/VG1/LV1

Replace a LVM2 drive with a larger one

My MythTV box has only 2 IDE ports so I can only have 4 devices. The MB doesn't have any SATA connectors so I'm a bit limited to what I can do disk-wise. In any case, the box, running RHEL 4, is setup as follows:

Disk hda hdb hdc hdd

Size 30GB 160GB n/a 120GB

Purpose OS and user files (/, /usr, /home, etc) MythTV disk /video, LVM2 DVD writer MythTV disk /video, LVM2

I have a PVR 250 and just record with the default settings so the recorded quality isn't that great but then again, I'm not that picky. With my recording and watching schedule this was leaving me with about 40GB of free space at any given time. With a new season starting I don't think this will be enough space. In doing some de-cluttering I found a 250GB disk lying around (don't ask). So I decided to replace the 120 with the 250. Not a large boost in size but it should let me record in peace for the upcoming season. It was difficult finding a single source of information on how to do this so I cobbled the following together from various sources. It worked for me, YMMV. In all of these steps /video (my VG) is mounted but mythbackend is stopped. The system was otherwise quiet. Back up

If possible you realy should back things up before starting this, I did. The trouble is that LVM encourages you to create volumes larger than a single disk so back ups become difficult. Prepare the disk In order to use the new disk I replaced the DVD writer with it, so I had 4 disks. I used fdisk to create a single partition for the whole disk and set the type to 8e (LVM2). In these examples, /dev/hdc1 is the new 250GB disk and /dev/hdd is the old 120GB disk. Now to make the new disk available to LVM:
# pvcreate /dev/hdc1 pvcreate -- physical volume "/dev/hdc1" successfully created

Add it to the volume group My volume group is named vg.

# vgextend vg /dev/hdc1 vgextend -- INFO: maximum logical volume size is 259.97 Gigabyte vgextend -- doing automatic backup of volume group "vg" vgextend -- volume group "vg" successfully extended

Move the data:

# pvmove /dev/hdd /dev/hdc1 /dev/hdd: Moved: 0.2% /dev/hdd: Moved: 0.4% /dev/hdd: Moved: 0.6% ... /dev/hdd: Moved: 100.0%

This took a LONG time. I saw it go over an hour myself, then I went to bed. It was done in the morning. As a benchmark, it took 3 hours to move a nearly full 250GB disk to a 500GB disk.

If you get the error message:

mirror: Required device-mapper target(s) not detected in your kernel

Then load the kernel module dm-mirror and do the pvmove command again. Remove the unused disk
# vgreduce vg /dev/hdd vgreduce -- doing automatic backup of volume group "vg" vgreduce -- volume group "vg" successfully reduced by physical volume: vgreduce -- /dev/hdd

Now I powered the system down and physically removed the 120GB disk and replaced it with the 250. I also re-connected the DVD writer. I was a bit worried that LVM wouldn't "know" that I had moved hdc1 to hdd but it didn't seem to care about the order of the disks. If you do a pvscan before removing the disk it will still show up but won't be part of a volume group. Reboot. Extend the volume group I need to tell the volume group that I have more space available. First, let's see how much space I can add:
# pvscan PV /dev/hdb VG vg lvm2 [149.05 GB/ 0 free] PV /dev/hdd1 VG vg lvm2 [110.92 GB/121.96 GB free] Total: 2 [259.97 GB] / in use: 2 [259.97 GB] / in no VG: 0 [0

Ok, so this tells me I have 121 GB and change free on the drive. Let us be conservative and add the 121.
# lvextend -L+121G /dev/vg/video lvextend -- extending logical volume "/dev/vg/video" to 380 GB

lvextend -- doing automatic backup of volume group "vg" lvextend -- logical volume "/dev/vg/video" successfully extended

Ok, now pvscan again to see if anything is left:

# pvscan PV /dev/hdb VG vg lvm2 [149.05 GB/ 0 free] PV /dev/hdd1 VG vg lvm2 [231.92 GB / 96MB free] Total: 2 [380.97 GB] / in use: 2 [380.97 GB] / in no VG: 0[0

Ok, let's go ahead and use the entire drive:

# lvextend -L+96M /dev/vg/video lvextend -- extending logical volume "/dev/vg/video" to 381 GB lvextend -- doing automatic backup of volume group "vg" lvextend -- logical volume "/dev/vg/video" successfully extended # pvscan PV /dev/hdb VG vg lvm2 [149.05 GB / 0 free] PV /dev/hdd1 VG vg lvm2 [232.88 GB / 0 free] Total: 2 [381.93 GB] / in use: 2 [381.93 GB] / in no VG: 0 [0

Extend the file system Now the space is available but the file system doesn't know about it. I need to extend it. I'm using ext3 so if you are using something else, you'll need another guide for this section. The drive must be mounted for this to work:
# ext2online /dev/vg/video ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b

Yes, there is no other output. This took just about 5 minutes for me. Let's see if it worked.
# df /video Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg-video 394197468 212914556 161266764 57% /video

Bingo! Check the file system Caution tells us to check the disk:

# umount /video # fsck -f /video fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/vg/video: 141/50069504 files (29.8% non-contiguous), 54799848/100120576 blocks # mount /video

Things look good, we're done. I fired up mythbackend and verified that my files were indeed there and playable, woo hoo. Resources Resize an LVM logical volume on ext2 or ext3 LVM HOWTO Extending MythTV recording space with LVM Notes I've done this twice more. I replaced the 160GB and 250GB disks with twin 500GB disks. I did the replacements one at a time, running through all of these steps each time.

Creating Snapshot
#lvcreate --size 100m --snapshot --name /dev/vg01_snapshots/snap /dev/vg00/lvol1

Create a new logical volume with a file system and mount it. [root@titanium ~]# lvcreate -L1G -n lv_test vg_titanium Logical volume "lv_test" created [root@titanium ~]# mke2fs -j /dev/vg_titanium/lv_test mke2fs 1.41.14 (22-Dec-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 33 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. [root@titanium ~]# mkdir /mnt/test [root@titanium ~]# mount /dev/vg_titanium/lv_test /mnt/test Create some test data then take a snapshot and create some more test data. [root@titanium ~]# touch /mnt/test/testdata-$(date +%Y%m%d%H%M%S) [root@titanium ~]# ls -l /mnt/test/ total 20 drwx------. 2 root root 16384 Sep 19 00:39 lost+found -rw-r--r--. 1 root root 0 Sep 19 00:39 testdata-20110919003936 [root@titanium ~]# SNAP_NAME="lv_test-snapshot-$(date +%Y%m%d%H%M%S)" [root@titanium ~]# lvcreate -s /dev/vg_titanium/lv_test -L 1G -n $SNAP_NAME Logical volume "lv_test-snapshot-20110919003936" created [root@titanium ~]# touch /mnt/test/testdata-$(date +%Y%m%d%H%M%S) [root@titanium ~]# ls -l /mnt/test/ total 24 drwx------. 2 root root 16384 Sep 19 00:39 lost+found -rw-r--r--. 1 root root 0 Sep 19 00:39 testdata-20110919003936 -rw-r--r--. 1 root root 0 Sep 19 00:39 testdata-20110919003937 Here is the actual merge command. [root@titanium ~]# lvconvert --merge /dev/vg_titanium/$SNAP_NAME Can't merge over open origin volume Merging of snapshot lv_test-snapshot-20110919003936 will start next activation. You will need to deactivate and activate to get the merge to start. You can immediately remount the filesystem as your view of it will be that of the snapshot once the merge has started. Once mounted you can check the data. [root@titanium ~]# umount /dev/vg_titanium/lv_test [root@titanium ~]# lvchange -an /dev/vg_titanium/lv_test

[root@titanium ~]# [root@titanium ~]# [root@titanium ~]# total 20 drwx------. 2 root -rw-r--r--. 1 root

lvchange -ay /dev/vg_titanium/lv_test mount /dev/vg_titanium/lv_test /mnt/test ls -l /mnt/test/ root 16384 Sep 19 00:39 lost+found root 0 Sep 19 00:39 testdata-20110919003936