Optimizing MySQL

performance with ZFS

Neelakanth Nadgir
Allan Packer
Sun Microsystems
Who are we?
Allan Packer
Principal Engineer, Performance
http://blogs.sun.com/allanp

Neelakanth Nadgir
Senior Engineer, Performance
http://blogs.sun.com/realneel

MySQL User Conference 2009

What do we do?
• Work in performance organization at Sun
Microsystems
• Work in the MySQL performance virtual team
• Check out a video about our group
> Google for “mysql optimization lab”

MySQL User Conference 2009

Agenda
• ZFS Introduction
• ZFS Performance features
• ZFS and MySQL
> MySQL IO model
> Best practices
> Performance Results
• ZFS FAQ

MySQL User Conference 2009

ZFS – The last word in Filesystems
• Developed at Sun circa 2004
• Opensource (CDDL)
> 47 patents have been donated to CDDL Patents Common
• Supported Platforms
> Officially supported on Solaris
> Default filesystem on OpenSolaris
> Read-Only access in MacOS 10.5
> Experimental feature on FreeBSD 7.1
> FUSE on Linux

MySQL User Conference 2009

ZFS – Core Features
• Data Integrity
> Everything is checksummed
• Immense capacity
> 128 bit filesystem
> Max size of a file is 264 bytes
> Each directory can hold 248 files
• Simple administration
> zpool & zfs are the only two commands you need to know
• Performance

MySQL User Conference 2009

ZFS – Design Principals
• Pooled Storage
> Common pool from which filesystems are allocated
• End-to-end data integrity
> Historically thought to be expensive, not really
> Alternative is unacceptable
• Everything is transactional
> Always consistent on disk
> Removes almost all constraints on IO order
> Think database transactions

MySQL User Conference 2009

ZFS – Design Principals
• Copy on Write
> Never overwrite live data
> On-disk state is always consistent.
– No fsck
• Entire storage pool is a tree of blocks rooted at
"uberblock"
> Transactions are COW of the tree
> Transaction group is committed when uberblock is
rewritten to point to new tree
> All levels of the tree are checksummed
> Checksum stored in parent node, separate from data
MySQL User Conference 2009
Copy-On-Write Transactions
1. Initial block tree 2. COW some blocks

3. COW indirect 4. Rewrite uberblock (atomic)

blocks

MySQL User Conference 2009

Constant-Time Snapshots

• At end of TX group, don't free COWed blocks

> Actually cheaper to take a snapshot than not!

Snapshot Current
uberblock uberblock

MySQL User Conference 2009

End-to-End Checksums
Disk Block Checksums ZFS Checksum Trees
• Checksum stored with data block
• Checksum stored in parent block pointer
• Any self-consistent block will pass
• Fault isolation between data and checksum
• Can't even detect stray writes
• Inherent FS/volume interface • Entire pool (block tree) is self-validating
Address Address
limitation • Enabling technology: ChecksumChecksum
ZFS stack integration
Data Data Address Address
Checksum Checksum ChecksumChecksum

Data Data

Only validates the media Validates the entire I/O path

✔ Bit rot ✔ Bit rot
✗ Phantom writes ✔ Phantom writes
✗ Misdirected reads and writes ✔ Misdirected reads and writes
✗ DMA parity errors ✔ DMA parity errors
✗ Driver bugs ✔ Driver bugs
✗ Accidental overwrite ✔
MySQL User Conference 2009
Accidental overwrite
Traditional Mirroring
1. Application issues a 2. Volume manager 3. Filesystem returns bad
read. Mirror reads the first passes bad block up to data to the application. If
disk, which has a corrupt filesystem. the data is modified, both
block. If it's a metadata block, the good & bad mirror copies
It can't tell. filesystem panics. If not... will then be corrupted.

Application Application Application

FS FS FS

xxVM mirror xxVM mirror xxVM mirror

MySQL User Conference 2009

 Self-Healing Data in ZFS
1. Application issues a 2. ZFS tries the second 3. ZFS returns good data
read. ZFS mirror tries the disk. Checksum indicates to the application and
first disk. Checksum that the block is good.
reveals that the block is repairs the damaged block.
corrupt on disk.

Application Application Application

ZFS mirror ZFS mirror ZFS mirror

MySQL User Conference 2009

ZFS Performance features
• Dynamic striping across all devices maximizes
throughput
• Copy on write makes most writes sequential
• Intelligent prefetch
• Multiple block sizes
• O(1) directory operations
• Explicit IO priority with deadline scheduling
• Globally optimal IO sorting and aggregation
• Concurrent writes
• Safely use write cache
MySQL User Conference 2009
ZFS read(2) code path MySQL
• ZFS Primary cache – The ARC 1
> primarycache=[all|metadata|none]
ARC
• ZFS Second level cache -
2
L2ARC
L2ARC
• When primarycache is used, 3
data is buffered in ARC
• If L2ARC is used, it is checked
before going to disk
• Prefetch is trigged if needed
• Reads have higher priority than
regular writes MySQL User Conference 2009
ZFS write(2) code path
• Regular writes are buffered in memory
• Periodically they are flushed to disk
> Usually a sequential write to disk
• Synchronous writes are written to the ZFS Intent
Log
> After periodic write, the ZIL is cleaned up
> ZIL aggregates IO from multiple writers
> Can use a separate disk (or SSD) for the ZIL
> ZIL can be disabled (Don't)
• ZFS employs byte-range locking to allow maximum
concurrency. i.e No Single Writer Lock
MySQL User Conference 2009
ZFS ARC (Filesystem buffer)
• Adaptive Replacement Cache
• Dynamically switches between MRU/MFU
• Caches data from all pools
• Dynamically shrinks or grows based on memory
pressure
• Survives full table scan
• Limitations
> Works better with 64bit kernel
> Works better with swap configured

MySQL User Conference 2009

MySQL IO Model
• Dependent on Storage engine
• Dependent on Workload
• Replication
> One thread reading and applying the binlog to the
datafiles (sequential reads, random writes)
> One thread updating the binlog (sequential writes)
• MyISAM
> Relies on filesystem to buffer data
> Index is buffered in the key cache

MySQL User Conference 2009

InnoDB IO Model
• InnoDB
> Reads are issued by user connection threads (N)
> Writes are done by asynchronous threads
– 1 for log and 1 for data files
– Configurable with Performance version
> Writes are either
– Synchronous writes
– Writes followed by a fsync()
> Doublewrite buffer

MySQL User Conference 2009

MySQL and ZFS Best practices

MySQL User Conference 2009

Best practices - Caching
• Prefer to cache inside MySQL/Innodb rather than
ARC
> Benchmark shows 7-200% improvement
> Same block is buffered inside Innodb as well as ARC
• Limit ARC Size
> Even though ARC is dynamic, more efficient to just limit it
• Cache only metadata for Innodb
> zfs set primarycache=metadata tank/db

MySQL User Conference 2009

Best practices – Record size
• Match recordsize to block size
> zfs set recordsize=16k tank/db
> Can be changed dynamically, but do this before creating
the database
• Prevents read-modify-write
• Read only data that you want and nothing more
• Innodb
> 16k recordsize for data
> 128k recordsize for log and binlog

MySQL User Conference 2009

Best practices – Prefetch
• ZFS has two kinds of prefetch
> File level prefetch AKA zfetch
> Low level prefetch AKA vdev prefetch
• Turn off file level prefetch
> set zfs:zfs_prefetch_disable = 1

• Low level prefetch is not trigged when recordsize is

set (i.e not 128k)
• Innodb prefetch assumes file is laid out in order of
primary key.
> Not true for ZFS
> Not configurable right now, but should be easy to fix
MySQL User Conference 2009
Best practices – IO
• ZFS IO scheduler prioritizes reads over regular
writes
> ZFS Log writes are still higher priority
> If IO queue is full, have to wait for empty slot
> Bug 6471212: will be fixed soon using reserved slots
• Prefer Raid0 or Mirroring over RaidZ
> RaidZ is not suitable for random IO
• Use L2ARC to reduce penalty of missing buffer
cache
> zpool add tank cache c2t0d0 c2t1d0

MySQL User Conference 2009

Best practices – Separate Intent Log
• ZFS log writes can use the Separate Intent log
> Usually NVRAM card or SSD
> Can be done dynamically.
> Match reliability of the pool
> Seen 10-20% improvement for certain workloads
• Use slog to get low latency writes
> zpool add tank log c2t0d0
> Watch out – Cannot remove a slog. Fix in progress

MySQL User Conference 2009

Best practices – Cache flush
• ZFS issues a cache flush after every transaction
group sync and synchronous writes
• Some vendors flush every time even if they have a
battery backed cache
> set zfs:zfs_nocacheflush = 1
• Be fair when you are comparing ZFS with other
filesystems which do not flush caches.

MySQL User Conference 2009

Best practices – Compression
• ZFS supports a pluggable compression
> Gzip and other algorithms
> Data is not compressed if less than 12.5% compression
• Scalable, asynchronous compression
> No need for query to wait for compression to complete
• CPU cost
> Compression is not free, but many algorithms to choose
from
• IO reduction
> CPU cost sometimes offset by IO reduction
MySQL User Conference 2009
Best practices – Innodb
• Follow general Innodb IO tunings
• Innodb provides checksum and compression
> So does ZFS
> But ZFS “self heals” instead of crashing :-)
• Disable innodb_doublewritebuffer to remove
redundant writes
> skip-innodb_doublewrite

MySQL User Conference 2009

Best practices – Backup/Restore
• ZFS snapshots are very cheap
• Need to quiesce the database before snapshot
> Flush tables with read lock
> Get a snapshot
> Unlock tables
• Zmanda Recovery Manager supports backup and
recovery using ZFS.
• ZFS clone is a read-write snapshot
> Useful in replication in a shared storage scenario

MySQL User Conference 2009

Applying best practices

MySQL User Conference 2009

ZFS COW penalty for table scans
• Since ZFS is Copy-on-write, Sequential scans will
be slower when compared to 'in-place' modification
• We ran sysbench read write for a week to study this
impact
> One hour sysbench read-write tes
> Followed by select count(*) from sbtest
> Repeat

MySQL User Conference 2009

ZFS COW penalty for table scans

Around 25% penalty after few hours

MySQL User Conference 2009
ZFS COW penalty for table scans
• Copying file over reorders the file in the optimal way.
• Idea for 'in-place' editing for db workloads
> Bug#6699230
• SSDs nullify this penalty

MySQL User Conference 2009

ZFS FAQ (for database people)
• ZFS needs more RAM – Not true
> Need one byte to cache one byte
> Metadata is usually 1% (and can be compressed)
• ZFS needs more CPU – Somewhat true
> Feature set provided is much stronger than for other FS
> Performance bugs are actively being fixed
> Checksumming is not free
> Some people want more CPU-hungry features (ex gzip)
> Industry benchmarks that run at 100% CPU utilization
will be at an disadvantage; however most customers
rarely run at 100% utilization
MySQL User Conference 2009
ZFS FAQ - DirectIO
• ZFS and Directio
> DirectIO is an overloaded term to mean several things
> No double caching – ZFS primarycache property
> Concurrent writes – ZFS Range locks
> Direct copy to application buffer – Not supported in ZFS
> No inflated IO – ZFS supports multiple recordsizes
> ZFS does not yet support the directio() hint.

MySQL User Conference 2009

More information
• ZFS Best Practices Guide
> http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

• ZFS Evil Tuning Guide

> http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

• Blogs
> http://blogs.sun.com/realneel
> http://blogs.sun.com/roch
• Mail to [email protected]

MySQL User Conference 2009

 Questions?
[email protected]

[email protected]
MySQL User Conference 2009