Ext3/4 file systems

Don Porter
CSE 506
 Logical Diagram
Binary Memory
Threads
Formats Allocators
User
Today’s Lecture
System Calls Kernel
RCU File System Networking Sync

Memory Device CPU

Management Drivers Scheduler

Hardware
Interrupts Disk Net Consistency
 Ext2 review
ò  Very reliable, “best-of-breed” traditional file system
design
ò  Much like the JOS file system you are building now
ò  Fixed location super blocks
ò  A few direct blocks in the inode, followed by indirect
blocks for large files
ò  Directories are a special file type with a list of file names
and inode numbers
ò  Etc.
 File systems and crashes
ò  What can go wrong?
ò  Write a block pointer in an inode before marking block as
allocated in allocation bitmap
ò  Write a second block allocation before clearing the first –
block in 2 files after reboot
ò  Allocate an inode without putting it in a directory –
“orphaned” after reboot
ò  Etc.
 Deeper issue
ò  Operations like creation and deletion span multiple on-
disk data structures
ò  Requires more than one disk write
ò  Think of disk writes as a series of updates
ò  System crash can happen between any two updates
ò  Crash between wrong two updates leaves on-disk data
structures inconsistent!
 Atomicity
ò  The property that something either happens or it doesn’t
ò  No partial results
ò  This is what you want for disk updates
ò  Either the inode bitmap, inode, and directory are updated
when a file is created, or none of them are
ò  But disks only give you atomic writes for a sector L

ò  Fundamentally hard problem to prevent disk corruptions

if the system crashes
 fsck
ò  Idea: When a file system is mounted, mark the on-disk
super block as mounted
ò  If the system is cleanly shut down, last disk write clears
this bit
ò  Reboot: If the file system isn’t cleanly unmounted, run
fsck

ò  Basically, does a linear scan of all bookkeeping and

checks for (and fixes) inconsistencies
 fsck examples
ò  Walk directory tree: make sure each reachable inode is
marked as allocated

ò  For each inode, check the reference count, make sure all
referenced blocks are marked as allocated

ò  Double-check that all allocated blocks and inodes are

reachable

ò  Summary: very expensive, slow scan of the entire file

system
 Journaling
ò  Idea: Keep a log of what you were doing
ò  If the system crashes, just look at data structures that
might have been involved
ò  Limits the scope of recovery; faster fsck!
 Undo vs. redo logging
ò  Two main choices for a journaling scheme (same in databases,
etc)

ò  Undo logging:

1) Write what you are about to do (and how to undo it)
ò  Synchronously
2) Then make changes on disk
3) Then mark the operations as complete
ò  If system crashes before commit record, execute undo steps
ò  Undo steps MUST be on disk before any other changes! Why?
 Redo logging
ò  Before an operation (like create)
1) Write everything that is going to be done to the log + a
commit record
ò  Sync
2) Do the updates on disk
3) When updates are complete, mark the log entry as obsolete
ò  If the system crashes during (2), re-execute all steps in
the log during fsck
 Which one?
ò  Ext3 uses redo logging
ò  Tweedie says for delete
ò  Intuition: It is easier to defer taking something apart than to
put it back together later
ò  Hard case: I delete something and reuse a block for something
else before journal entry commits
ò  Performance: This only makes sense if data comfortably fits
into memory
ò  Databases use undo logging to avoid loading and writing large
data sets twice
 Atomicity revisited
ò  The disk can only atomically write one sector

ò  Disk and I/O scheduler can reorder requests

ò  Need atomic journal “commit”

 Atomicity strategy
ò  Write a journal log entry to disk, with a transaction
number (sequence counter)
ò  Once that is on disk, write to a global counter that
indicates log entry was completely written
ò  This single write is the point at which a journal entry is
atomically “committed” or not
ò  Sometimes called a linearization point
ò  Atomic: either the sequence number is written or not;
sequence number will not be written until log entry on
disk
 Batching
ò  This strategy requires a lot of synchronous writes
ò  Synchronous writes are expensive
ò  Idea: let’s batch multiple little transactions into one
bigger one
ò  Assuming no fsync()
ò  For up to 5 seconds, or until we fill up a disk block in the
journal
ò  Then we only have to wait for one synchronous disk write!
 Complications
ò  We can’t write data to disk until the journal entry is
committed to disk
ò  Ok, since we buffer data in memory anyway
ò  But we want to bound how long we have to keep dirty
data (5s by default)
ò  JBD adds some flags to buffer heads that transparently
handles a lot of the complicated bookkeeping
ò  Pins writes in memory until journal is written
ò  Allows them to go to disk afterward
 More complications
ò  We also can’t write to the in-memory version until we’ve
written a version to disk that is consistent with the
journal

ò  Example:
ò  I modify an inode and write to the journal
ò  Journal commits, ready to write inode back
ò  I want to make another inode change
ò  Cannot safely change in-memory inode until I have either
written it to the file system or created another journal entry
 Another example
ò  Suppose journal transaction1 modifies a block, then
transaction 2 modifies the same block.

ò  How to ensure consistency?

ò  Option 1: stall transaction 2 until transaction 1 writes to fs
ò  Option 2 (ext3): COW in the page cache + ordering of
writes
 Yet more complications
ò  Interaction with page reclaiming:
ò  Page cache can pick a dirty page and tell fs to write it back
ò  Fs can’t write it until a transaction commits
ò  PFRA chose this page assuming only one write-back;
must potentially wait for several
ò  Advanced file systems need the ability to free another
page, rather than wait until all prerequisites are met
 Write ordering
ò  Issue, if I make file 1 then file 2, can I have a situation
where file 2 is on disk but not file 1?
ò  Yes, theoretically
ò  API doesn’t guarantee this won’t happen (journal
transactions are independent)
ò  Implementation happens to give this property by grouping
transactions into a large, compound transactions
(buffering)
 Checkpointing
ò  We should “garbage collect” our log once in a while
ò  Specifically, once operations are safely on disk, journal
transaction is obviated
ò  A very long journal wastes time in fsck
ò  Journal hooks associated buffer heads to track when they get
written to disk

ò  Advances logical start of the journal, allows reuse of those

blocks
 Journaling modes
ò  Full data + metadata in the journal

ò  Lots of data written twice, batching less effective, safer

ò  Ordered writes

ò  Only metadata in the journal, but data writes only allowed after
metadata is in journal
ò  Faster than full data, but constrains write orderings (slower)
ò  Metadata only – fastest, most dangerous

ò  Can write data to a block before it is properly allocated to a file

 Revoke records
ò  When replaying the journal, don’t redo these operations
ò  Mostly important for metadata-only modes
ò  Example: Once a file is deleted and the inode is reused,
revoke the creation record in the log
ò  Recreating and re-deleting could lose some data written to
the file
 ext3 summary
ò  A modest change: just tack on a journal

ò  Make crash recovery faster, less likely to lose data

ò  Surprising number of subtle issues

ò  You should be able to describe them
ò  And key design choices (like redo logging)
 ext4
ò  ext3 has some limitations that prevent it from handling
very large, modern data sets
ò  Can’t fix without breaking backwards compatibility
ò  So fork the code
ò  General theme: several changes to better handle larger
data
ò  Plus a few other goodies
 Example
ò  Ext3 fs limited to 16 TB max size
ò  32-bit block numbers (2^32 * 4k block size), or “address”
of blocks on disk
ò  Can’t make bigger block numbers on disk without
changing on-disk format
ò  Can’t fix without breaking backwards compatibility
ò  Ext4 – 48 bit block numbers
 Indirect blocks vs. extents
ò  Instead of represent each block, represent large
contiguous chunks of blocks with an extent

ò  More efficient for large files (both in space and disk
scheduling)

ò  Ex: Disk sectors 50—300 represent blocks 0—250 of file

ò  Vs.: Allocate and initialize 250 slots in an indirect block
ò  Deletion requires marking 250 slots as free
 Extents, cont.
ò  Worse for highly fragmented or sparse files
ò  If no 2 blocks are contiguous, will have an extent for each
block
ò  Basically a more expensive indirect block scheme
ò  Propose a block-mapped extent, which essentially reverts
to a more streamlined indirect block
 Static inode allocations
ò  When you create an ext3 or ext4 file system, you create
all possible inodes

ò  Disk blocks can either be used for data or inodes, but
can’t change after creation

ò  If you need to create a lot of files, better make lots of

inodes

ò  Why?
 Why?
ò  Simplicity

ò  Fixed location inodes means you can take inode number, total
number of inodes, and find the right block using math
ò  Dynamic inodes introduces another data structure to track this
mapping, which can get corrupted on disk (losing all contained
files!)
ò  Bookkeeping gets a lot more complicated when blocks change
type
ò  Downside: potentially wasted space if you guess wrong
number of files
 Directory scalability
ò  An ext3 directory can have a max of 32,000 sub-
directories/files
ò  Painfully slow to search – remember, this is just a simple
array on disk (linear scan to lookup a file)
ò  Replace this in ext4 with an HTree
ò  Hash-based custom BTree
ò  Relatively flat tree to reduce risk of corruptions
ò  Big performance wins on large directories – up to 100x
 Other goodies
ò  Improvements to help with locality
ò  Preallocation and hints keep blocks that are often accessed
together close on the disk
ò  Checksumming of disk blocks is a good idea
ò  Especially for journal blocks
ò  Fsck on a large fs gets expensive
ò  Put used inodes at front if possible, skip large swaths of
unused inodes if possible
 Summary
ò  ext2 – Great implementation of a “classic” file system

ò  ext3 – Add a journal for faster crash recovery and less
risk of data loss

ò  ext4 – Scale to bigger data sets, plus other features

ò  Total FS size (48-bit block numbers)
ò  File size/overheads (extents)
ò  Directory size (HTree vs. a list)