GFS: The Google File System

Michael Siegenthaler
Cornell Computer Science

CS 6464
10th March 2009
 Motivating Application: Search

• Crawl the whole web

• Store it all on
“one big disk”
• Process users’ searches
on “one big CPU”

• $$$$$
• Doesn’t scale

2
 Google Platform Characteristics

• Lots of cheap PCs, each with disk and CPU

– High aggregate storage capacity
– Spread search processing across many
CPUs
• How to share data among PCs?
 Google Platform Characteristics

• 100s to 1000s of PCs in cluster

• Many modes of failure for each PC:
– App bugs, OS bugs
– Human error
– Disk failure, memory failure, net failure,
power supply failure
– Connector failure
• Monitoring, fault tolerance, auto-recovery
essential
6
 Google File System: Design Criteria

• Detect, tolerate, recover from failures

automatically
• Large files, >= 100 MB in size
• Large, streaming reads (>= 1 MB in size)
– Read once
• Large, sequential writes that append
– Write once
• Concurrent appends by multiple clients (e.g.,
producer-consumer queues)
– Want atomicity for appends without synchronization
overhead among clients
8
 GFS: Architecture

• One master server (state replicated on

backups)
• Many chunk servers (100s – 1000s)
– Spread across racks; intra-rack b/w greater
than inter-rack
– Chunk: 64 MB portion of file, identified by 64-
bit, globally unique ID
• Many clients accessing same and different
files stored on same cluster

9
GFS: Architecture (2)

10
 Master Server

• Holds all metadata:

– Namespace (directory hierarchy)
– Access
Holds control information
all metadata (per-file)
in RAM; very fast
operations
– Mappingon filefiles
from system metadata
to chunks
– Current locations of chunks (chunkservers)
• Delegates consistency management
• Garbage collects orphaned chunks
• Migrates chunks between chunkservers
 Chunkserver

• Stores 64 MB file chunks on local disk

using standard Linux filesystem, each with
version number and checksum
• Read/write requests specify chunk handle
and byte range
• Chunks replicated on configurable number
of chunkservers (default: 3)
• No caching of file data (beyond standard
Linux buffer cache)
12
 Client

• Issues control (metadata) requests to

master server
• Issues data requests directly to
chunkservers
• Caches metadata
• Does no caching of data
– No consistency difficulties among clients
– Streaming reads (read once) and append
writes (write once) don’t benefit much from
caching at client
13
 Client API

• Not a filesystem in traditional sense

– Not POSIX compliant
– Does not use kernel VFS interface
– Library that apps can link in for storage
access
• API:
– open, delete, read, write (as expected)
– snapshot: quickly create copy of file
– append: at least once, possibly with gaps
and/or inconsistencies among clients
14
 Client Read

• Client sends master:

– read(file name, chunk index)
• Master’s reply:
– chunk ID, chunk version number, locations of replicas
• Client sends “closest” chunkserver w/replica:
– read(chunk ID, byte range)
– “Closest” determined by IP address on simple rack-
based network topology
• Chunkserver replies with data

15
 Client Write

• Some chunkserver is primary for each chunk

– Master grants lease to primary (typically for 60 sec.)
– Leases renewed using periodic heartbeat messages
between master and chunkservers
• Client asks master for primary and secondary
replicas for each chunk
• Client sends data to replicas in daisy chain
– Pipelined: each replica forwards as it receives
– Takes advantage of full-duplex Ethernet links

16
Client Write (2)

17
 Client Write (3)

• All replicas acknowledge data write to client

• Client sends write request to primary
• Primary assigns serial number to write request,
providing ordering
• Primary forwards write request with same serial
number to secondaries
• Secondaries all reply to primary after completing
write
• Primary replies to client

18
 Client Record Append

• Google uses large files as queues between

multiple producers and consumers
• Same control flow as for writes, except…
• Client pushes data to replicas of last chunk of
file
• Client sends request to primary
• Common case: request fits in current last chunk:
– Primary appends data to own replica
– Primary tells secondaries to do same at same byte
offset in theirs
– Primary replies with success to client
19
 Client Record Append (2)

• When data won’t fit in last chunk:

– Primary fills current chunk with padding
– Primary instructs other replicas to do same
– Primary replies to client, “retry on next chunk”
• If record append fails at any replica, client
retries operation
– So replicas of same chunk may contain different data
—even duplicates of all or part of record data
• What guarantee does GFS provide on
success?
– Data written at least once in atomic unit
20
 GFS: Consistency Model

• Changes to namespace (i.e., metadata) are

atomic
– Done by single master server!
– Master uses log to define global total order of
namespace-changing operations

21
 GFS: Consistency Model (2)

• Changes to data are ordered as chosen by a

primary
– All replicas will be consistent
– But multiple writes from the same client may be
interleaved or overwritten by concurrent operations
from other clients
• Record append completes at least once, at
offset of GFS’s choosing
– Applications must cope with possible
duplicates
22
GFS: Data Mutation Consistency

Write Record Append

serial defined
success defined
interspersed with
concurrent consistent
inconsistent
success but
undefined
failure inconsistent

23
 Applications and
Record Append Semantics
• Applications should use self-describing
records and checksums when using
Record Append
– Reader can identify padding / record
fragments
• If application cannot tolerate duplicated
records, should include unique ID in
record
– Reader can use unique IDs to filter duplicates

24
 Logging at Master

• Master has all metadata information

– Lose it, and you’ve lost the filesystem!
• Master logs all client requests to disk
sequentially
• Replicates log entries to remote backup
servers
• Only replies to client after log entries safe
on disk on self and backups!

25
 Chunk Leases and Version Numbers

• If no outstanding lease when client

requests write, master grants new one
• Chunks have version numbers
– Stored on disk at master and chunkservers
– Each time master grants new lease,
increments version, informs all replicas
• Master can revoke leases
– e.g., when client requests rename or snapshot
of file
26
 What If the Master Reboots?

• Replays log from disk

– Recovers namespace (directory) information
– Recovers file-to-chunk-ID mapping
• Asks chunkservers which chunks they hold
– Recovers chunk-ID-to-chunkserver mapping
• If chunk server has older chunk, it’s stale
– Chunk server down at lease renewal
• If chunk server has newer chunk, adopt its
version number
– Master may have failed while granting lease
27
 What if Chunkserver Fails?

• Master notices missing heartbeats

• Master decrements count of replicas for all
chunks on dead chunkserver
• Master re-replicates chunks missing
replicas in background
– Highest priority for chunks missing greatest
number of replicas

28
 File Deletion

• When client deletes file:

– Master records deletion in its log
– File renamed to hidden name including deletion
timestamp
• Master scans file namespace in background:
– Removes files with such names if deleted for longer
than 3 days (configurable)
– In-memory metadata erased
• Master scans chunk namespace in background:
– Removes unreferenced chunks from chunkservers

29
 Limitations

• Security?
– Trusted environment, trusted users
– But that doesn’t stop users from interfering
with each other…
• Does not mask all forms of data corruption
– Requires application-level checksum

30
31
32
33
34
35
(for a single file)

36
 Recovery Time

• Experiment: killed 1 chunkserver

– Clonings limited to 40% of the chunkservers
and 50 Mbps each to limit impact on
applications
– All chunks restored in 23.2 min
• Experiment: killed 2 chunkservers
– 266 of 16,000 chunks reduced to single
replica
– Higher priority re-replication for these chunks
– Achieved 2x replication within 2 min 37
38
39
 GFS: Summary
• Success: used actively by Google to support
search service and other applications
– Availability and recoverability on cheap hardware
– High throughput by decoupling control and data
– Supports massive data sets and concurrent appends
• Semantics not transparent to apps
– Must verify file contents to avoid inconsistent regions,
repeated appends (at-least-once semantics)
• Performance not good for all apps
– Assumes read-once, write-once workload (no client
caching!)
40