Google File System

Introduction
 Originated in 2003.
 GFS is designed for system to system interaction,
not user to system.
 Network of inexpensive machines running on
Linux operating systems
 Motivation
 Google needed a good distributed file system
 Redundant storage of massive amounts of data on
cheap and unreliable computers

 Why not use an existing file system?

 Google’s problems are different from anyone else’s
• Different workload and design priorities
 GFS is designed for Google apps and workloads
 Google apps are designed for GFS
 Google Workload Characteristics
 Most files are mutated by appending new data –
large sequential writes
 Random writes are very uncommon
 Files are written once, then they are only read
 Reads are sequential
 Large streaming reads and small random reads
 High sustained throughput favoured over low
latency
 Google Workload Characteristics
 Google applications:
 Data analysis programs that scan through data repositories
 Data streaming applications
 Archiving
 Applications producing (intermediate) search results
 Assumptions
 High component failure rates
 Inexpensive commodity components fail all the time
 “Modest” number of HUGE files
 Justa few million
 Each is 100MB or larger; multi-GB files typical
 GFS Design Decisions
 Files stored as chunks
 Fixed size (64MB)
 Single master to coordinate access, keep metadata
 Simple centralized management
 Application Programming Interface is different
than that found for NFS
 GFS Architecture
 Single master for a cluster
 Multiple chunkservers
 Chunks
 Files are divided into fixed-size chunks
 Each chunk has an identifier, chunk handle,
assigned by the master at the time of chunk
creation
 Each chunk is replicated 3 times
 Chunk size is fixed size (64MB)
 Chunkservers
 Store chunks on local disks as Linux files
 Read/write chunk data specified by a chunk handle
and byte range
 Master
 Stores metadata
 The file and chunk namespaces
 Mapping from files to chunks
 Locations of each chunk’s replicas (referred to as chunk
locations)
 Interacts with clients
 Creates chunk replicas
 Master
 Orchestrates chunk modifications across multiple
replicas
 Ensures atomic concurrent appends
 Locks concurrent operations

 Deletes old files (via garbage collection)

 Interactions
 Assume a read
 The client translates the file name and byte offset
specified by the application into a chunk index
within the file
 The master replies with the corresponding chunk
handle (information needed to find a chunk) and
locations of the replicas.
 The client then sends a request to one of the
replicas most likely the closest one
 Note: Further reads of the same chunk require no
more client-master interaction
 Chunk Size
 Chunk size is 64 MB
 Larger than typical file system block sizes
 Advantages
 Reduces a client’s need to interact with the master
 Reduce network overhead by keeping a persistent TCP
connection to the chunkserver over a period of time
 Reduces the size of the metadata stored on the master
 Chunk Size
 Disadvantages
A small file consists of a small number of chunks
 The chunkservers storing these chunks may become hot
spots if many clients are accessing the same file
• Does not occur very much in practice
 Metadata On Master
 Metadata – Information about the data:
 File
names
 Mapping of file names to chunk IDs
 Chunk locations

 Metadata is kept in memory

 File names and chunk mappings are also kept
persistent in an operation log
 Chunk locations are kept in memory only
 Itwill be lost during the crash
 The master asks chunk servers about their chunks at
startup – builds a table of chunk locations
 Why Keep Metadata In Memory?
 To keep master operations fast
 Master can periodically scan its internal state in
the background, in order to implement:
 Re-replication
(in case of chunk server failures)
 Chunk migration (for load balancing)
 Why Not Keep Chunk Locations
Persistent?
 Chunk location – which chunk server has a replica
of a given chunk
 Master polls chunk servers for that information on
startup
 Thereafter, master keeps itself up-to-date:
 It controls all initial chunk placement, migration and re-
replication
 It monitors chunkserver status with regular HeartBeat
messages
 Why Not Keep Chunk Locations
Persistent?
 Motivation: simplicity
 Eliminates the need to keep master and
chunkservers synchronized
 Synchronization would be needed when
chunkservers:
 Join and leave the cluster
 Change names
 Fail and restart
 Operation Log
 Historical record of metadata changes
 Maintains logical order of concurrent operations
 Log is used for recovery – the master replays it in
the event of failures
 Master periodically checkpoints the log
 Updates of Replicated Data
 Each mutation (modification) is performed at all
the replicas
 Modifications are applied in the same order across
all replicas
 For each chunk there is a primary chunk
 The primary picks a serial order for all mutations
to the chunk
 The client pushes data to all replicas
 The primary tells the replicas in which order they
should apply modifications
Updates of Replicated Data (cont.)

1. Client asks master for replica

locations
2. Master responds
3. Client pushes data to all replicas;
replicas store it in a buffer cache
4. Client sends a write request to the
primary (identifying the data that
had been pushed)
5. Primary forwards request to the
secondaries (identifies the order)
6. The secondaries respond to the
primary
7. The primary responds to the client
 Failure Handling During Updates
 If a write fails at the primary:
 The primary may report failure to the client – the client
will retry
 If the primary does not respond, the client retries from
Step 1 by contacting the master
 If a write succeeds at the primary, but fails at
several replicas
 The client retries several times (Step 3-7)
 Atomic Record Appends
 Atomic append is a write – but GFS (the primary
replica) chooses the offset where the append
happens
 This way GFS can decide on serial order of
concurrent appends without client synchronization
 Useful when there are a lot of file appends from
multiple clients and it doesn’t matter in what order
the appends are done
 Atomic Record Appends
 The client pushes the data (a record) to all
replicas of the last chunk of the file
 Sends request to primary
 Case 1: Record fits within the maximum size of
chunk
 Primary appends data to its replica
 Primary tells the secondaries to write the data at the
exact offset
 Atomic Record Appends
 Case 2: What if appending the record would cause
the chunk to exceed the maximum size
 Pad the chunk to the maximum size
 Tell the secondaries to do the same thing
 Reply to client indicating that the operation should be
retried on the next chunk
 Deployment in Google
 Many GFS clusters
 Hundreds/thousands of storage nodes
 Managing petabytes of data
 Summary
 GFS demonstrates how to support large-scale
processing workloads on commodity hardware
 design to tolerate frequent component failures
 optimize for huge files that are mostly appended and
read
 feel free to relax and extend FS interface as required
 go for simple solutions (e.g., single master)

 GFS has met Google’s storage needs