Lecture24 DFS PartI 25nov 2014
Lecture24 DFS PartI 25nov 2014
CS 15-440
Mohammad Hammoud
1
Today…
Last Session:
Fault Tolerance- Part II
Today’s Session:
Distributed File Systems – Part I
Announcements:
PS4 grades will be out tomorrow
Project 4 is due on Dec 3rd by midnight
PS5 is due on Dec 4th by midnight
Final Exam is on Monday Dec 8th at 9:00AM in Room 1031. It will
be comprehensive, but open book and notes
2
Intended Learning Outcomes:
Distributed File Systems
Considered: a reasonably critical and ILO7 Explain distributed file systems as a paradigm for
comprehensive perspective.
general-purpose distributed systems, and analyze
Thoughtful: Fluent, flexible and efficient
perspective. its various aspects and architectures
Masterful: a powerful and illuminating
perspective.
ILO7.1 Define distributed file systems (DFSs) and explain various
architectures
ILO7 ILO7.2 Analyze various aspects of DFSs including
processes, communication, naming,
ILO ILO
synchronization, consistency and replication, and
7.1 7.2 fault tolerance
Discussion on Distributed File Systems
4
Distributed File Systems
Why File Systems?
Client Server
DFS Component Placement (2)
Three basic concerns govern the DFS components
placement strategy:
11
DFS Aspects
Aspect Description
Architecture How are DFSs generally organized?
Processes • Who are the cooperating processes?
• Are processes stateful or stateless?
Communication • What is the typical communication paradigm followed
by DFSs?
• How do processes in DFSs communicate?
Naming How is naming often handled in DFSs?
Synchronization What are the file sharing semantics adopted by DFSs?
Consistency and Replication What are the various features of client-side caching as well
as server-side replication?
Fault Tolerance How is fault tolerance handled in DFSs?
DFS Aspects
Aspect Description
Architecture How are DFSs generally organized?
Processes • Who are the cooperating processes?
• Are processes stateful or stateless?
Communication • What is the typical communication paradigm followed
by DFSs?
• How do processes in DFSs communicate?
Naming How is naming often handled in DFSs?
Synchronization What are the file sharing semantics adopted by DFSs?
Consistency and Replication What are the various features of client-side caching as well
as server-side replication?
Fault Tolerance How is fault tolerance handled in DFSs?
Architectures
Client-Server Distributed File Systems
Cluster-Based Distributed File Systems
Symmetric Distributed File Systems
Architectures
Client-Server Distributed File Systems
Cluster-Based Distributed File Systems
Symmetric Distributed File Systems
Network File System
Many distributed file systems are organized along the lines of
client-server architectures
NFS comes with a protocol that describes precisely how a client can
access a file stored on a (remote) NFS file server
Replies from
server
Server
Client
File
Requests from
client to access
remote file File stays
on server
In this model, clients:
Are offered transparent accesses to a file system that is managed by a
remote server
Are normally unaware of the actual location of files
Are offered an interface to a file system similar to the interface offered by a
conventional local file system
Upload/Download Model
A contrary model, referred to as upload/download model, allows a
client to access a file locally after having downloaded it
from the server
File moved to client
Server
Client
File
New File
The Internet’s FTP service can be used this way when a client
downloads a complete file, modifies it, and then puts it back
The Basic NFS Architecture
Client Server
Network
Architectures
Client-Server Distributed File Systems
Cluster-Based Distributed File Systems
Symmetric Distributed File Systems
Data-Intensive Applications
Today there is a deluge of large data-intensive applications
The cluster-based file system divides and distributes Big Data, using
file striping techniques, for allowing concurrent data accesses
Examples:
Cloud Computing Oriented: Google File System (GFS)
HPC Oriented: Parallel Virtual File System (PVFS)
File Striping Techniques
Server clusters are often used for distributed applications and their
associated file systems are adjusted to satisfy their requirements
a b d e a
a
c e c b b
d
c
d e
Round-Robin Distribution (1)
How to stripe a file over multiple machines?
Round-Robin is typically a reasonable default solution
Logical File
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Stripe Size
Striping Unit
0 4 8 12 1 5 9 13 2 6 10 14 3 7 11 15
Client I: 512K write, offset 0 Client II: 512K write, offset 512
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15
0 4 8 12 1 5 9 13 2 6 10 14 3 7 11 15
Logical File
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Stripe Size
Striping Unit
0 2 4 6 1 3 5 7 8 10 12 14 9 11 13 15
Group Size = 2
2D Round-Robin Distribution (2)
2D distribution can limit the number of servers per client
Client I: 512K write, offset 0 Client II: 512K write, offset 512
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15
0 2 4 6 1 3 5 7 8 10 12 14 9 11 13 15
Group Size = 2
GFS Data Distribution Policy
The Google File System (GFS) is a scalable DFS for data-intensive
applications
GFS divides large files into multiple pieces called chunks or blocks
(by default 64MB) and stores them on different data servers
This design is referred to as block-based design
Each GFS chunk has a unique 64-bit identifier and is stored as a file
in the lower-layer local file system on the data server
Load Imbalance
GFS Architecture
The storage and compute capabilities of a cluster are organized in
two ways:
1. GFS
Co-locate storage and compute in the same node
2. Separate storage nodes from compute nodes
File name, chunk index
Contact address
PVFS divides large files into multiple pieces called stripe units (by
default 64KB) and stores them on different data servers
This design is referred to as object-based design
0M
Blk Blk Blk Blk
64M 0 0 0 1
Blk Blk Blk Blk
128M 1 1 2 2
Blk Blk Blk Blk
192M 2 3 3 3
Blk Blk Blk Blk
256M 4 4 4 5
Blk Blk Blk Blk
5 5 6 6
320M
Blk
384M 6
Load Balance
PVFS Architecture
The storage and compute capabilities of a cluster are organized in
two ways:
1. Co-locate storage and compute in the same node
2. Separate storage nodesPVFS
from compute nodes
Metadata
Manager
Architectures
Client-Server Distributed File Systems
Cluster-Based Distributed File Systems
Symmetric Distributed File Systems
Ivy
Fully symmetric organizations that are based on peer-to-peer
technology also exist
E.g., NFSv4
DFS Aspects
Aspect Description
Architecture How are DFSs generally organized?
Processes • Who are the cooperating processes?
• Are processes stateful or stateless?
Communication • What is the typical communication paradigm followed
by DFSs?
• How do processes in DFSs communicate?
Naming How is naming often handled in DFSs?
Synchronization What are the file sharing semantics adopted by DFSs?
Consistency and Replication What are the various features of client-side caching as well
as server-side replication?
Fault Tolerance How is fault tolerance handled in DFSs?
Communication in DFSs
Communication in DFSs is mainly based on remote procedure
calls (RPCs)
GFS uses RPC and may break a read into multiple RPCs to
increase parallelism
Lookup
Up until NFSv4, the client was made responsible
for making the server’s life as easy as possible
Lookup name
by keeping requests simple
Read
Time
The drawback becomes apparent when considering
Read file data
the use of NFS in a wide-area system
Time
Read file data
RPC in Coda: RPC2
Another enhancement to RPCs has been developed as part of the
Coda file system (Kistler and Satyanarayanan, 1992) and referred
to as RPC2
Each time an RPC is made, the RPC2 client code starts a new thread
that sends an invocation request to the server then blocks until it
receives an answer
Client Client
Invalidate Invalidate
Reply Reply
Client Client
Time Time