Distributed Systems

Principles and Paradigms

Maarten van Steen

VU Amsterdam, Dept. Computer Science

Room R4.20, [email protected]

Chapter 11: Distributed File Systems

Version: December 4, 2011

1 / 17

Contents

Chapter
01: Introduction
02: Architectures
03: Processes
04: Communication
05: Naming
06: Synchronization
07: Consistency & Replication
08: Fault Tolerance
09: Security
10: Distributed Object-Based Systems
11: Distributed File Systems
12: Distributed Web-Based Systems
13: Distributed Coordination-Based Systems

2 / 17 2 / 17

Distributed File Systems 11.1 Architecture Distributed File Systems 11.1 Architecture

Distributed File Systems

General goal
Try to make a file system transparently available to remote clients.

1. File moved to client

Client Server Client Server

Old file

New file

Requests from
client to access File stays 2. Accesses are
3. When client is done,
remote file on server done on client
file is returned to

Remote access model Upload/download model

3 / 17 3 / 17
 Distributed File Systems 11.1 Architecture Distributed File Systems 11.1 Architecture

Example: NFS Architecture

NFS
NFS is implemented using the Virtual File System abstraction, which is
now used for lots of different operating systems.

Client Server

System call layer System call layer

Virtual file system Virtual file system

(VFS) layer (VFS) layer

Local file Local file

system interface NFS client NFS server system interface

RPC client RPC server

stub stub

Network

4 / 17 4 / 17

Distributed File Systems 11.1 Architecture Distributed File Systems 11.1 Architecture

Example: NFS Architecture

Essence
VFS provides standard file system interface, and allows to hide
difference between accessing local or remote file system.

Question
Is NFS actually a file system?

5 / 17 5 / 17

Distributed File Systems 11.1 Architecture Distributed File Systems 11.1 Architecture

NFS File Operations

Oper. v3 v4 Description
Create Yes No Create a regular file
Create No Yes Create a nonregular file
Link Yes Yes Create a hard link to a file
Symlink Yes No Create a symbolic link to a file
Mkdir Yes No Create a subdirectory
Mknod Yes No Create a special file
Rename Yes Yes Change the name of a file
Remove Yes Yes Remove a file from a file system
Rmdir Yes No Remove an empty subdirectory
Open No Yes Open a file
Close No Yes Close a file
Lookup Yes Yes Look up a file by means of a name
Readdir Yes Yes Read the entries in a directory
Readlink Yes Yes Read the path name in a symbolic link
Getattr Yes Yes Get the attribute values for a file
Setattr Yes Yes Set one or more file-attribute values
Read Yes Yes Read the data contained in a file
Write Yes Yes Write data to a file
6 / 17 6 / 17
 Distributed File Systems 11.1 Architecture Distributed File Systems 11.1 Architecture

Cluster-Based File Systems

Observation
With very large data collections, following a simple client-server
approach is not going to work for speeding up file accesses, apply
striping techniques by which files can be fetched in parallel.

File block of file a File block of file e

a b c d e
a b c d e
a b c d e

Whole-file distribution

a b a b a b
c e c d c d
d e e

File-striped system
7 / 17 7 / 17

Distributed File Systems 11.1 Architecture Distributed File Systems 11.1 Architecture

Example: Google File System

file name, chunk index
GFS client Master
contact address

Instructions Chunk-server state

Chunk ID, range

Chunk server Chunk server Chunk server
Chunk data
Linux file Linux file Linux file
system system system

The Google solution

Divide files in large 64 MB chunks, and distribute/replicate chunks across
many servers:
The master maintains only a (file name, chunk server) table in main
memory minimal I/O
Files are replicated using a primary-backup scheme; the master is kept
out of the loop
8 / 17 8 / 17

Distributed File Systems 11.1 Architecture Distributed File Systems 11.1 Architecture

P2P-based File Systems

Node where a file system is rooted

File system layer Ivy Ivy Ivy

Block-oriented storage DHash DHash DHash

DHT layer Chord Chord Chord

Network

Basic idea
Store data blocks in the underlying P2P system:
Every data block with content D is stored on a node with hash h(D).
Allows for integrity check.
Public-key blocks are signed with associated private key and looked up
with public key.
A local log of file operations to keep track of hblockID, h(D)i pairs.

9 / 17 9 / 17
 Distributed File Systems 11.5 Synchronization Distributed File Systems 11.5 Synchronization

File sharing semantics

Client machine #1
Problem
When dealing with distributed file a b
systems, we need to take into account Process
A
the ordering of concurrent read/write a b c

operations and expected semantics 2. Write "c" 1. Read "ab"

(i.e., consistency).
File server
Original file
Single machine a b

a b
Process
A 3. Read gets "ab"
a b c
Client machine #2

Process
a b
B
Process
B
1. Write "c" 2. Read gets "abc"

(a) (b)
10 / 17 10 / 17

Distributed File Systems 11.5 Synchronization Distributed File Systems 11.5 Synchronization

File sharing semantics

Semantics
UNIX semantics: a read operation returns the effect of the last
write operation can only be implemented for remote access
models in which there is only a single copy of the file
Transaction semantics: the file system supports transactions on a
single file issue is how to allow concurrent access to a
physically distributed file
Session semantics: the effects of read and write operations are
seen only by the client that has opened (a local copy) of the file
what happens when a file is closed (only one client may actually
win)

11 / 17 11 / 17

Distributed File Systems 11.5 Synchronization Distributed File Systems 11.5 Synchronization

Example: File sharing in Coda

Essence
Coda assumes transactional semantics, but without the full-fledged
capabilities of real transactions. Note: Transactional issues reappear in
the form of this ordering could have taken place.

Session S A
Client

Open(RD) File f Invalidate

Close
Server

Close
Open(WR) File f

Client

Time
Session S B

12 / 17 12 / 17
 Distributed File Systems 11.6 Consistency and Replication Distributed File Systems 11.6 Consistency and Replication

Consistency and replication

Observation
In modern distributed file systems, client-side caching is the preferred
technique for attaining performance; server-side replication is done for fault
tolerance.

Observation
Clients are allowed to keep (large parts of) a file, and will be notified when
control is withdrawn servers are now generally stateful
1. Client asks for file
Client Server
2. Server delegates file
Old file

Local copy 3. Server recalls delegation

Updated file
4. Client sends returns file

13 / 17 13 / 17

Distributed File Systems 11.6 Consistency and Replication Distributed File Systems 11.6 Consistency and Replication

Example: Client-side caching in Coda

Session S A Session SA
Client A
Open(RD) Close Close
Open(RD)
Invalidate
Server File f (callback break) File f

File f OK (no file transfer)

Open(WR)
Open(WR) Close Close
Client B
Time
Session S B Session S B

Note
By making use of transactional semantics, it becomes possible to
further improve performance.

14 / 17 14 / 17

Distributed File Systems 11.6 Consistency and Replication Distributed File Systems 11.6 Consistency and Replication

Example: Server-side replication in Coda

Server Server
S1 S3

Client Broken Client

Server
A network B
S2

Main issue
Ensure that concurrent updates are detected:
Each client has an Accessible Volume Storage Group (AVSG): is a
subset of the actual VSG.
Version vector CVVi (f )[j] = k Si knows that Sj has seen version k of f .
Example: A updates f S1 = S2 = [+1, +1, +0]; B updates
f S3 = [+0, +0, +1].

15 / 17 15 / 17
 Distributed File Systems 11.7 Fault Tolerance Distributed File Systems 11.7 Fault Tolerance

High availability in P2P systems

Problem
There are many fully decentralized file-sharing systems, but because
churn is high (i.e., nodes come and go all the time), we may face an
availability problem replicate files all over the place (replication
factor: rrep ).

Alternative
Apply erasure coding:
Partition a file F into m fragments, and recode into a collection F
of n > m fragments
Property: any m fragments from F are sufficient to reconstruct F .
Replication factor: rec = n/m

16 / 17 16 / 17

Distributed File Systems 11.7 Fault Tolerance Distributed File Systems 11.7 Fault Tolerance

Replication vs. erasure coding

Comparison
With an average node availability a, 2.2
rrep
and required file unavailability , we rec 2.0
have for erasure coding:
1.8
rec m  
rec m i
1 = i
a (1 a)rec mi 1.6
i =m
1.4
and for file replication: 0.2 0.4 0.6 0.8 1
Node availability
1 = 1 (1 a)rrep

17 / 17 17 / 17