DISTRIBUTED FILE SYSTEM

DISTRIBUTED FILE
SYSTEMS

Topics

DISTRIBUTED FILE SYSTEM

Introduction
File Service Architecture
DFS: Case Studies
Case Study: Sun NFS
Case Study: The Andrew File System

Introduction

DISTRIBUTED FILE SYSTEM

File system were originally developed for

centralized computer systems and
desktop computers.
File system was as an operating system
facility providing a convenient
programming interface to disk storage.

Introduction

DISTRIBUTED FILE SYSTEM

Distributed file systems support the

sharing of information in the form of files
and hardware resources.
With the advent of distributed object
systems (CORBA, Java) and the web, the
picture has become more complex.
Figure 1 provides an overview of types of
storage system.

Introduction

DISTRIBUTED FILE SYSTEM

Sharing Persis Distributed

Consistency Example
tence
cache/replicas maintenance
Mainmemory

1
1

Filesystem
Distributedfilesystem

RAM
UNIXfilesystem

Web

SunNFS
Webserver

Distributedsharedmemory

Ivy(Ch.18)

Remoteobjects(RMI/ORB)
Persistentobjectstore

1
1

CORBA
CORBAPersistent
ObjectService
OceanStore(Ch.10)

Peertopeerstoragesystem

Figure 1. Storage systems and their properties

Typesofconsistencybetweencopies:

1strictonecopyconsistency
approximateconsistency
Xnoautomaticconsistency

Introduction

DISTRIBUTED FILE SYSTEM

Figure 2 shows a typical layered module

structure for the implementation of a nondistributed file system in a conventional
operating system.

Introduction

DISTRIBUTED FILE SYSTEM

Directory module:

relates file names to file IDs

File module:

relates file IDs to particular files

Access control module:

checks permission for operation requested

File access module:

reads or writes file data or attributes

Block module:

accesses and allocates disk blocks

Device module:

disk I/O and buffering

Figure 2. File system modules

7

Introduction

DISTRIBUTED FILE SYSTEM

File systems are responsible for the

organization, storage, retrieval, naming,
sharing and protection of files.
Files contain both data and attributes.
A typical attribute record structure is
illustrated in Figure 3.

Introduction

DISTRIBUTED FILE SYSTEM

File length
Creation timestamp
Read timestamp
Write timestamp
Attribute timestamp
Reference count
Owner
File type
Access control list

Figure 3. File attribute record structure

9

Introduction

DISTRIBUTED FILE SYSTEM

Figure 4 summarizes the main operations

on files that are available to applications in
UNIX systems.

10

Introduction

DISTRIBUTED FILE SYSTEM

Figure 4. UNIX file system operations

filedes=open(name,mode)
filedes=creat(name,mode)
status=close(filedes)
count=read(filedes,buffer,n)
count=write(filedes,buffer,n)
pos=lseek(filedes,offset,
whence)
status=unlink(name)
status=link(name1,name2)
status=stat(name,buffer)

Opensanexistingfilewiththegivenname.
Createsanewfilewiththegivenname.
Bothoperationsdeliverafiledescriptorreferencingtheopen
file.Themodeisread,writeorboth.
Closestheopenfilefiledes.
Transfersnbytesfromthefilereferencedbyfiledestobuffer.
Transfersnbytestothefilereferencedbyfiledesfrombuffer.
Bothoperationsdeliverthenumberofbytesactuallytransferred
andadvancethereadwritepointer.
Movesthereadwritepointertooffset(relativeorabsolute,
dependingonwhence).
Removesthefilenamefromthedirectorystructure.Ifthefile
hasnoothernames,itisdeleted.
Addsanewname(name2)forafile(name1).
Getsthefileattributesforfilenameintobuffer.

11

Introduction

DISTRIBUTED FILE SYSTEM

Distributed File system requirements

Related requirements in distributed file systems
are:

Transparency
Concurrency
Replication
Heterogeneity
Fault tolerance
Consistency
Security
Efficiency

12

DISTRIBUTED FILE SYSTEM

Transparency

Access transparency: Client programs should be unaware

of the distribution of files. A single set of operations is
provided for access to local and remote files. Programs
Location transparency: Client programs should see a
uniform file name space.
Mobility transparency: Neither client programs nor system
administration tables in client nodes need to be changed
when files are moved.
Performance transparency: Client programs should
continue to perform satisfactorily while the load on the
service varies within a specified range.
Scaling transparency: The service can be expanded by
incremental growth to deal with a wide range of loads and
network sizes.
13

DISTRIBUTED FILE SYSTEM

Distributed File system requirement

Concurrent file updates Changes to a file by one client

should not interfere with the operation of other clients
simultaneously accessing or changing the same file
File replication In a file service that supports replication, a
file may be represented by several copies of its contents at
different locations.
Hardware and operating system heterogeneity The
service interfaces should be defined so that client and server
software can be implemented for different operating systems
and computers.
Fault tolerance The central role of the file service in
distributed systems makes it essential that the service
continue to operate in the face of client and server failures.
14

DISTRIBUTED FILE SYSTEM

Distributed File system requirement

Consistency: The file contents seen by all of the processes

accessing or updating a given file are those that they would
see if only a single copy of the file contents existed
Security: In distributed file systems, there is a need to
authenticate client requests so that access control at the
server is based on correct user identities and to protect the
contents of request and reply messages with digital
signatures and (optionally) encryption of secret data.
Efficiency A distributed file service should offer facilities
that are of at least the same power and generality as those
found in conventional file systems and should achieve a
comparable level of performance.
15

DISTRIBUTED FILE SYSTEM

File Service Architecture

An architecture that offers a clear separation of

the main concerns in providing access to files is
obtained by structuring the file service as three
components:
A flat file service
A directory service
A client module.

The relevant modules and their relationship is

shown in Figure 5.

16

DISTRIBUTED FILE SYSTEM

File Service Architecture

Client computer
Application Application
program
program

Server computer
Directory service

Flat file service

Client module

Figure 5. File service architecture

17

DISTRIBUTED FILE SYSTEM

File Service Architecture

The Client module implements exported

interfaces by flat file and directory services on
server side.
Responsibilities of various modules can be
defined as follows:
Flat file service:
Concerned with the implementation of operations on the
contents of file. Unique File Identifiers (UFIDs) are used
to refer to files in all requests for flat file service
operations. UFIDs are long sequences of bits chosen so
that each file has a unique among all of the files in a
distributed system.
18

DISTRIBUTED FILE SYSTEM

File Service Architecture

Directory service:

Provides mapping between text names for the

files and their UFIDs. Clients may obtain the
UFID of a file by quoting its text name to directory
service. Directory service supports functions
needed generate directories, to add new files to
directories.

19

DISTRIBUTED FILE SYSTEM

File Service Architecture

Client module:

It runs on each computer and provides integrated

service (flat file and directory) as a single API to
application programs. For example, in UNIX
hosts, a client module emulates the full set of
Unix file operations.
It holds information about the network locations
of flat-file and directory server processes; and
achieve better performance through
implementation of a cache of recently used file
blocks at the client.
20

DISTRIBUTED FILE SYSTEM

File Service Architecture

Flat file service interface:

Figure 6 contains a definition of the interface to a flat

file service.

21

DISTRIBUTED FILE SYSTEM

File Service Architecture

22

DISTRIBUTED FILE SYSTEM

File Service Architecture

Access control
In distributed implementations, access
rights checks have to be performed at
the server because the server RPC
interface is an otherwise unprotected
point of access to files.
Directory service interface
Figure 7 contains a definition of the RPC
interface to a directory service.

23

DISTRIBUTED FILE SYSTEM

File Service Architecture

24

DISTRIBUTED FILE SYSTEM

File Service Architecture

Hierarchic file system
A hierarchic file system such as the one that UNIX provides
consists of a number of directories arranged in a tree structure.

File Group
A file group is a collection of files that can be located on
any server or moved between servers while maintaining
the same names.
A similar construct is used in a UNIX file system.
It helps with distributing the load of file serving
between several servers.
File groups have identifiers which are unique
throughout the system (and hence for an open
system, they must be globally unique).

25

DISTRIBUTED FILE SYSTEM

File Service Architecture

To construct a globally unique ID we use some unique

attribute of the machine on which it is created, e.g. IP
number, even though the file group may move
subsequently.

File Group ID:

32bits

16bits

IPaddress

date

26

DISTRIBUTED FILE SYSTEM

DFS: Case Studies

NFS (Network File System)

Developed by Sun Microsystems (in 1985)

Most popular, open, and widely used.
NFS protocol standardized through IETF (RFC 1813)

AFS (Andrew File System)

Developed by Carnegie Mellon University as part of Andrew
distributed computing environments (in 1986)
A research project to create campus wide file system.
Public domain implementation is available on Linux
(LinuxAFS)
It was adopted as a basis for the DCE/DFS file system in
the Open Software Foundation (OSF, www.opengroup.org)
DEC (Distributed Computing Environment

27

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Figure 8 shows the architecture of Sun NFS.

28

DISTRIBUTED FILE SYSTEM

NFS architecture
Client computer

Server computer

Application Application
program
program
UNIX
system calls
Virtual file system

Operations
on local files
UNIX
file
system

Other
file system

UNIX kernel

UNIX kernel
Operations
on
remote files

NFS
client

Figure 8. NFS architecture

Virtual file system

NFS
server

UNIX
file
system

NFS protocol
(remote operations)

29

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

The file identifiers used in NFS are called

file handles.
fh=filehandle:
Filesystemidentifier inodenumber inodegeneration

30

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

A simplified representation of the RPC

interface provided by NFS version 3
servers is shown in Figure 9.

31

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

A simplified representation of the RPC

interface provided by NFS version 3
servers is shown in Figure 9.

32

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

NFS access control and authentication

The NFS server is stateless server, so the user's identity
and access rights must be checked by the server on
each request.
In the local file system they are checked only on the files access
permission attribute.

Every client request is accompanied by the userID and

groupID
It is not shown in the Figure 8.9 because they are inserted by the
RPC system.

Kerberos has been integrated with NFS to provide a

stronger and more comprehensive security solution.

33

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Mount service
Mount operation:

mount(remotehost, remotedirectory, localdirectory)

Server maintains a table of clients who have mounted

filesystems at that server.
Each client maintains a table of mounted file systems
holding:
< IP address, port number, file handle>

Remote file systems may be hard-mounted or softmounted in a client computer.

Figure 10 illustrates a Client with two remotely mounted
file stores.

34

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Server 1

Client
(root)

(root)

export

...

(root)

usr

nfs

Remote

people

big jon bob

vmunix

Server 2

mount
...

Remote
students

staff

mount

users

jim ann jane joe

Note:Thefilesystemmountedat/usr/studentsintheclientisactuallythesub
treelocatedat/export/peopleinServer1;
thefilesystemmountedat/usr/staffintheclientisactuallythesubtree
locatedat/nfs/usersinServer2.
35
Figure 10. Local and remote file systems accessible on an NFS client

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Automounter
The automounter was added to the UNIX implementation
of NFS in order to mount a remote directory dynamically
whenever an empty mount point is referenced by a
client.
Automounter has a table of mount points with a
reference to one or more NFS servers listed against
each.
it sends a probe message to each candidate server and
then uses the mount service to mount the filesystem at
the first server to respond.

Automounter keeps the mount table small.

36

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Automounter Provides a simple form of replication
for read-only filesystems.
E.g. if there are several servers with identical copies
of /usr/lib then each server will have a chance of
being mounted at some clients.

37

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Server caching
Similar to UNIX file caching for local files:
pages (blocks) from disk are held in a main memory
buffer cache until the space is required for newer
pages. Read-ahead and delayed-write optimizations.
For local files, writes are deferred to next sync event
(30 second intervals).
Works well in local context, where files are always
accessed through the local cache, but in the remote
case it doesn't offer necessary synchronization
guarantees to clients.

38

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

NFS v3 servers offers two strategies for
updating the disk:
Write-through - altered pages are written to
disk as soon as they are received at the
server. When a write() RPC returns, the
NFS client knows that the page is on the
disk.
Delayed commit - pages are held only in the
cache until a commit() call is received for
the relevant file. This is the default mode
used by NFS v3 clients. A commit() is
issued by the client whenever a file is
closed.
39

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Client caching
Server caching does nothing to reduce
RPC traffic between client and server
further optimization is essential to reduce
server load in large networks.
NFS client module caches the results of
read, write, getattr, lookup and readdir
operations
synchronization of file contents (one-copy
semantics) is not guaranteed when two or
more clients are sharing the same file.

40

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Timestamp-based validity check
It reduces inconsistency, but doesn't
eliminate it.
It is used for validity condition for cache
entries at the client:
(T - Tc < t) v (Tmclient = Tmserver)
t
freshnessguarantee
Tc timewhencacheentrywaslast
validated
Tm timewhenblockwaslast
updatedatserver
T currenttime

41

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

t is configurable (per file) but is typically set
to 3 seconds for files and 30 secs. for
directories.
it remains difficult to write distributed
applications that share files with NFS.

42

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Other NFS optimizations
Sun RPC runs over UDP by default (can use TCP
if required).
Uses UNIX BSD Fast File System with 8-kbyte
blocks.
reads() and writes() can be of any size
(negotiated between client and server).
The guaranteed freshness interval t is set
adaptively for individual files to reduce getattr()
calls needed to update Tm.
File attribute information (including Tm) is
piggybacked in replies to all file requests.
43

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

NFS performance
Early measurements (1987) established that:
Write() operations are responsible for only 5% of
server calls in typical UNIX environments.
hence write-through at server is acceptable.

Lookup() accounts for 50% of operations -due to

step-by-step pathname resolution necessitated by
the naming and mounting semantics.

More recent measurements (1993) show high

performance.
see www.spec.org for more recent measurements.

44

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

NFS summary
NFS is an excellent example of a simple,
robust, high-performance distributed
service.
Achievement of transparencies are other
goals of NFS:
Access transparency:
The API is the UNIX system call interface for
both local and remote files.

45

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Location transparency:
Naming of filesystems is controlled by client
mount operations, but transparency can be
ensured by an appropriate system configuration.

Mobility transparency:
Hardly achieved; relocation of files is not
possible, relocation of filesystems is possible,
but requires updates to client configurations.

Scalability transparency:
File systems (file groups) may be subdivided
and allocated to separate servers.
Ultimately, the performance limit is determined
by the load on the server holding the most
heavily-used filesystem (file group).
46

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Replication transparency:
Limited to read-only file systems; for writable
files, the SUN Network Information Service
(NIS) runs over NFS and is used to replicate
essential system files.

Hardware and software operating system

heterogeneity:
NFS has been implemented for almost every
known operating system and hardware platform
and is supported by a variety of filling systems.

Fault tolerance:
Limited but effective; service is suspended if a
server fails. Recovery from failures is aided by
the simple stateless design.
47

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Consistency:
It provides a close approximation to one-copy
semantics and meets the needs of the vast
majority of applications.
But the use of file sharing via NFS for
communication or close coordination between
processes on different computers cannot be
recommended.

Security:
Recent developments include the option to use
a secure RPC implementation for authentication
and the privacy and security of the data
transmitted with read and write operations.

48

DISTRIBUTED FILE SYSTEM

Case Study: Sun NFS

Efficiency:
NFS protocols can be implemented for use in
situations that generate very heavy loads.

49

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

Like NFS, AFS provides transparent

access to remote shared files for UNIX
programs running on workstations.
AFS is implemented as two software
components that exist at UNIX processes
called Vice and Venus.
(Figure 11)

50

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

Workstations

Servers

User Venus
program

Vice

UNIX kernel

UNIX kernel
User Venus
program
UNIX kernel

Network

Vice
Venus
User
program
UNIX kernel

UNIX kernel

Figure 11. Distribution of processes in the Andrew File System

51

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

The files available to user processes running on
workstations are either local or shared.
Local files are handled as normal UNIX files.
They are stored on the workstations disk and
are available only to local user processes.
Shared files are stored on servers, and copies of
them are cached on the local disks of
workstations.
The name space seen by user processes is
illustrated in Figure 12.
52

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

Local

Shared
/ (root)

tmp

bin

. . .

vmunix

cmu

bin

Symbolic
links

Figure 12. File name space seen by clients of AFS

53

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

The UNIX kernel in each workstation and

server is a modified version of BSD UNIX.
The modifications are designed to intercept
open, close and some other file system calls
when they refer to files in the shared name
space and pass them to the Venus process in
the client computer.
(Figure 13)

54

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

Workstation

User
program

Venus
UNIX file
system calls

Non-local file
operations

UNIX kernel
UNIX file system
Local
disk

Figure 13. System call interception in AFS

55

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

Figure 14 describes the actions taken by Vice,
Venus and the UNIX kernel when a user process
issues system calls.

56

Userprocess
open(FileName,
mode)

UNIXkernel

DISTRIBUTED
FILE
SYSTEM
Net
Vice

If FileNamereferstoa
fileinsharedfilespace,
Checklistoffilesin
passtherequestto
localcache.Ifnot
Venus.
presentorthereisno
valid callbackpromise
,
sendarequestforthe
filetotheViceserver
thatiscustodianofthe
volumecontainingthe
file.

Openthelocalfileand
returnthefile
descriptortothe
application.
read(FileDescriptor,
Buffer,length)

Venus

Placethecopyofthe
fileinthelocalfile
system,enteritslocal
nameinthelocalcache
listandreturnthelocal
nametoUNIX.

Transferacopyofthe
fileanda callback
promisetothe
workstation.Logthe
callbackpromise.

Performanormal
UNIXreadoperation
onthelocalcopy.

write(FileDescriptor, Performanormal
Buffer,length)
UNIXwriteoperation
onthelocalcopy.
close(FileDescriptor) Closethelocalcopy
andnotifyVenusthat
thefilehasbeenclosed. Ifthelocalcopyhas
beenchanged,senda
copytotheViceserver
thatisthecustodianof
thefile.

Replacethefile
contentsandsenda
callbacktoallother
clientsholding57
callback
promisesonthefile.

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

Figure 15 shows the RPC calls provided by AFS
servers for operations on files.

58

DISTRIBUTED FILE SYSTEM

Case Study: The Andrew File System (AFS)

Fetch(fid)>attr,data
Store(fid,attr,data)
Create()>fid
Remove(fid)
SetLock(fid,mode)

ReleaseLock(fid)
RemoveCallback(fid)
BreakCallback(fid)

Returnstheattributes(status)and,optionally,thecontentsoffile
identifiedbythefidandrecordsacallbackpromiseonit.
Updatestheattributesand(optionally)thecontentsofaspecified
file.
Createsanewfileandrecordsacallbackpromiseonit.
Deletesthespecifiedfile.
Setsalockonthespecifiedfileordirectory.Themodeofthe
lockmaybesharedorexclusive.Locksthatarenotremoved
expireafter30minutes.
Unlocksthespecifiedfileordirectory.
InformsserverthataVenusprocesshasflushedafilefromits
cache.
ThiscallismadebyaViceservertoaVenusprocess.Itcancels
thecallbackpromiseontherelevantfile.

Figure 15. The main components of the Vice service interface

59