Distributed File Systems
CS-4513 Distributed Computing Systems
(Slides include materials from Operating System Concepts, 7th ed., by Silbershatz, Galvin, & Gagne, Modern Operating Systems, 2nd ed., by Tanenbaum, and Distributed Systems: Principles & Paradigms, 2nd ed. By Tanenbaum and Van Steen)
CS-4513, D-Term 2007
Distributed File Systems
Distributed Files Systems (DFS)
A special case of distributed system Allows multi-computer systems to share files
Even when no other IPC or RPC is needed
Sharing devices
Special case of sharing files
E.g.,
NFS (Suns Network File System) Windows NT, 2000, XP Andrew File System (AFS) & others
CS-4513, D-Term 2007 Distributed File Systems 2
Distributed File Systems (continued)
One of most common uses of distributed computing Goal: provide common view of centralized file system, but distributed implementation.
Ability to open & update any file on any machine on network All of synchronization issues and capabilities of shared local files
CS-4513, D-Term 2007 Distributed File Systems 3
Naming of Distributed Files
Naming mapping between logical and physical objects. A transparent DFS hides the location where in the network the file is stored. Location transparency file name does not reveal the files physical storage location.
File name denotes a specific, hidden, set of physical disk blocks. Convenient way to share data. Could expose correspondence between component units and machines.
Location independence file name does not need to be changed when the files physical storage location changes.
Better file abstraction. Promotes sharing the storage space itself. Separates the naming hierarchy from the storage-devices hierarchy.
CS-4513, D-Term 2007 Distributed File Systems 4
DFS Three Naming Schemes
1. Mount remote directories to local directories, giving the appearance of a coherent local directory tree
Mounted remote directories can be accessed transparently. Unix/Linux with NFS; Windows with mapped drives
2. Files named by combination of host name and local name;
Guarantees a unique system wide name Windows Network Places, Apollo Domain A single global name structure spans all the files in the system. If a server is unavailable, some arbitrary set of directories on different machines also becomes unavailable.
Distributed File Systems 5
3. Total integration of component file systems.
CS-4513, D-Term 2007
Mounting Remote Directories (NFS)
CS-4513, D-Term 2007
Distributed File Systems
Mounting Remote Directories (continued)
Note: names of files are not unique
As represented by path names
E.g.,
Server A sees : /users/steen/mbox Client A sees: /remote/vu/mbox Client B sees: /work/me/mbox
Consequence: Cannot pass file names around haphazardly
CS-4513, D-Term 2007 Distributed File Systems 7
Mounting Remote Directories in NFS
More later
CS-4513, D-Term 2007
Distributed File Systems
DFS File Access Performance
Reduce network traffic by retaining recently accessed disk blocks in local cache Repeated accesses to the same information can be handled locally.
All accesses are performed on the cached copy.
If needed data not already cached, copy of data brought from the server to the local cache.
Copies of parts of file may be scattered in different caches.
Cache-consistency problem keeping the cached copies consistent with the master file.
Especially on write operations
CS-4513, D-Term 2007 Distributed File Systems 9
DFS File Caches
In client memory
Performance speed up; faster access Good when local usage is transient Enables diskless workstations
On client disk
Good when local usage dominates (e.g., AFS) Caches larger files Helps protect clients from server crashes
CS-4513, D-Term 2007 Distributed File Systems 10
DFS Cache Update Policies
When does the client update the master file?
I.e. when is cached data written from the cache to the file?
Write-through write data through to disk ASAP
I.e., following write() or put(), same as on local disks. Reliable, but poor performance.
Delayed-write cache and then written to the server later.
Write operations complete quickly; some data may be overwritten in cache, saving needless network I/O. Poor reliability
unwritten data may be lost when client machine crashes Inconsistent data
Variation scan cache at regular intervals and flush dirty blocks.
CS-4513, D-Term 2007
Distributed File Systems
11
DFS File Consistency
Is locally cached copy of the data consistent with the master copy? Client-initiated approach
Client initiates a validity check with server. Server verifies local data with the master copy
E.g., time stamps, etc.
Server-initiated approach
Server records (parts of) files cached in each client. When server detects a potential inconsistency, it reacts
CS-4513, D-Term 2007 Distributed File Systems 12
DFS Remote Service vs. Caching
Remote Service all file actions implemented by server.
RPC functions Use for small memory diskless machines Particularly applicable if large amount of write activity
Cached System
Many remote accesses handled efficiently by the local cache
Most served as fast as local ones.
Servers contacted only occasionally
Reduces server load and network traffic. Enhances potential for scalability.
Reduces total network overhead
CS-4513, D-Term 2007 Distributed File Systems 13
DFS File Server Semantics
Stateless Service
Avoids state information in server by making each request self-contained. Each request identifies the file and position in the file. No need to establish and terminate a connection by open and close operations. Poor support for locking or synchronization among concurrent accesses
CS-4513, D-Term 2007 Distributed File Systems 14
DFS File Server Semantics (continued)
Stateful Service
Client opens a file (as in Unix & Windows). Server fetches information about file from disk, stores in server memory,
Returns to client a connection identifier unique to client and open file. Identifier used for subsequent accesses until session ends.
Server must reclaim space used by no longer active clients. Increased performance; fewer disk accesses. Server retains knowledge about file
E.g., read ahead next blocks for sequential access E.g., file locking for managing writes
Windows
CS-4513, D-Term 2007 Distributed File Systems 15
DFS Server Semantics Comparison
Failure Recovery: Stateful server loses all volatile state in a crash.
Restore state by recovery protocol based on a dialog with clients. Server needs to be aware of crashed client processes
orphan detection and elimination.
Failure Recovery: Stateless server failure and recovery are almost unnoticeable.
Newly restarted server responds to self-contained requests without difficulty.
CS-4513, D-Term 2007 Distributed File Systems 16
DFS Server Semantics Comparison
(continued)
Penalties for using the robust stateless service:
longer request messages slower request processing
Some environments require stateful service.
Server-initiated cache validation cannot provide stateless service. File locking (one writer, many readers).
CS-4513, D-Term 2007 Distributed File Systems 17
DFS Replication
Replicas of the same file reside on failure-independent machines. Improves availability and can shorten service time. Naming scheme maps a replicated file name to a particular replica.
Existence of replicas should be invisible to higher levels. Replicas must be distinguished from one another by different lower-level names.
Updates
Replicas of a file denote the same logical entity Update to any replica must be reflected on all other replicas.
CS-4513, D-Term 2007
Distributed File Systems
18
Example Distributed File Systems
NFS Suns Network File System (ver. 3)
Tanenbaum & van Steen, Chapter 11
NFS Suns Network File System (ver. 4)
Tanenbaum & van Steen, Chapter 11
AFS the Andrew File System
See Silbershatz 17.6
CS-4513, D-Term 2007 Distributed File Systems 19
NFS
Sun Network File System (NFS) has become de facto standard for distributed UNIX file access. NFS runs over LAN
even WAN (slowly)
Any system may be both a client and server
Basic idea:
Remote directory is mounted onto local directory Remote directory may contain mounted directories within
CS-4513, D-Term 2007 Distributed File Systems 20
Mounting Remote Directories (NFS)
CS-4513, D-Term 2007
Distributed File Systems
21
Nested Mounting (NFS)
CS-4513, D-Term 2007
Distributed File Systems
22
NFS Implementation
NFS
CS-4513, D-Term 2007
Distributed File Systems
23
NFS Operations
Lookup
Fundamental NFS operation Takes pathname, returns file handle
File Handle
Unique identifier of file within server Persistent; never reused Storable, but opaque to client
64 bytes in NFS v3; 128 bytes in NFS v4
Most other operations take file handle as argument
CS-4513, D-Term 2007 Distributed File Systems 24
Other NFS Operations (version 3)
read, write link, symlink mknod, mkdir rename, rmdir readdir, readlink getattr, setattr create, remove
Conspicuously absent
open, close
CS-4513, D-Term 2007
Distributed File Systems
25
NFS v3 A Stateless Service
Server retains no knowledge of client
Server crashes invisible to client
All hard work done on client side Every operation provides file handle Server caching
Performance only Based on recent usage
Client caching
Client checks validity of caches files Client responsible for writing out caches
CS-4513, D-Term 2007 Distributed File Systems 26
NFS v3 A Stateless Service (continued)
No locking! No synchronization! Unix file semantics not guaranteed
E.g., read after write
Session semantics not even guaranteed
E.g., open after close
CS-4513, D-Term 2007
Distributed File Systems
27
NFS v3 A Stateless Service (continued)
Solution: global lock manager
Separate from NFS
Typical locking operations
Lock acquire lock (non-blocking) Lockt test a lock Locku unlock a lock Renew renew lease on a lock
CS-4513, D-Term 2007
Distributed File Systems
28
NFS Implementation
Remote procedure calls for all operations
Implemented in Sun ONC XDR is interface definition language
Network communication is client-initiated
RPC based on UDP (non-reliable protocol) Response to remote procedure call is de facto acknowledgement
Lost requests are simply re-transmitted
As many times as necessary to get a response!
CS-4513, D-Term 2007 Distributed File Systems 29
NFS Caching
On client open(), client asks server if its cached attribute blocks are up to date. Once file is open, different client processes can write it and get inconsistent data.
Modified data is flushed back to the server every 30 seconds.
CS-4513, D-Term 2007 Distributed File Systems 30
NFS Failure Recovery
Server crashes are transparent to client
Each client request contains all information Server can re-fetch from disk if not in its caches Client retransmits request if interrupted by crash
(i.e., no response)
Client crashes are transparent to server
Server maintains no record of which client(s) have cached files.
CS-4513, D-Term 2007 Distributed File Systems 31
Summary NFS
That was version 3 of NFS
Stateless file system High performance, simple protocol Based on UDP
Everything has changed in NFS version 4
First published in 2000 Clarifications published in 2003 Almost complete rewrite of NFS
CS-4513, D-Term 2007 Distributed File Systems 32
NFS Version 4
Stateful file service Based on TCP reliable transport protocol More ways to access server Compound requests
I.e., multiple RPC calls in same packet
More emphasis on security Mount protocol integrated with rest of NFS protocol
CS-4513, D-Term 2007 Distributed File Systems 33
NFS Version 4
CS-4513, D-Term 2007
Distributed File Systems
34
NFS Version 4 (continued)
Additional RPC operations
Long list for managing files, caches, validating versions, etc. Also security, permissions, etc.
Also
Open() and close(). With a server crash, some information may have to be recovered
See
Silbershatz, p. 653
http://www.tcpipguide.com/free/t_TCPIPNetworkFileSystemNFS. htm
CS-4513, D-Term 2007 Distributed File Systems 35
Questions?
CS-4513, D-Term 2007
Distributed File Systems
36
Andrew File System (AFS)
Completely different kind of file system Developed at CMU to support all student computing. Consists of workstation clients and dedicated file server machines.
CS-4513, D-Term 2007
Distributed File Systems
37
Andrew File System (AFS)
Stateful Single name space
File has the same names everywhere in the world.
Lots of local file caching
On workstation disks For long periods of time Originally whole files, now 64K file chunks.
Good for distant operation because of local disk caching
CS-4513, D-Term 2007 Distributed File Systems 38
AFS
Need for scaling led to reduction of client-server message traffic.
Once a file is cached, all operations are performed locally. On close, if the file is modified, it is replaced on the server.
The client assumes that its cache is up to date! Server knows about all cached copies of file
Callback messages from the server saying otherwise.
CS-4513, D-Term 2007 Distributed File Systems 39
AFS
On file open()
If client has received a callback for file, it must fetch new copy Otherwise it uses its locally-cached copy.
Server crashes
Transparent to client if file is locally cached Server must contact clients to find state of files
See Silbershatz 17.6
CS-4513, D-Term 2007 Distributed File Systems 40
Distributed File Systems Summary
Performance is always an issue
Tradeoff between performance and the semantics of file operations (especially for shared files).
Caching of file blocks is crucial in any file system, distributed or otherwise.
As memories get larger, most read requests can be serviced out of file buffer cache (local memory). Maintaining coherency of those caches is a crucial design issue.
Current research addressing disconnected file operation for mobile computers.
CS-4513, D-Term 2007 Distributed File Systems 41
Reading Assignment
Silbershatz, Chapter 17 or Tanenbaum, Modern Operating Systems
8.3 and 10.6.4
or Tanenbaum & van Steen, Chapter 11
CS-4513, D-Term 2007
Distributed File Systems
42
Questions?
CS-4513, D-Term 2007
Distributed File Systems
43
