CS 556: Distributed Systems

Fundamentals
Overview of Lecture
„ Fundamentals: terminology and
components of a reliable distributed
computing system
„ Communication technologies and their
properties
„ Basic communication services
„ Internet protocols
„ End-to-end argument
Some terminology
„ A program is the code you type in
„ A process is what you get when you run it
„ A message is used to communicate between
processes. Arbitrary size.
„ A packet is a fragment of a message that might
travel on the wire. Variable size but limited, usually
to 1400 bytes or less.
„ A protocol is an algorithm by which processes
cooperate to do something using message
exchanges.
More terminology
„ A network is the infrastructure that links the
computers, workstations, terminals, servers, etc.
„ It consists of routers
„ They are connected by communication links
„ A network application is one that fetches needed
data from servers over the network
„ A distributed system is a more complex application
designed to run on a network. Such a system has
multiple processes that cooperate to do something.
A network is like a “mostly
reliable” post office
Why isn’t it totally reliable?
„ Links can corrupt messages
„ Rare in the high quality ones on the Internet
“backbone”
„ More common with wireless connections, cable
modems, ADSL
„ Routers can get overloaded
„ When this happens they drop messages
„ As we’ll see, this is very common
„ But protocols that retransmit lost packets can
increase reliability
How do distributed systems differ from
network applications?
„ Distributed systems may have many components but
are often designed to mimic a single, non-distributed
process running at a single place.
„ “State” is spread around in a distributed system
„ Networked application is free-standing and centered around
the user or computer where it runs. (E.g. “web browser.)
„ Distributed system is spread out, decentralized. (E.g. “air
traffic control system”)
What about the Web?
„ Browser is independent: fetches data you request
when you ask for it.
„ Web servers don’t keep track of who is using them.
Each request is self-contained and treated
independently of all others.
„ Cookies don’t count: they sit on your machine
„ And the database of account info doesn’t count either… this
is “ancient” history, nothing recent
„ ... So the web has two network applications that talk
to each other
„ The browser on your machine
„ The web server it happens to connect with… which has a
database “behind” it
What about the Web?

Cookie identifies this

user, encodes past
preferences

HTTP request
Database
Web browser with
stashed cookies
 What about the Web?

Web servers immediately

forget the interaction

Reply updates cookie

What about the Web?

Web servers have no

memory of the interaction

Purchase is a “transaction”
on the database
What about the Web?
„ But… the data center that serves your
request may be a complex distributed system
„ Many servers and perhaps multiple physical sites
„ Opinions about which clients should talk to which
servers
„ Data replicated for load balancing and high
availability
„ Complex security and administration policies
„ So: we have a “networked application” talking
to a “distributed system”
Other examples of distributed
systems
„ Air traffic control system with workstations
for the controllers
„ Banking/brokerage trading system that
coordinates trading (risk management) at
multiple locations
„ Factory floor control system that monitors
devices and replans work as they go
on/offline
Is the Web “reliable”?
„ We want to build distributed systems that can be
relied upon to do the correct thing and to provide
services according to the user’s expectations
„ Not all systems need reliability
„ If a web site doesn’t respond, you just try again later
„ If you end up with two wheels of brie, well, throw a party!
„ Reliability is a growing requirement in “critical”
settings but these remain a small percentage of the
overall market for networked computers
„ And as we’ve mentioned, it entails satisfying multiple
properties…
Reliability is a broad term
„ Fault-Tolerance: remains correct despite failures
„ High or continuous availability: resumes service after failures,
doesn’t wait for repairs
„ Performance: provides desired responsiveness
„ Recoverability: can restart failed components
„ Consistency: coordinates actions by multiple components, so
they mimic a single one
„ Security: authenticates access to data, services
„ Privacy: protects identity, locations of users
“Failure” also has many
meanings
„ Halting failures: component simply stops
„ Fail-stop: halting failures with notifications
„ Omission failures: failure to send/recv. message
„ Network failures: network link breaks
„ Network partition: network fragments into two or
more disjoint subnetworks
„ Timing failures: action early/late; clock fails, etc.
„ Byzantine failures: arbitrary malicious behavior
Examples of failures
„ My PC suddenly freezes up while running a text
processing program. No damage is done. This is a
halting failure
„ A network file server tells its clients that it is about to
shut down, then goes offline. This is a failstop
failure. (The notification can be trusted)
„ An intruder hacks the network and replaces some
parts with fakes. This is a Byzantine failure.
More terminology
„ A real-world network is what we work on. It has
computers, links that can fail, and some problems
synchronizing time. But this is hard to model in a
formal way.
„ An asynchronous distributed system is a theoretical
model of a network with no notion of time
„ A synchronous distributed system, in contrast, has
perfect clocks and bounds all events, like message
passing.
Model we’ll use?
„ Our focus is on real-world networks, halting failures,
and extremely practical techniques
„ The closest model is the asynchronous one; we use it
to reason about protocols
„ Most often, employ asynchronous model to illustrate
techniques we can actually implement in real-world settings
„ And usually employ the synchronous model to obtain
impossibility results
„ Question: why not prove impossibility results in an
asynchronous model, or use the synchronous one to
illustrate techniques that we might really use?
 OSI protocol layers:
Oft-cited Standard
Application The program using a communication connection
Presentation Software to encode data into messages, and decode on reception
Session Logic associated with guaranteeing end-to-end reliability and
flow control, if desired
Transport Software for fragmenting big messages into small packets
Network Routing functionality, limited to small packets
Data-link The protocol that represents packets on the wire
Hardware Hardware for representing bits on the wire

• OSI is tied to a TCP-style of connection

• Match with modern protocols is poor
 Internet Layering Model
Internet Protocol Stack
Layer 5 Application
Layer 4 Transport
Layer 3 Network
Layer 2 Link
Layer 1 Physical

„ Each layer uses the function of the layer below

„ Each layer exports functionality to layer above
„ This layering of protocol behavior called a
“protocol stack”
„ Aka, the “TCP/IP stack”
Internet protocol suite
„ Can be understood in terms of OSI
„ Defines “addressing” standard, basic network
layer (IP packets, limited to 1400 bytes), and
session protocols (TCP, UDP, UDP-multicast)
„ For example, TCP is a “session” protocol
„ Includes standard “domain name service”
that maps host names to IP addresses
„ DNS itself is tree-structured and caches data
Major internet protocols
„ TCP, UDP, FTP, Telnet
„ Email: Simple Mail Transfer Protocol (SMTP)
„ News: Network News Transfer Protocol (NNTP)
„ DNS: Domain name service protocol
„ NIS: Network information service (a.k.a. “YP”)
„ LDAP: Protocol for talking to the management information
database (MIB) on a computer
„ NFS: Network file system protocol for UNIX
„ X11: X-server display protocol
„ Web: HyperText Transfer Protocol (HTTP), and SSL (one of the
widely used security protocols)
Typical hardware options
„ Ethernet: 10Mbit CSMA technology, limited to 1400
byte packets. Uses single coax cable.
„ FDDI: twisted pair, self-repairing if cable breaks
„ Bridged Ethernet: common in big LAN’s, ring with
multiple ethernet segments
„ Fast Ethernet: 100Mbit version of ethernet
„ ATM: switching technology for fiber optic paths. Can
run at 155Mbits/second or more. Very reliable, but
mostly used in telephone systems.
Implications for reliability?
„ Protocol designers have problems predicting
the properties of local-area networks
„ Latencies and throughput may vary widely
even in a single installation
„ Hardware properties differ widely; often,
must assume the least-common-denominator
„ Packet loss a minor problem in hardware
itself
 Technology trends
Did the sudden growth
in LAN speed give us the
CPU MIPS Web?
700
600
500 Memory MB
400
300 LAN Mbits
200
100 WAN Mbits
0
0

O/S
99

99

00

0
0
1

-2

-2

overhead
5-

0-

0
8

0
9

0
19

19

Source: Scientific American, Sept. 1995

 Typical latencies (milliseconds)
WAN, disk
Disk I/O latencies are
1000
fairly constant
100 Ethernet due to
10 RPC physical
1 ATM limitations
roundtrip
0.1
WAN
0.01 roundtrip
0

00

5
99

Note dramatic drop in

0
9

0
-2
-1

-1

-2
5
0

0
5

LAN latencies over ATM:

9
9

00
8

9
9
9
1

This is the hardware used

in telephone systems
O/S latency: the most expensive
overhead on LAN communication!

40
35
30
25
20 O/S
15 overhead as
10 percentage
5
0
1985- 1995-
1990 2000
Broad observations
„ A discontinuity is currently occurring in
communication speeds
„ Disks have “maxed out” and hence are looking slower
and slower
„ Memory of remote computers looks “closer and
closer”
„ O/S imposed communication latencies has risen in
relative terms over past decade!
Implications?
„ The revolution in WAN communication we are
now seeing is not surprising and will continue
„ Look for a shift from disk storage towards
more use of access to remote objects “over
the network”
„ O/S overhead is already by far the main
obstacle to low latency and this problem will
seem worse and worse unless O/S
communication architectures evolve in major
ways.
More Implications
„ Look for full motion video to the workstation by
around 2010 or 2015… today we already see this in
bits and pieces but not as a routine option
„ Low LAN latencies: an unexploited “niche”
„ One puzzle: what to do with extremely high data
throughput but relatively high WAN latencies
„ O/S architecture and whole concept of O/S must
change to better exploit the “pool of memory” of a
cluster of machines; otherwise, disk latencies will
loom higher and higher
Reliability and performance
„ Some think that more reliable means “slower”
„ Indeed, it usually costs time to overcome failure
„ For example, if a packet is lost probably need to resend it, and may
need to solicit the retransmission
„ But for many applications, performance is a big part of the
application itself: too slow means “not reliable” for these!
„ Reliable systems thus must look for highest possible
performance
„ ... but unlike unreliable systems, they can’t cut corners in ways
that make them flakey but faster
Moving up (the stack)
„ OSI hierarchy basically stops above the
session layer
„ In fact it assumes that applications know
about one-another and has a TCP model
„ Client looks up the server… connects…
sends a request. Response comes back
„ But how did the client know which
server it wanted?
Discovery
„ Consider the problem of discovering the
right server to connect with
„ Your computer needs current map data for
some place, perhaps an amusement park
„ Can think of it in terms of layers – the basic
park layout, overlaid with extra data from
various services, such as “length of the line for
the Cyclone Coaster” or “options for vegetarian
dining near here”
Why is discovery hard?
„ Client has opinions
„ You happen to like vegetarian food, but not spicy
food. So your search is partly controlled by client
goals
„ But a given service might have multiple servers
(e.g. Amazon might have data centers in Europe
and in the US…) and may want your request to go
to a particular one
„ Once we find the server name we need to map it
to an IP address
„ And the Internet itself has routing “opinions” too
So… four layers of discovery
„ Potentially, we might want to customize
each one of these layers to get a given
application functionality to work!
„ The OSI architecture didn’t include any
of these layers, so this is an example of
a situation where we need much more
than OSI!
Other things we might need
„ Standard ways to handle
„ Reliability, in all the senses we listed
„ Life cycle management
„ Automated startup of services, if someone asks
for one and it isn’t running; backup; etc…
„ Automated migration and load-balancing,
monitoring, parameter adaptation, self-
diagnosis and repair…
„ Tools for integrating legacy applications
with new, modern ones
Concept of a middleware
platform
„ These are big software systems that
automate many aspects of application
management and development
„ In this course we’ll discuss
„ CORBA – by now a stable and slightly
outmoded platform focused on “objects”
„ Web Services – the hot new “service
oriented architecture”
Layers: Modern perspective
End-user applications

Built over and with…

Middleware platform

Built over and with…

Internet and Web Standards (TCP, XML, etc)

For example
„ Imagine a banking system with many
programs, one at each branch
„ And suppose that only some can talk to
others due to firewalls and other
restrictions
„ E.g. A can talk to B and B can talk to C,
but A can’t talk to C
How to handle this?
„ In the distant past, people cooked up
all sorts of weird hacks
„ Today, a standard approach is to build
a routing layer
„ Inside the application, it would
automatically forward messages towards
their destinations
„ Thus A can talk to C (via B)
Once we have this…
„ Now we can split our brains, in a good way:
„ Above this routing layer, we write code as if
routing from anyone to anyone was automatic
„ Inside the routing layer, we implement this
functionality
„ Below the routing layer we just do point-to-point
messaging where the bank permits it and we
never end up trying to send messages over links
not available to us
This layering looks elegant!
„ It lets us focus attention on issues in
one place and simplifies code as a
result
„ Also helpful when debugging…

„ Platform architectures simply take the

same approach further
Using a platform
„ In this class many people will work with
„ Java/J2EE: An outgrowth from CORBA which is
closely integrated with developer tools and very
easy to use
„ Microsoft C# (or C++) on .NET in Visual Studio:
similar in concept but focused more on Web
Services
„ Often just using their editor and clicking
“build and run” is enough to use the service
framework!
„ But you inherit its power… and limits… and this
course is about learning them!
Can we evade limits?
„ Absolutely!
„ For example, the reliability model in Web
Services doesn’t automate data replication
„ We’ll learn how to implement replication
„ And we’ll also see that one can even use
these ideas in a Web Services setting!
„ … but it can be a pain