Communication: Foundations Layered Protocols

Basic networking model

Distributed Systems Application protocol

Application 7
(3rd Edition) Presentation
Presentation protocol
6
Session protocol
Session 5
Transport protocol
Transport 4
Maarten van Steen Andrew S. Tanenbaum Network
Network protocol
3
Data link protocol
Data link 2
Physical protocol
Physical 1

Chapter 04: Communication Network

Edited by: Hicham G. Elmongui Drawbacks

Focus on message-passing only
Often unneeded or unwanted functionality
Violates access transparency

The OSI reference model 2 / 26

Communication: Foundations Layered Protocols Communication: Foundations Layered Protocols

Middleware layer An adapted layering scheme

Observation
Middleware is invented to provide common services and protocols that can be Application protocol
used by many different applications Application

A rich set of communication protocols Middleware

Middleware protocol
(Un)marshaling of data, necessary for integrated systems
Naming protocols, to allow easy sharing of resources Host-to-host protocol
Operating
Security protocols for secure communication system
Scaling mechanisms, such as for replication and caching Physical/Link-level protocol
Hardware

Note
Network
What remains are truly application-specific protocols... such as?

Middleware protocols 3 / 26 Middleware protocols 4 / 26

Communication: Foundations Types of Communication Communication: Foundations Types of Communication

Types of communication Types of communication

Transient versus persistent
Distinguish...
Synchronize at Synchronize at Synchronize after
Synchronize at Synchronize at Synchronize after request submission request delivery processing by server
request submission request delivery processing by server Client
Client
Request
Request
Transmission
Transmission interrupt
interrupt
Storage
Storage facility
facility
Reply
Reply
Server Time
Server Time

Transient communication: Comm. server discards message when it

Transient versus persistent communication cannot be delivered at the next server, or at the receiver.
Asynchronous versus synchronous communication Persistent communication: A message is stored at a communication
server as long as it takes to deliver it.
5 / 26 6 / 26
Communication: Foundations Types of Communication Communication: Foundations Types of Communication

Types of communication Client/Server

Places for synchronization

Synchronize at Synchronize at Synchronize after Some observations
request submission request delivery processing by server
Client Client/Server computing is generally based on a model of transient
synchronous communication:
Request

Transmission Client and server have to be active at time of communication

interrupt Client issues request and blocks until it receives reply
Storage Server essentially waits only for incoming requests, and subsequently
facility
processes them
Reply

Server Time
Drawbacks synchronous communication
Client cannot do any other work while waiting for reply
Failures have to be handled immediately: the client is waiting
At request submission The model may simply not be appropriate (mail, news)
At request delivery
After request processing

7 / 26 8 / 26

Communication: Foundations Types of Communication Communication: Remote procedure call Basic RPC operation

Messaging Basic RPC operation

Observations
Application developers are familiar with simple procedure model
Message-oriented middleware Well-engineered procedures operate in isolation (black box)
There is no fundamental reason not to execute procedures on separate
Aims at high-level persistent asynchronous communication: machine
Processes send each other messages, which are queued Wait for result
Sender need not wait for immediate reply, but can do other things Client

Middleware often ensures fault tolerance Conclusion Call remote Return

procedure from call
Communication between caller & callee
can be hidden by using procedure-call Request Reply
mechanism. Server
Call local procedure Time
and return results

9 / 26 10 / 26

Communication: Remote procedure call Basic RPC operation Communication: Remote procedure call Parameter passing

Basic RPC operation RPC: Parameter passing

Client machine Server machine
There’s more than just wrapping parameters into a message
Server process
Client process
1. Client call to Client and server machines may have different data representations (think
Implementation 6. Stub makes
procedure
of doit local call to “doit” of byte ordering)
r = doit(a,b)
Server stub
r = doit(a,b)
Wrapping a parameter means transforming a value into a sequence of
proc: “doit”
Client stub
proc: “doit”
bytes
type1: val(a) 2. Stub builds type1: val(a) 5. Stub unpacks
message
Client and server have to agree on the same encoding:
type2: val(b) message type2: val(b)

proc: “doit” 4. Server OS

Client OS type1: val(a)
type2: val(b)
Server OS hands message How are basic data values represented (integers, floats, characters)
to server stub
How are complex data values represented (arrays, unions)
3. Message is sent
across the network
Conclusion
1 Client procedure calls client stub. 6 Server does local call; returns result to stub.
2 Stub builds message; calls local OS. 7 Stub builds message; calls OS.
Client and server need to properly interpret messages, transforming them into
3 OS sends message to remote OS. 8 OS sends message to client’s OS. machine-dependent representations.
4 Remote OS gives message to stub. 9 Client’s OS gives message to stub.
5 Stub unpacks parameters; calls 10 Client stub unpacks result; returns to client.
server.
11 / 26 12 / 26
Communication: Remote procedure call Parameter passing Communication: Remote procedure call Variations on RPC

RPC: Parameter passing Asynchronous RPCs

Some assumptions
Copy in/copy out semantics: while procedure is executed, nothing can be Essence
assumed about parameter values. Try to get rid of the strict request-reply behavior, but let the client continue
All data that is to be operated on is passed by parameters. Excludes without waiting for an answer from the server.
passing references to (global) data.
Wait for Callback to client
Client
Conclusion acceptance

Full access transparency cannot be realized. Call remote Return

procedure from call Return
results
Accept
A remote reference mechanism enhances access transparency Request request

Remote reference offers unified access to remote data Server Call local procedure Time
Remote references can be passed as parameter in RPCs
Note: stubs can sometimes be used as such references

13 / 26 Asynchronous RPC 14 / 26

Communication: Remote procedure call Variations on RPC Communication: Remote procedure call Example: DCE RPC

Sending out multiple RPCs RPC in practice

Uuidgen

Essence Interface
definition file
Sending an RPC request to a group of servers.
IDL compiler

Server Call local procedure

Client code Client stub Header Server stub Server code

#include #include
Callbacks to client
C compiler C compiler C compiler C compiler
Client

Client Client stub Server stub Server

Call remote object file object file object file object file
procedures

Runtime Runtime
Linker Linker
library library
Server Call local procedure Time

Client Server
binary binary

Multicast RPC 15 / 26 Writing a Client and a Server 16 / 26

Communication: Remote procedure call Example: DCE RPC Communication: Message-oriented communication Simple transient messaging with sockets

Client-to-server binding (DCE) Transient messaging: sockets

Berkeley socket interface
Issues Operation Description
socket Create a new communication end point
(1) Client must locate server machine, and (2) locate the server. bind Attach a local address to a socket
listen Tell operating system what the maximum number of pending
connection requests should be
Directory machine
accept Block caller until a connection request arrives
connect Actively attempt to establish a connection
Directory
server send Send some data over the connection
3. Look up server
2. Register service receive Receive some data over the connection
Client machine
Server machine close Release the connection

5. Do RPC 1. Register port

Server
Client Server
socket bind listen accept receive send close
4. Ask for port DCE
daemon Port Synchronization point Communication
table
socket connect send receive close
Client

Binding a client to a server 17 / 26 18 / 26

Communication: Message-oriented communication Advanced transient messaging Communication: Message-oriented communication Advanced transient messaging

Making sockets easier to work with MPI: When lots of flexibility is needed
Sockets deemed insufficient
Observation
They were at wrong level of abstraction by supporting only simple send
Sockets are rather low level and programming mistakes are easily made.
and receive operations
However, the way that they are used is often the same (such as in a
client-server setting). They were not considered suitable for the proprietary protocols developed
for high-speed interconnection networks
Alternative: ZeroMQ
Message-Passing Interface (MPI)
Provides a higher level of expression by pairing sockets: one for sending
messages at process P and a corresponding one at process Q for receiving a standard for message passing that is hardware and platform
messages. All communication is asynchronous. independent
is designed for parallel applications and as such is tailored to transient
Three patterns communication
Request-reply
makes direct use of the underlying network
Publish-subscribe
Pipeline assumes that serious failures such as process crashes or network
partitions are fatal and do not require automatic recovery
Using messaging patterns: ZeroMQ 19 / 26 The Message-Passing Interface (MPI) 20 / 26

Communication: Message-oriented communication Message-oriented persistent communication Communication: Message-oriented communication Message-oriented persistent communication

Message-oriented middleware General model

Essence Queue managers

Asynchronous persistent communication through support of middleware-level Queues are managed by queue managers. An application can put messages
queues. Queues correspond to buffers at communication servers. only into a local queue. Getting a message is possible by extracting it from a
local queue only ⇒ queue managers need to route messages.
Operations
Routing
Operation Description
Look up
put Append a message to a specified queue Source queue contact address
of destination
Destination queue
manager manager
get Block until the specified queue is nonempty, and queue manager

remove the first message Logical

queue-level
poll Check a specified queue for messages, and remove address (name)

the first. Never block Address lookup

Local OS Local OS
notify Install a handler to be called when a message is put database

into the specified queue Contact

address
Network

Message-queuing model 21 / 26 General architecture of a message-queuing system 22 / 26

Communication: Message-oriented communication Message-oriented persistent communication Communication: Message-oriented communication Message-oriented persistent communication

Message broker Message broker: general architecture

Observation
Source Message broker Destination
Message queuing systems assume a common messaging protocol: all
applications agree on message format (i.e., structure and data representation) Application Application

Broker plugins Rules

Broker handles application heterogeneity in an MQ system Interface Interface

Queuing
Transforms incoming messages to target format layer

Very often acts as an application gateway Local OS Local OS Local OS

May provide subject-based routing capabilities (i.e., publish-subscribe
capabilities)

Message brokers 23 / 26 Message brokers 24 / 26

Communication: Multicast communication Application-level tree-based multicasting Communication: Multicast communication Flooding-based multicasting

Application-level multicasting Flooding

Multicasting as Broadcasting
Construct an overlay network per multicast group.

Essence Performance
Organize nodes of a distributed system into an overlay network and use that A node belonging to several groups, will, in principle, need to maintain a
network to disseminate data: separate list of its neighbors for each group of which it is a member.
Oftentimes a tree, leading to unique paths
Alternatively, also mesh networks, requiring a form of routing Essence
P simply sends a message m to each of its neighbors. Each neighbor will
forward that message, except to P, and only if it had not seen m before.

Performance
The more edges, the more expensive!

25 / 26 26 / 26