COMMUNICATION

Communication between two computers on a distributed system refers to communication between

processes running on these computers. E.g. Resource manager processes on individual computers.
A distributed operating system will therefore need to provide for Inter-process Communication (IPC)
mechanisms to facilitate such communication activities.
Purpose for IPC
1. Data Transfer
2. Sharing Data
3. Event notification
4. Resource Sharing and Synchronization
5. Process Control
For IPC to take place, information sharing between the two processes is required and takes two forms:
1. Original Sharing shared data approach
Computers in a distributed system do not share memory and therefore message passing is the basic IPC
mechanism. The communicating processes communicate directly with each other.
2. A message-passing system is a subsystem of a distributed operating system that provides a set of
message-based IPC protocols and does so by shielding the details of complex network protocols and
multiple heterogeneous platforms from programmers.
All programmers therefore need to write programs that use simple communication primitives such send,
receive etc.
Desired Features of A Good Message-Passing System
a) Simplicity simple and easy to use
b) Uniform Semantics for both local and remote communication
c) Efficiency by reducing the number of message exchanges during the communication process
d) Reliability should cope with failure problems and guarantee the delivery of a message.
e) Correctness occurs when there is need for a sender to send to several receivers and a receiver
to receive from several senders. Issues such as atomicity, ordered delivery and survivability are
addressed.
f) Flexibility to suite (support) different applications with different requirements
g) Security should provide secure end-to-end communication
h) Portability both the message passing system and the applications written by using the
primitives of the IPC protocols of the message passing.
Issues in IPC Message Passing
A message is a block of information formatted by a sending process in manner that it is meaningful to
the receiving process.
It is made up of:
i. A fixed length header with 3 parts:
Address of the sending and receiving processes
Sequence number message ID (useful incases of losses or duplications)
Structural information the type (data or pointer to data included) and the length
of the message
ii. A variable size collection of typed data objects
Issues to be addressed
i. Who is the sender?
ii. Who is the receiver?
iii. Is there one receiver or many receivers?
iv. Is the message guaranteed to have been accepted by the receiver(s)?
v. Does the sender need to wait for reply?
vi. What should be done if a catastrophic event such as a node crash or a communication link
failure occurs during the course of communication?
vii. What should be done if the receiver is not ready to accept the message: will the message be
discarded or stored in a buffer? In the case of buffering, what should be done if the buffer is
full?
viii. If there are several outstanding messages for a receiver, can it choose the order which to service
the outstanding messages?
Synchronization:
There are 2 Semantics:
(I)Blocking Semantics - their invocation blocks the execution of their invoker.
On send, the sending process is blocked until the receiver acknowledges and on receive, the receiving
process is blocked until the message is received. (Called Synchronous communication)
(ii)Non-Blocking Semantics their invocation does not block the execution of their invoker
On send, the sending process is allowed to proceed and on receive, the receiving process proceeds and
returns control almost immediately. (Called Asynchronous communication)

Communication: Non-Blocking Semantics
How does the receiving process know that the message has arrived in the buffer?Two ways are
i. Polling using a test primitive to test the buffer status
ii. Interrupt a software interrupt to the receiving process is issued once the message has filled
the buffer and it is ready for use by the process
Communication: Buffering
1. Null Buffer (No Buffering) -No place to temporary store the message. The message remains in
the senders address space until the sender is ready or the message is simple discarded
2. Single Message Buffer-A buffer with capacity to store a single message
3. Unbounded-Capacity Buffer-To receive all the un-received messages, (Practically impossible).
4. Finite-Bound (or Multiple-Message Buffer)-Finite length of n messages. Sender must wait if
buffer is full. Need to address buffer overflow.
Communication: Multidatagram Messages
Remember Maximum-Transfer-Unit (MTU) of network? The maximum data size that can be transmitted.
Each fragment of a message is put into a packet called datagram.
Single-datagram messages vs. multidatagram messages
Disassembling and reassembling of multidatagram messages is usually handled by the message-passing
system. Encoding (into a stream form) and decoding of the message is carried out for easy transmission
Communication : Encoding and decoding of message data
A message data should be meaningful to the receiving process. I.e. the structure of the program objects
should be preserved while they are being transmitted from the address space of the sending process to
the address space of the receiving process.
Therefore there is need to convert data at both ends hence encoding-decoding requirements in
message passing IPC.
PROCESS ADDRESSING
Addressing (or naming) of the parties (sender and receiver)
a) Explicit Addressing the process is explicitly named as a parameter
b) Implicit Addressing the process name is not supplied
c) Functional addressing The primitive used identifies a service (c below) rather than a
process
Examples:
a) Send(process_id, message) send a message to the process identified by process_id
b) Receive(process_id, message) receive a message from the process identified by process_id
c) Send_any(service_id, message) send a message to any process that provides the service of
type service_id
d) Receive(process_id, message) receive a message from any process and return the process
identifier of the process from which the message was received.
(a)A simple way to identify a process is by use of the machine_id and the local_id (process identifier or
port identifier). This does not require global coordination (an advantage) but does not allow processes
to migrate to other machines (a drawback)
(b)Link-based process address with 3 fields:
Machine_id where the process is created
Local_id local id of the process generated at the machine where it is created
Machine_id last known location of the process
FAILURE HANDLING
Failures during IPC may lead to:
a) Loss of request message the communication link has failed or the receiver is down
b) Loss of response message the communication link has failed or the sender is down
c) Unsuccessful execution of the request receiver crashes while the request is being processed
These are addressed by designing a reliable IPC protocol based on the idea of internal retransmissions of
messages after timeouts and the return acknowledgement message to the sender by the receiver.
( Class discussion How do we handle each case)

SEMANTICS FOR RELIABLE IPC PROTOCOL
1.Four-Message Reliable IPC protocol
The client sends a message to the server
The kernel of the server returns an acknowledgement message.
The server returns reply message
The client sends an acknowledgement message
2.Three-Message Reliable IPC protocol
The client sends a message to the server
The server returns reply message
The client sends an acknowledgement message
A problem if a request processing takes a long time and this may be interpreted as lost message
leading to unnecessary re-transmissions. Use of a timer at the receivers can be used to address this
problem
3. Two-Message Reliable IPC protocol
The client sends a message to the server
The server returns reply message
It obeys at-least-once semantics at least one execution.

Idempotency in Handling of Duplicate Request Messages
Idempotency repeatability.
Idempotent operations produces same results without any side effects if repeated, while
nonidempotent operations do not necessarily produce the same results.
Exactly-once semantics is needed for nonidempotent operations. This can be done by use of a unique
id for every request that the client makes and set up a reply cache in the kernels address space on the
server machine.

Keeping Track of Lost and Out-of-Sequence Packets
Stop-and-wait protocol acknowledge packet separately. This leads to communication
overhead
Blast-protocol a single acknowledgement for all packets but may fail if a node crashes.
Use of a bitmap include the total number of packets and the position of each packet in the
headers of all packets.

MESSAGE PASSING APPLICATION
Sockets provide the communications channel
Typically viewed as a command + arguments
Sent via sockets with header/length/body/trailer
Note that sockets are buffered (with fixed size)
The socket example takes a client server architecture
Communication:
Message Passing Application (Server)
import java.lang.*;
import java.io.*;
import java.net.*;
class Server {
public static void main(String args[]) {
String data = "Toobie ornaught toobie";
try {
ServerSocket srvr = new ServerSocket(1234);
Socket skt = srvr.accept();
System.out.print("Server has connected!\n");
PrintWriter out = new
Communication:
Message Passing Application (Client)
import java.lang.*;
import java.io.*;
import java.net.*;

class Client {
public static void main(String args[]) {
try {
Socket skt = new Socket("localhost", 1234);
BufferedReader in = new BufferedReader(new
InputStreamReader(skt.getInputStream()));
System.out.print("Received string: '");


Message Passing Application: Assignment
You are required to write a client server application that uses the concept of sockets. The
application should be able to:
From the client side, a request for the services offered by the server is made.
From the list of services given, the client chooses the service that it needs then request
for that service from the server.
The server replies back to the client with a solution
At least 3 services must be implemented at the server side
GROUP COMMUNICATION
The most common and elementary message-based interaction is one-to-one communication (point to
point or unicast communication).For performance and ease of programming there is need for group
communication
Types of group communication:
a) One-to-many(single sender & multiple receivers): broadcast
b) Many-to-one(multiple senders & single receiver): synchronization, collective
communication
c) Many-to-many(multiple senders & multiple receiver): gather and scatter
For many applications, the recipient of a message is a group of related process.
Many services in DS are implemented by a group of processes, possibly distributed in multiple
machines.
Examples of use:
(a) Fault tolerance based on replicated servers: The service is implemented by a group of (replicated)
server processes. A client process multicast(Send-to-group) request message to the server group. Each
group member performs identical operation on a request.
(b) Locating objects in distributed service: For example, a distributed file service is implemented by a
group of servers. The files are not replicated. A client looking for a particular file multicast the request
message to the server group. The member holding the wanted file responds.
(c) Updating replicated data: Data may be replicated to improve performance (response time) and
reliability. The data is replicated on multiple servers which forms a server group. Update requests are
multicasts to the group.
(d) Multiple notification: For example, a group of processes need be notified of certain event. When the
event occurs, a message can be multicasted to the group. A specific example is in the multicasting of
flight information to a group of display processes in an airport terminal system.
Group Management
A Group is a collection of processes: G = {P1, P2, , Pn}
Group management functions:
Forming a group: Create-group (out gid: group-id) - returns a group identifier
Joining a Group: Join-group (in gid: group-id) - makes the caller process a member of the group
gid.
Leaving a group: Leave-group (in gid: group-id)- remove the caller process from the group.
Delete a group: Delete-group (in gid: group-id) - Only authorized process (such as the creator)
can delete a group
Group Addressing
A two-level naming scheme is normally used for group addressing.
The high-level group name (ASCII)- is independent of the location info of the processes
in the group.
The low-level number depends on the underlying h/w
Dynamic group vs static group
Closed Group: Processes outside the group cannot multicast to the group, although an outsider process
can send to individual member of the group.
Group communication: Implementation
Centralised: Use a group management server.
All group membership function requests are handled by the server.
Server keeps database of all active groups (membership) in the system.
Pro: implementation ease.
Con: server may be a reliability and performance bottleneck. Also a single point of
failure
Distributed: Each group member keeps track of membership. membership function requests are
multicast to all members.
Pro: reliable
Cons: complexity and overheads
Group communication : Functions (Primitives)
Send to group: Send_Group (in gid: group-id, msg: message)
Receive from group: Receive_Group (in gid; group-id, out msg: message) Receive any message
sent to group gid.
Group Send and Receive have similar issues as IPC with respect to:
a)Blocking and non-blocking:
Unicast: A message is sent to each of the members of the group.
Number of message = group size
Inefficient
If underlying communication system supports broadcast to machines (e.g. Ethernet). A
broadcast message is sent to all machines.
The group communication facility within the kernel on respective machine takes in the
message if there are members (of the group) on that machine.
If underlying communication system supports multicast to machines. Then the message
can be multicasted to only those machines on which there are members.
b)Group Atomocity:
- Group atomicity is concerned with whether the group is always a whole or can be partitioned
(with respect to group operations).
Atomic multicast: When a message is sent to the group, all members of the group receive the message.
Otherwise, no member receives the message. i.e., there is never the case that some members receive
the message while others do not. It is all (member) or none.
Group Atomicity-Example in which atomicity is needed: Replicated data update. If not, the replicated
data may get out of steps with each other.
Example in which atomicity is not needed: Locating objects in distributed service. It is sufficient that the
server holding the object receives the message. If the message to this server is lost, the client can try
again (say on time-out). Multiple notification in a flight information display system.

Message ordering
The Order of message received by a group member can be important
Some message ordering required
Absolute ordering
Consistent/total ordering
Causal ordering
FIFO ordering
Message ordering:
1.FIFO order
Rule: Messages received in the same order as they were sent.
Implementation: Messages assigned a sequence number
Example: TCP-This is the weakest ordering.

2. Absolute order
Rule: Mi must be delivered before mj if Ti < Tj
Implementation:A clock synchronized among machines
A sliding time window used to commit message delivery whose timestamp is in this window.
Drawback: Too strict constraint, No absolute synchronized clock, No guarantee to catch all tardy/slow
messages
3. Consistent/Total ordering
Rule: Messages received in the same order (regardless of their timestamp).
Implementation: A message sent to a sequencer, assigned a sequence number, and finally multicast to
receivers. A message retrieved in incremental order at a receiver
Example: Replicated database updates
Drawback: A centralized algorithm.
4. Causal ordering
Rule: Happened-before relation
If e
k
i, e
l
i h and k < l, then e
k
i e
l
i,
If ei = send(m) and ej = receive(m), then ei ej,
If e e and e e, then e e

REMOTE PROCEDURE CALL
RPC is a model that specifies how cooperating processes on different nodes in a heterogeneous
computing environment can communicate and coordinate activities.
It is an approach to providing distributed computing services in a heterogeneous computing
environment.
The semantics of RPC are almost identical to the semantics of the traditional procedure call.
The major difference:
a normal procedure call takes place between procedures of a single process in the same
memory space on a single system,
RPC takes place between processes on clients and servers in a heterogeneous
computing environment.
RPC concept :: to make a remote procedure call appear like a local procedure call.
The goal is to hide the details of the network communication (namely, the sending and receiving of
messages). The calling procedure should not be aware that the called procedure is executing on a
different machine.

Reasons for RPC popularity:
a) Simple call syntax
b) Familiar semantics
c) Its specification of a well-defined interface - supports compile time type checking and
automated interface generation.
d) Ease of use clean and simple semantics
e) Generality
f) Efficiency due to rapid communication
g) Can be used as an IPC mechanism for both across machines and within a give machine
RPC MODEL:
Similar to ordinary procedure call model in the following ways:
a. Caller places arguments to the procedure
b. Control transferred to sequence of instructions
c. Procedure executed in a new execution environment
d. Control returns to the caller.
In RPC however, the procedure may be local or remote. RPC uses message passing scheme for
information exchange.
Transparency in RPC:
Both local and remote procedure calls should be (effectively) indistinguishable to programmers.
This requires:
o Semantic transparency (meaning)
o Syntactic transparency (Structural rules)
What differentiates remote from local PC?
Procedure executed in a different environment
More prone to failures
Consumes more time
These makes semantic transparency more difficult to implement.
NB:When making a RPC:
The calling environment is suspended.
Procedure parameters are transferred across the network to the environment where
the procedure is to execute.
The procedure is executed there.
When the procedure finishes, the results are transferred back to the calling
environment.
Execution resumes as if returning from a regular procedure call.
RPC between Client and Server
RPC: Steps
1. The client procedure calls a client stub passing parameters in the normal way.
2. The client stub marshals the parameters, builds the message, and calls the local OS.
3. The client's OS sends the message (using the transport layer) to the remote OS.
4. The server remote OS gives transport layer message to a server stub.
5. The server stub demarshals the parameters and calls the desired server routine.
6. The server routine does work and returns result to the server stub via normal procedures.
7. The server stub marshals the return values into the message and calls local OS.
8. The server OS (using the transport layer) sends the message to the client's OS.
9. The client's OS gives the message to the client stub
10. The client stub demarshals the result, and execution returns to the client.
Steps (diagrammatically)
Practical example using C++
RPC-Elements : Based on stubs concepts that provides a normal procedure call, abstraction is achieved
by concealing from the programs interface underlying RPC system.
Has 5 elements
The client
The client stub
The RPC Runtime communication package
The server stub
The server



Implementing RPC Mechanisms :RPC-Elements
RPC: Five elements of an RPC implementation

1.RPC: Stub Generation
2.RPC: IDL
3.RPC: Messages
4.RPC: Marshalling Arguments and Results
5.RPC: Server Management

Server implementation
Stateful servers stores clients information from one RPC to the next. Maintains state
information from one call to another call.
Stateless servers- A stateless server does not maintain any client state information. So
every request must accompany with all the necessary parameters.
Server Management - Examples
1.Steteful Servers
Read(fid, n, buffer) get n bytes of data from file whose Id is fid into the buffer named
buffer.
Write(fid, n buffer) - the server takes n bytes of data from the specified buffer, writes
into fid at the byte position currently addressed by read-write pointer.
2.Stateless Servers
Read(filename, position, n, buffer) get n bytes of data from file whose Id is filename
into the buffer named buffer.
Write(filename,position, n buffer) - the server takes n bytes of data from the specified
buffer, writes into fid at the byte position position.
RPC: Server Management
Server Creation
o Instance-per-call server They exist only for the duration of a single call. Such a server is
created by the RPC-Runtime when the call arrives. The server is deleted when the call
has been executed
o Instance-per-session servers-exist for the entire session for which the client and server
interact
o Persistent servers
o Generally remains indefinitely
o Usually shared by many clients
o Servers are created and installed before the clients
o RPC: Server Management
o Persistent servers
Each server independently exports its service by registering itself with the binding agent
When a client contacts the binding agent for a particular service, the binding agent selects a
server of that type and returns its address to the client
The client then interacts with the server. An advantage of this approach is it can improve
performance, since it interleaves requests of several clients
Care should be taken in designing the procedures so that interleaved concurrent requests from
different clients do not interfere with each other.
RPC: Parameter-Passing-Semantics:
a) Call-by-value parameters copied onto a message .The message is one carrying the procedure
call.
b) Call-by-reference passing pointers is meaningless because of the differences in address
spaces. Only possible in case of shared memory.
RPC: Call Semantics
Failures during message passing may affect the RPC, hence the need for call semantics:
Possibly Or May-Be Call Semantics caller continues after a time-out
At-Least-Once Call Semantics
Exactly-Once Call Semantics
Communication Protocols for RPCs
The Request Protocol (the R Protocol)
Nothing to return
Acknowledgement not required
Caller continues with execution after the call
Provides May-Be-Call Semantics
This RPC is asynchronous
E.g. A time server node in a DS may send synchronization messages every T seconds so
losing a message or two is acceptable
Communication Protocols for RPCs
The Request/Reply Protocol (the RR Protocol)
Suitable for simple RPCs all arguments/results fit in a single packet buffer and has less
transmission time
Based on implicit acknowledgement
Has no failure-handling capabilities
Uses timeouts-retries technique to handle failures, hence provides at-least-once call
semantics
Can support exactly-once semantics if the server keeps records of replies.
Communication Protocols for RPCs
The Request/Reply/Acknowledgement-Reply Protocol (the RRA Protocol)
Client acknowledge the receipt of reply messages and the server uses this to delete
reply information from its cache
Since acknowledgement messages may get lost, both messages (the reply and the
acknowledgement) have identifiers for matching purposes
Complicated RPCs
Involving long duration
Periodic probing of the server by the client
Periodic generation of acknowledgement by the server
Involving arguments/results that are too large
Use several physical RPCs to handle one large logical RPC
Use multidatagram messages
Server Binding
Client stub needs to know the location of the server through binding
Servers export their operations and clients import via the RPCRuntime.
Interface names (type and instance) are used for servers
Servers can be located by use of broadcasting or Binding Agent.
RPC in Heterogeneous Environments:
Three types of heterogeneity to be addressed:
a) Data representation how the particular machine represents data. E.g. 1s or 2s compliments
b) Transport protocol making the RPC system independent of the protocol
c) Control protocol must be independent of the control information in each transport packet
Delay decisions until bind time
Light Weight RPC
Recall the differences between Monolithic-kernel operating systems and microkernel operating systems.
The two types of communication traffic (cross-domain and cross-machine)?
The LRPC is a communication facility designed and optimized for cross-domain communications
Uses simple control/data transfer and simple stubs.
Designed for concurrency
Optimizations for Better Performance
Concurrent access to multiple servers using threads, early reply approach or call buffering
approach
Serving multiple request simultaneously
Reducing per call workload of servers
Reply caching of Idempotent remote procedures
Proper selection of timeout values
Proper design of RPC protocol specification
Communication