Distributed System

Dr\ Eman Monir

Faculty of Computers and Artificial Intelligence

Benha University
Winter 2023
 Problems with sockets

Sockets interface is straightforward

• [connect]
• read/write
• [disconnect]

BUT … it forces read/write mechanism

• We usually use a procedure call

To make distributed computing look more like

centralized:
• I/O is not the way to go

3/5/2023
 RPC
1984: Birrell & Nelson
• Mechanism to call procedures on other machines

Remote Procedure Call

Goal: it should appear to the programmer

that a normal call is taking place

3/5/2023
 How do regular
procedure calls work
in programming
languages?
 Regular procedure calls
Machine instructions for call & return but the
compiler really makes the procedure call abstraction
work:
• Parameter passing
• Local variables
• Return data

3/5/2023
 Regular procedure calls
You write:
x = f(a, “test”, 5);

The compiler parses this and generates code to:

a. Push the value 5 on the stack
b. Push the address of the string “test” on the stack
c. Push the current value of a on the stack
d. Generate a call to the function f
In compiling f, the compiler generates code to:
a. Push registers that will be clobbered on the stack to save the values
b. Adjust the stack to make room for local and temporary variables
c. Before a return, unadjust the stack, put the return data in a register, and issue a
return instruction

3/5/2023
 Implementing RPC
No architectural support for remote procedure calls
Simulate it with tools we have
(local procedure calls)

Simulation makes RPC a

language-level construct
instead of an
operating system construct

3/5/2023
 Implementing RPC
The trick:

Create stub functions to make it appear to the user that

the call is local

Stub function contains the function’s interface

3/5/2023
 Stub functions
1. Client calls stub (params on stack)

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
2. Stub marshals params to net message

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
3. Network message sent to server

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
4. Receive message: send to stub

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
5. Unmarshal parameters, call server func

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
6. Return from server function

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
7. Marshal return value and send message

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
8. Transfer message over network

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
9. Receive message: direct to stub

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Stub functions
10. Unmarshal return, return to client code

client functions server functions

server stub
client stub
(skeleton)

network routines network routines

client server
 Benefits
• Procedure call interface

• Writing applications is simplified

• RPC hides all network code into stub functions
• Application programmers don’t have to worry about
details
• Sockets, port numbers, byte ordering

• RPC: presentation layer in OSI model

3/5/2023
RPC has issues
 Parameter passing
Pass by value
• Easy: just copy data to network message

Pass by reference
• Makes no sense without shared memory

3/5/2023
 Passing Value Parameters

• Steps involved in doing remote computation through RPC

2-8

3/5/2023
 Pass by reference?
1. Copy items referenced to message buffer
2. Ship them over
3. Unmarshal data at server
4. Pass local pointer to server stub function
5. Send new values back

To support complex structures

• Copy structure into pointerless representation
• Transmit
• Reconstruct structure with local pointers on server

3/5/2023
 Representing data
No such thing as
incompatibility problems on local system

Remote machine may have:

• Different byte ordering
• Different sizes of integers and other types
• Different floating point representations
• Different character sets
• Alignment requirements

3/5/2023
 Representing data
IP (headers) forced all to use big endian byte ordering
for 16 and 32 bit values
• Most significant byte in low memory
• Sparc, 680x0, MIPS, PowerPC G5
• x86/Pentiums use little endian

main() { Output on a Pentium:

unsigned int n; 44, 33, 22, 11
char *a = (char *)&n;
Output on a PowerPC:
n = 0x11223344; 11, 22, 33, 44
printf("%02x, %02x, %02x, %02x\n",
a[0], a[1], a[2], a[3]);
}

3/5/2023
 Representing data
Need standard encoding to enable communication
between heterogeneous systems
• e.g. Sun’s RPC uses XDR (eXternal Data Representation)
• ASN.1 (ISO Abstract Syntax Notation)

3/5/2023
 Representing data
Implicit typing
• only values are transmitted, not data types or parameter
info
• e.g., Sun XDR

Explicit typing
• Type is transmitted with each value
• e.g., ISO’s ASN.1, XML

3/5/2023
 Where to bind?
Need to locate host and correct server process

3/5/2023
 Where to bind? – Solution 1
Maintain centralized DB that can locate a
host that provides a particular service
(Birrell & Nelson’s 1984 proposal)

3/5/2023
 Where to bind? – Solution 2
A server on each host maintains a DB of locally
provided services

Solution 1 is problematic for Sun NFS – identical file

servers serve different file systems

3/5/2023
 Transport protocol
Which one?

• Some implementations may offer only one

(e.g. TCP)

• Most support several

• Allow programmer (or end user) to choose

3/5/2023
 When things go wrong
• Local procedure calls do not fail
• If they core dump, entire process dies

• More opportunities for error with RPC:

• Transparency breaks here

• Applications should be prepared to deal with RPC failure

3/5/2023
 When things go wrong
• Semantics of remote procedure calls
• Local procedure call: exactly once

• A remote procedure call may be called:

• 0 times: server crashed or server process died before
executing server code
• 1 time: everything worked well
• 1 or more: excess latency or lost reply from server and
client retransmission

3/5/2023
 More issues
Performance
• RPC is slower … a lot slower

Security
• messages visible over network
• Authenticate client
• Authenticate server

3/5/2023
 Programming with RPC
Language support
• Most programming languages (C, C++, Java, …) have no
concept of remote procedure calls
• Language compilers will not generate client and server
stubs

Common solution:
• Use a separate compiler to generate stubs (pre-
compiler)

3/5/2023
 Interface Definition Language
• Allow programmer to specify remote procedure
interfaces
(names, parameters, return values)

• Pre-compiler can use this to generate client and server

stubs:
• Marshaling code
• Unmarshaling code
• Network transport routines
• Conform to defined interface
• Similar to function prototypes

3/5/2023
 RPC compiler
client code (main)

client stub

data conv. compiler client

RPC
IDL compiler
headers

data conv. compiler server

server skeleton
Code you write

Code RPC compiler generates server functions

3/5/2023
 Writing the program
Client code has to be modified
• Initialize RPC-related options
• Transport type
• Locate server/service
• Handle failure of remote procedure call

Server functions
• Generally need little or no modification

3/5/2023
 RPC API
What kind of services does an RPC system need?
• Name service operations
• Export/lookup binding information (ports, machines)
• Support dynamic ports

• Binding operations
• Establish client/server communications using appropriate
protocol (establish endpoints)

• Endpoint operations
• Listen for requests, export endpoint to name server

3/5/2023
 RPC API
What kind of services does an RPC system need?

• Security operations
• Authenticate client/server
• Internationalization operations
• Marshaling/data conversion operations
• Stub memory management
• Dealing with “reference” data, temporary buffers
• Program ID operations
• Allow applications to access IDs of RPC interfaces
3/5/2023
 RPC in Practice - DCE RPC
• The Distributed Computing Environment (DCE) RPC is
developed by the Open Software Foundation
(OSF)/Open Group.
• DCE is a middleware executing as an abstraction
layer between (network) operating systems and
distributed applications.
• Microsoft derived its version of RPC from DCE RPC
(e.g., MIDL compiler, etc.)
• DCE includes a number of services:
• Distributed file service
• Directory service
• Security service
3/5/2023
• Distributed time service
 DCE RPC
• The main goal of the DCE RPC is to make it possible
for a client to access a remote service by simply
calling a local procedure.
• Developing a RPC application:
• Writing a client and a server: Let the developer
concentrate on only the client- and server-specific code;
let the RPC system (generators and libraries) do the rest.
Interface Definition Language (IDL) is used to specify the
variables (data) and functions (methods).
• Binding a client to a server
• Performing an RPC

3/5/2023
 Writing a Client and a Server

2-14

• The steps in writing a client and a server in DCE RPC.

3/5/2023
 Client-to-Server Binding (DCE RPC)
• Issues: Client must locate server machine and
locate the server.
• Execution of Client and Server
• The server registers its procedures with the
portmapper.
• Client must locate server machine: The client contacts
the portmapper to determine which port the server is
listening to.
• The client communicates with the server on the
assigned port.
• DCE uses a separate daemon for each server
machine.
3/5/2023
 Binding a Client to a Server
• Client-to-server binding in DCE.
2-15

3/5/2023
 Sun RPC/ONC (Open Network Computing) RPC

• It is designed for client-server communication over Sun

NFS network file system.
• UDP or TCP can be used. Generally, if UDP is used, the
message length is restricted to 64 KB, but 8 - 9 KB in
practice.
• The Sun XDR is originally intended for external data
representation (XDR).
• Valid data types supported by XDR include int, unsigned int,
long, structure, fixed array, string (null terminated char *),
binary encoded data (for other data types such as lists).

3/5/2023
 Sun/ONC RPC

• Steps to create a Sun RPC application:

1. Create the IDL file (sum.x), the client main code (rsum.c),
and the server function code (sum_serv.c)
2. Run the RPC generator (rpcgen sum.x) to generate client
stub (sum_clnt.c), server stub (sum_svc.c), header file
(sum.h), and data coversion file (sum_xdr.c).
3. Compile the client and server programs (make).
4. Start the server (server).
5. Run the client (rsum hostname 10).

3/5/2023
 Interface Definition Language

• The notation of Sun RPC IDL:

 a program number and a version number are
supplied.
 The procedure number is used as a procedure
definition.
 Single input parameter and output result are
being passed.

3/5/2023
 Files interface in ONC IDL
const MAX = 1000;
typedef int FileIdentifier;
typedef int FilePointer;
typedef int Length; struct readargs {
struct Data { FileIdentifier f;
int length; FilePointer position;
char buffer[MAX]; Length length;
}; };
struct writeargs {
FileIdentifier f; program FILEREADWRITE {
FilePointer position; version VERSION {
Data data; void WRITE(writeargs)=1; 1
}; Data READ(readargs)=2; 2
}=2;
} = 9999;
 Sun/ONC RPC
• The interface compiler rpcgen can be used to generate
the following from the interface definition.
 client stub procedures (sum_clnt.c)
 server main procedure, dispatcher (system calls) and server
stub procedures (sum_svc.c)
 XDR marshalling and unmarshalling procedures used by
dispatcher and client, server stub procedures. (sum_xdr.c)
• Binding:
 portmapper records program number, version number, and
port number.
 If there are multiple instance running on different machines,
clients make multicast remote procedure calls by
broadcasting them to all the port mappers.
3/5/2023
ONC RPC Interface Compiler
 ONC RPC
• Authentication:
 Additional fields are provided in the request-reply
message.
 Server program should check the authentication and then
execute.
 Different authentication protocols can be supported -
none, UNIX style, shared key, Kerberos style.
 A field in RPC header indicates which style is used.

3/5/2023
 Example (ONC RPC)
• long sum(long) example
 client localhost 10
 result: 55
• Need RPC specification file (sum.x)
 defines procedure name, arguments & results
• Run (interface compiler) rpcgen sum.x
 generates sum.h, sum_clnt.c, sum_xdr.c, sum_svc.c
 sum_clnt.c & sum_svc.c: Stub routines for client & server
 sum_xdr.c: XDR (External Data Representation) code takes
care of data type conversions

3/5/2023
 RPC IDL File (sum.x)
struct sum_in {
long arg1;
};
struct sum_out {
long res1;
};
program SUM_PROG {
version SUM_VERS {
sum_out SUMPROC(sum_in) = 1; /* procedure number = 1*/
} = 1; /* version number = 1 */
} = 0x32123000; /* program number */

3/5/2023
 Example (Sun RPC)
• Program-number is usually assigned as follows:
 0x00000000 - 0x1fffffff defined by SUN
 0x20000000 - 0x3fffffff defined by user
 0x40000000 - 0x5fffffff transient
 0x60000000 - 0xffffffff reserved

3/5/2023
 RPC Client Code (rsum.c)
#include ''sum.h''
main(int argc, char* argv[]) {
CLIENT* cl; sum_in in; sum_out *outp;
// create RPC client handle; need to know server's address
cl = clnt_create(argv[1], SUM_PROG, SUM_VERS, ''tcp'');
in.arg1 = atol(argv[2]); // number to be squared
// Call RPC; note convention of RPC function naming
if ( (outp = sumproc_1(&in, cl)) == NULL)
err_quit(''%s'', clnt_sperror(cl, argv[1]);
printf(''result: %ld\n'', outp->res1);
}
3/5/2023
 RPC Server Code (sum_serv.c)
#include "sum.h"
sum_out* sumproc_1_svc (sum_in *inp, struct svc_req *rqstp)
{ // server function has different name than client call
static sum_out out;
int i;
out.res1 = inp->arg1;
for (i = inp->arg1 - 1; i > 0; i--)
out.res1 += i;
return(&out);
}
// server's main() is generated by rpcgen

3/5/2023
 Compilation Linking
• rpcgen sum.x
• cc -c rsum.c -o rsum.o
• cc -c sum_clnt.c -o sum_clnt.o
• cc -c sum_xdr.c -o sum_xdr.o
• cc -o client rsum.o sum_clnt.o sum_xdr.o
• cc -c sum_serv.c -o sum_serv.o
• cc -c sum_svc.c -o sum_svc.o
• cc -o server sum_serv.o sum_svc.o sum_xdr.o

3/5/2023
 Internal Details of ONC RPC
• Initialization
 Server runs: register RPC with port mapper on server host
(rpcinfo –p)
 Client runs: clnt_create contacts server's port mapper and
establishes TCP/UDP connection with server
• Client
 Client calls local procedure (client stub: sumproc_1), that is
generated by rpcgen. Client stub packages arguments, puts
them in standard format (XDR), and prepares network
messages (marshaling).
 Network messages are sent to remote system by the client
stub.
 Network transfer is accomplished with TCP or UDP.
3/5/2023
 Internal Details of ONC RPC
• Server
 Server stub (generated by rpcgen) unmarshals arguments
from network messages. Server stub executes local
procedure (sumproc_1_svc) passing arguments received
from network messages.
 When server procedure is finished, it returns to server stub
with return values.
 Server stub converts return values (XDR), marshals them
into network messages, and sends them back to client
• Back to Client
 Client stub reads network messages from kernel
 Client stub returns results to client function

3/5/2023
Sending data over the network

62
 Stream of bytes
struct item { char name[64];
unsigned long id; int number_in_stock; float rating; double
price;
} scratcher = {

"Bear Claw Black Telescopic Back Scratcher,"

,00120
,332
,4.6
5.99
}
Is stored in memory as:
6 43 20 72 61 65 42c 61 77 20 42 6c 61 63 6b ... 20 54

3/5/2023 63
 Representing data
No such thing as
incompatibility problems on local system

Remote machine may have:

– Different byte ordering
– Different sizes of integers and other types
– Different floating point representations
– Different character sets
– Alignment requirements

3/5/2023 64
 Representing data
IP (headers) forced all to use big endian byte ordering for 16- and 32-bit values
Big endian: Most significant byte in low memory
IP headers use big endian
– SPARC < V9, Motorola 680x0, older PowerPC

Little endian: Most significant byte in high memory

– Intel/AMD IA-32, x64

Bi-endian: Processor may operate in either mode

– ARM, PowerPC, MIPS, SPARC V9, IA-64 (Intel Itanium)

main() {
unsigned int n; Output on an Intel CPU:
44, 33, 22, 11
char *a = (char *)&n;

n = 0x11223344; Output on a PowerPC:

11, 22, 33, 44
printf("%02x, %02x, %02x, %02x\n",
a[0], a[1], a[2], a[3]);
}

3/5/2023 65
 Representing data: serialization
Need standard encoding to enable communication between
heterogeneous systems
•Serialization
– Convert data into a pointerless format: an array of bytes

•Examples
– XDR (eXternal Data Representation), used by ONC RPC
– JSON (JavaScript Object Notation)
– W3C XML Schema Language
– ASN.1 (ISO Abstract Syntax Notation)
– Google Protocol Buffers

3/5/2023 66
 Serializing data
Implicit typing
–only values are transmitted, not data types or parameter info
–e.g., ONC XDR (RFC 4506)

Explicit typing
–Type is transmitted with each value
–e.g., ISO’s ASN.1, XML, protocol buffers, JSON

Marshaling vs. serialization – almost synonymous:

Serialization: converting an object into a sequence of bytes that can be sent over
a network

Marshaling: bundling parameters into a form that can be reconstructed (unmarshaled)

by another process. May include object ID or other state. Marshaling uses
serialization.

3/5/2023 67
 XML: eXtensible Markup Language
<ShoppingCart>
<Items>
<Item>
<ItemID> 00120 </ItemID>
<Item> Bear Claw Black Telescopic Back Scratcher </Item>
<Price> 5.99 </Price>
/<Item>
<item>
<ItemID> 00121 </ItemID>
<Item> Scalp Massager </Item>
<Price> 5.95 </Price>
/<Item>
/<Items>
/<ShoppingCart>

Benefits: Problems:
– Human-readable –Verbose: transmit more data than needed
– Human-editable –Longer parsing time
– Interleaves structure with text (data) –Data conversion always required for
numbers

3/5/2023 68
 JSON: JavaScript Object Notation
• Lightweight (relatively efficient) data interchange format
– Introduced as the “fat-free alternative to XML”
– Based on JavaScript
• Human writeable and readable
• Self-describing (explicitly typed)
• Language independent
• Easy to parse
• Currently converters for 50+ languages
• Includes support for RPC invocation via JSON-RPC

3/5/2023 69
 JSON Example
{"menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem[ :"
"{value": "New," "onclick": "CreateNewDoc,}")(
"{value:" "Open", "onclick": "OpenDoc,}")(
"{value:" "Close", "onclick": "CloseDoc}")(
]
}
}}

from json.org/example.html
3/5/2023 70
 Google Protocol Buffers
• Efficient mechanism for serializing structured data
– Much simpler, smaller, and faster than XML

• Language independent

• Define messages
– Each message is a set of names and types

• Compile the messages to generate data access classes for

your language

• Used extensively within Google. Currently over 48,000

different message types defined.
– Used both for RPC and for persistent storage
3/5/2023 71
 Example (from the Developer Guide)
message Person{
required string name = 1;
required int32 id = 2;
optional string email = 3;

enum PhoneType { MOBILE = 0;

HOME = 1;
WORK = 2;
}

message PhoneNumber { required string number =

1;
optional PhoneType type = 2 [default = HOME];
}

repeated PhoneNumber phone = 4;

}

3/5/2023 72
 Example (from the Developer Guide)

Person person; person.set_name("John Doe");

person.set_id(1234);
person.set_email("[email protected]");
fstream output("myfile", ios::out | ios::binary);
person.SerializeToOstream(&output);

3/5/2023 73
 Efficiency example (from the Developer Guide)
<person> person{
<name>John Doe</name> name: "John Doe"
<email>[email protected]</email> email: "[email protected]"
/<person> }

XML version Text (uncompiled) protocol buffer

• Binary encoded message: ~28 bytes long, 100-200 ns to parse

• XML version: ≥69 bytes, 5,000-10,000 ns to parse
• In general,
– 10-3x smaller data
– 100-20times faster to marshal/unmarshal
– Easier to use programmatically

3/5/2023 74
 Advantages of Remote Procedure Call
Some of the advantages of RPC are as follows −
• Remote procedure calls support process oriented
and thread oriented models.
• The internal message passing mechanism of RPC is
hidden from the user.
• The effort to re-write and re-develop the code is
minimum in remote procedure calls.
• Remote procedure calls can be used in distributed
environment as well as the local environment.
• Many of the protocol layers are omitted by RPC to
improve performance.

3/5/2023
 Disadvantages of Remote Procedure Call

Some of the disadvantages of RPC are as follows −

• The remote procedure call is a concept that can be

implemented in different ways. It is not a standard.
• There is no flexibility in RPC for hardware
architecture. It is only interaction based.
• There is an increase in costs because of remote
procedure call.

3/5/2023