Distributed Systems

Remote Procedure Calls

(RPC)
Peter Druschel
(with thanks to Yiting Xia)

1
Remote Procedure Call (RPC)
• Goal
• Invoke a procedure (function) on a remote server
• Make it appear similar to an ordinary procedure (function) call

Network

Client Server

• Invokes procedure • Executes procedure

2
Remote Procedure Call (RPC)
• Goal
• Invoke procedure on a remote server
• Make it appear similar to an ordinary procedure call
Network
f(x,y) { Client Server
z = x + y
… x = 5
return z y = 8
Request
} z = f(x,y)
f(x,y) {
x = 5 z = x + y
y = 8 …
z = f(x,y) Response return z
print(z) }
print(z) 3
 RPC Software Structure
Client
Application Network Server Handler

z = f(x,y)

Request

f(x,y) {
z = x + y
return z
}
Response

print(z)
4
 RPC Software Structure
Client
Client Stub Network Server Handler
Application
z = f(x,y)

Marshal
Request
Convert arguments to binary format
e.g., x = 5, y = 7

000…101 000…111 f(x,y) {

z = x + y
1st 4 bytes 2nd 4 bytes return z
}
Also called serialization
Response
Different functions have different #para,
types  marshalling is function-specific!
Pack f, para1, para2, …
print(z)
5
 RPC Software Structure
Client Client RPC
Client Stub Network Server Handler
Application Library
z = f(x,y)

Marshal

Send Request

• Have arguments as byte string

• Have name of function to invoke f
• Connect to the server
• Transmit bytes to server (sockets) f(x,y) {
z = x + y
return z
}
Response

print(z)
6
 RPC Software Structure
Client Client RPC
Client Stub Network Server Handler
Application Library
z = f(x,y)

Marshal

Send Request

………
After sending bytes, wait for
Wait
the response f(x,y) {
…… z = x + y
return z
}
Response

print(z)
7
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Handler
Application Library Library

……
z = f(x,y)

Marshal Continually listening

Wait
Accept incoming connections
Send Request

……
………
Wait
f(x,y) {
…… z = x + y
return z
}
Response

print(z)
8
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive
Wait
• Read bytes off network f(x,y) {
…… • Inspect name of function z = x + y
• Dispatch payload (bytes) return z
}
Response

print(z)
9
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive
Wait Unmarshal f(x,y) {
…… z = x + y
Reverse of marshal return z
convert bytes to variables }
Response 000…101 000…111

1st 4 bytes 2nd 4 bytes

x = 5, y = 7
print(z)
Also called deserialization 10
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive
Wait Unmarshal
f(x,y) {
……
z = x + y
return z
}
Response

print(z)
11
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive
Wait Unmarshal f(x,y) {
…… z = x + y
return z
}
Marshal
Response
• Receive return value from f
• Want to send result to client
• Pack response value into byte string

print(z)
12
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive

……
Wait Unmarshal f(x,y) {
……
Wait z = x + y
return z

……
}
Marshal
Response
Send

• Send result back to client on

existing connection
print(z) • Write bytes
13
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive

……
Wait Unmarshal f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}
Marshal
Response
Send

……
Receive

• Read bytes off network Wait

print(z)
• Hand back to waiting client

……
14
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive

……
Wait Unmarshal f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}
Marshal
Response
Send
Convert bytes to variable

……
Receive
Unmarshal Wait
print(z)

……
15
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive

……
Wait Unmarshal f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}
Marshal
Response
Send

……
Receive
Unmarshal Wait
print(z)

……
16
 RPC Software Structure
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
• Function-specific stubs
Receive
• RPC library deals with networking

……
and dispatching Wait Unmarshal f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}
Marshal
Response
Send

……
Receive
Unmarshal Wait
print(z)

……
17
More Details
• Access control and authentication
• Argument validation
• Client/server authentication, data encryption
• Complex data types
• Marshalling is tricky for arrays, pointers, objects, etc.
• Locate server
• Client needs to bind to server’s network adddress
• Name service maintains binding of function name -> IP address, port
• Multi-threading
• Client may have multiple threads sending to server  match response to
waiting thread
• Server may create one thread per request or use a thread pool 18
Key Properties
• Easy to write applications
• Clients invoke procedures
• Familiar programming model
• A good match for a lot of distributed applications
• Client/server commonplace
• Hides network communication details from applications
• Marshalling and unmarshalling by stubs
• Networking, multiplexing, dispatching by RPC library
• Widely used
• gRPC, JSON-RPC, Apache Thrift
• Support in many modern programming languages 19
Failures
• Goal
• Hide details from client applications and server handlers (as far as possible)
• Failure scenarios
• Server/client may crash and reboot
• Network transmission
• Dropped messages (a request or response just disappears)
• Delayed or duplicate messages
• Re-ordered messages (client sends A then B, server receives B then A)
• Network unavailable (temporarily unable to connect)

20
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library

……
z = f(x,y)

Marshal Wait
Send Request

……
………
Receive

……
Wait Unmarshal f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}
Marshal
Response
Send

……
Receive
Unmarshal Wait
print(z)

……
21
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

22
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

………
Client:
• No response
Wait
• Waits forever
… … … ... … …

23
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

………
Receive

……
Wait Unmarshal f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}
Marshal
Response
Send

……
Wait

……
24
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

………
Client: Receive
• No response

……
Wait Unmarshal
• Waits forever f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}
Marshal
Response
Send

……
Wait

……
25
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

………
Receive

……
Wait Unmarshal f(x,y) {
… … … ... … …
Wait z = x + y
return z

……
}

26
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

………
Client: Receive
• No response

……
Wait Unmarshal
• Waits forever f(x,y) {
… … … ... … …
Wait z = x + y
The cause of failure is unknown return z
to the client

……
}

27
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

………
Client: Receive
• No response

……
Wait Unmarshal
• Waits forever f(x,y) {
… … … ... … …
Wait z = x + y
The cause of failure is hidden return z
from client

……
}
Solution:
• Timeout (max wait time)
• Retry

28
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

……
?
Wait

……
Retry Request

Receive
f(x,y) {
z = x + y
Response return z
Send }
Receive
print(z)
29
 Failure Example
Client Client RPC Server RPC
Client Stub Network Server Stub Server Handler
Application Library Library
z = f(x,y)

Marshal

Send Request

……
?
Wait
Problem:
• Function executed twice or not at all ……
• May update state twice Request
• E.g., account.balance += 10 Retry
Receive
f(x,y) {
z = x + y
Response return z
Send }
Receive
print(z)
30
RPC Semantics
• At-least-once
• For each RPC, repeat request message until success; else, client gives up and raises
an exception
• If successful, server function was executed at least once
• At-most-once
• For each RPC, repeat request message until success, but the server function is
executed (to completion) only once
• If successful, function was executed once; else, may have executed or not
• Exactly-once
• For each RPC, if successful server function is executed exactly once
• Requires fault-tolerant server function
• Difficult to implement efficiently in practice
31
At-least-once
• Simple implementation
• Client re-transmits request until response arrives
• Raise exception if too many unsuccessful retransmissions
• Server keeps no state about past invocations

32
At-least-once
• Client sends a “debit $10 from account” RPC

33
At-least-once
• put(x,value), then get(x): expect answer to be value

34
At-least-once
• put(x,value), then get(x): expect answer to be value

35
At-least-once: Use Cases
• Read-only operations
• get(x) only, no writes
• Idempotent operations
• Stateless computation, e.g., z = x + y
• Or if application has its own mechanisms to cope with uncertainty

36
At-most-once
• In RPC paper and also Lab 1!
• Duplicate detection
• Server detects duplicate requests
• Returns previous reply instead of re-running server function
• How?
• Server sees the same function, the same arguments twice?
• No! Applications may intend to call the same function with the same arguments
twice

37
Duplicate Detection
• Client include a unique transaction ID (xid) with each RPC request
• Client uses the same xid for retransmitted requests

if seen[xid]:
retval = old[xid]
else:
retval = handler()
old[xid] = retval
seen[xid] = true

return retval

38
Unique XIDs
• Combine a unique client ID (e.g., IP address) with the current time of
the day
• Combine a unique client ID with a monotonic sequence number
• If client crashes and reboots, can it reuse the same client ID?
• No!
• Big random number
• Probabilistic, no guarantee

39
Discard Server States
• Problem: seen[]and old[]will grow without bound
• Client acknowledgement
• When the client gets a response to xid, it won’t retransmit the request
• Client can notify server “I’m done with xid, delete it”
• Have to tell server about each and every xid?
• Use cumulative ACKs
• Client sends “seen all replies <= X”
• Like TCP sequence numbers and ACKs

40
Concurrent Requests
• Problem: how to handle a duplicate request while the original is still
executing?
• Server doesn’t know reply yet. We don’t want to run the procedure twice!
• Add an INPROGRESS flag per executing RPC
• Server waits for the procedure to finish, or ignores it

41
Server Crash or Reboot
• Problem: server may crash and reboot
• After server restart:
• Can’t tell whether handler() was called
• Client will re-send request (no response due to crash)
• Server MUST reply with exception!

42
Server Crash or Reboot
• How to decide if a request is a retransmission of one sent before crash?
• Unique nonce
• Server has a number that uniquely identifies epochs (periods between
successive reboots)
• Client obtains server’s nonce when it first connects during binding
• Client sends server nonce in every RPC request
• Server checks nonce
• If equal, then any previous transmission will be in server’s seen[]and old[]
• If not equal, return exception
• Client unsure about state after server crash
• Maybe you transferred money, maybe you didn’t, but never transmitted twice
• Explicit query of state at the server
43
Server Crash or Reboot
• How to ensure unique nonce?
• Server can store nonce in persistent memory
• Use boot time

44
Exactly-once
• Requires at-most-once semantics plus transactional server procedure
• Fault Tolerance (transactions) covered in later lectures
• Not implemented in practice
• Low performance

45
Example: Lock server request to
acquire lock
• At-least-once
• Retransmitted acquire request may result in failure (lock is already taken)
• Delayed request may retake the lock after it’s released  deadlock!
• Lock application needs to deal with duplicates, no help from RPC semantics
• At-most-once
• In case of exception, client may have the lock or not
• Need application-specific protocol
• Ask server “do I have the lock or not?”
• Server needs to remember states across reboots
• Store locks and their holders on disk

46
Summary
• RPC
• A simple abstraction as if invoking procedures locally
• Software structure: stubs and RCP library
• Failure handling
• At-least-once semantics
• At-most-once semantics
• Exactly-once semantics
• Enjoy Lab 1!

47