Monsoon 2022

Distributed Computing
- Topology Abstraction and Overlays

Dr. Rajendra Prasath

Indian Institute of Information Technology Sri City, Chittoor

28th September 2022 (http://rajendra.2power3.com)

 > Distributed Computing?
➤ How will you design a Distributed Algorithm?
Overview

➤ Learn to Solve using Distributed Algorithms

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 2
Recap: Distributed Systems
A Distributed System:
è A collection of independent systems that appears to
its users as a single coherent system
è A system in which hardware and software
components of networked computers communicate
and coordinate their activity only by passing
messages
è A computing platform built with many computers
that:
è Operate concurrently
è Are physically distributed (have their own failure
modes)
è Are linked by a network
è Have independent clocks

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 3

Recap: Characteristics
è Concurrent execution of processes:
è Non-determinism, Race Conditions,
Synchronization, Deadlocks, and so on
è No global clock
è Coordination is done by message exchange
è No Single Global notion of the correct time
è No global state
è No Process has a knowledge of the current global
state of the system
è Units may fail independently
è Network Faults may isolate computers that are
still running
è System Failures may not be immediately known

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 4

 What did you learn so far?
è Goals / Challenges in Message Passing systems
è Distributed Sorting / Space-Time diagram
è Partial Ordering / Total Ordering
è Causal Precedence Relation
è Concurrent Events
è Causal Ordering
è Logical Clocks vs Physical Clocks
è Global Snapshot Detection
è Termination Detection Algorithm
è Leader Election in Rings
Recap

è [Now] Topology Abstraction and Overlays …

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 5

 > About this Lecture
What do we learn today?
➤ Recap: Leader Election in Rings
➤ LCR algorithm

➤ Topology Abstraction and Overlays

➤ Interconnection Topologies
➤ Abstraction - Basic Concepts
Today’s Focus

➤ Interconnection Patterns suitable for message

propagation
➤ Types of Algorithms and their executions
➤ Measures and Metrics

Let us explore these topics è è è

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 6

 Topology Abstraction and
Overlays
in Distributed Systems

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 7

 Ring Networks
è In an oriented ring, processes have a consistent
notion of left and right

Clockwise (right)

Anti-clockwise (left)

è For example, if messages are forwarded on right

channel, they will cycle clockwise around the ring

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 8

 Why Study Rings?
• Simple starting point, easy to analyze
• Abstraction of a token ring
• Lower bounds and impossibility results for ring
topology also apply to arbitrary topologies

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 9

Interconnection Topologies
è Various Interconnection Networks
è Abstraction of the overall networks
è Message Propagation
è Distributed Processing
è Computational Complexity

è Overlays
è Sampling the underlying network topology

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 10

Basic Terminologies
è System Model: Undirected (weighted)
graph G = (V, E), where n = |V|
è Model the underlying topology in such a way
that the pattern of message passing /
communication could be efficiently handled
è Easy to maintain and apply logics on the
abstraction of the underlying topology

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 11

Physical Topology
è Physical topology
è Nodes: network nodes, routers, all end hosts
(whether participating or not)
è Edges: all LAN, WAN links, direct edges
between end hosts

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 12

Logical Topology
è Logical topology (application context)
è Nodes: end hosts where application
executes
è Edges: logical channels among these nodes
è Fully connected or any subgraph – partial
system view, needs multi-hop paths

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 13

Superimposed Topology
è Superimposed topology (also called as
“topology overlay”):
è superimposed on logical topology
è Goal: efficient information gathering,
distribution, or search (as in P2P overlays)
è Examples: ring, tree, mesh, hypercube

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 14

Interconnection Topologies

è A few examples

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 15

Interconnection Topologies (contd)
è Hypercube of Dimension 4

è k-ary d-cudes (generalized version)

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 16
Basic Concepts
è Application execution vs. control algorithm
execution, each with own events
è Control algorithm:
è for monitoring and auxiliary functions, e.g.,
reaching consensus, global state detection
(deadlock, termination etc.), check pointing and
è superimposed on application execution, but
does not interfere
è its send, receive, internal events are transparent
to application execution

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 17

Classifications
è Centralized and distributed algorithms
è Centralized: asymmetric roles; client-server
configuration; processing and bandwidth
bottleneck; point of failure
è Distributed: more balanced roles of nodes,
difficult to design perfectly distributed
algorithms (e.g., snapshot algorithms, tree-based
algorithms)

è Symmetric and asymmetric algorithms

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 18

Important Concepts
è Anonymous algorithm:
è Process ids are not used to make any execution (run-time)
decisions
è Structurally elegant but hard to design, or impossible,
(anonymous leader election is impossible)
è Uniform algorithm:
è Cannot use n, the number of processes, as a parameter
è Allows scalability; process leave/join is easy and only
neighbors need to be aware of logical topology changes
è Adaptive algorithm:
è Let k (≤ n) be the number of processes participating in the
context of a problem X when X is being executed.
Complexity should be expressible as a function of k, not n
è For example, Mutual Exclusion is a good example

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 19

Deterministic vs Nondeterministic
Deterministic vs. nondeterministic executions
è Nondeterministic exec: contains at least 1 nondeterministic
receive; deterministic execution has no nondeterministic
receive
èNondeterministic receive: can receive a message from any
source
èDeterministic receive: source is specified
Difficult to reason with
è Asynchronous system: re-execution of deterministic program will produce
same partial order on events ((used in debugging, unstable predicate
detection etc.)
è Asynchronous system: re-execution of nondeterministic program may
produce different partial order (unbounded delivery times and
unpredictable congestion, variable local CPU scheduling delays)

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 20

Synchronous vs Asynchronous
Synchronous:
è Upper bound on message delay
è Known bounded drift rate of clock with respect to the
real time
è Known upper bound for process to execute a logical
step

Asynchronous: above criteria not satisfied

è Spectrum of models in which some combo of criteria
satisfied
è Algorithm to solve a problem depends greatly on this
model

Distributed systems are inherently asynchronous

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 21
Algorithms / Channels
è Wait-free algorithms (for synchronization
operations)
è resilient to n - 1 process failures, i.e., operations of any
process must complete in bounded number of steps,
irrespective of other processes
è very robust, but expensive
è possible to design for mutual exclusion
è may not always be possible to design, for example,
the producer-consumer problem
è Communication channels
è point-to-point: FIFO, non-FIFO
è At application layer, FIFO usually provided by
network stack

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 22

Process failures (Sync + Async syst.)
è Fail-stop: Properly functioning process stops execution.
Other processes learn about the failed process (thru some
mechanism)
è Crash: Properly functioning process stops execution. Other
processes do not learn about the failed process
è Receive omission: Properly functioning process fails by
receiving only some of the messages that have been sent to
it, or by crashing.
è Send omission: Properly functioning process fails by sending
only some of the messages it is supposed to send, or by
crashing. Incomparable with receive omission model.
è General omission: Send omission + receive omission
è Byzantine (or malicious) failure: Process may (mis) behave
anyhow, including sending fake messages. Authentication
facility => If a faulty process claims to have received a
message from a correct process, that is verifiable.

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 23

Process Failures (contd.)
è Process failures à Timing failures (sync systems):
è General omission failures, or process violating bounds on
time to execute a step
è More severe than general omission failures
Failure models influence design of algorithms
è Link failures
è Crash failure: Properly functioning link stops carrying
messages
è Omission failure: Link carries only some of the messages
sent on it, not others
è Byzantine failure: Link exhibits arbitrary behavior,
including creating fake and altering messages sent on it
è Link failures à Timing failures (sync systems):
è messages delivered faster/slower than specified
behavior

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 24

Complexity Measures
è Each metric specified using
è lower bound (Omega)
è upper bound (big O)
è exact bound(Theta)

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 25

Metrics
è Space complexity per node

è System-wide space complexity (= n space

complexity per node). E.g., worst case may
never occur at all nodes simultaneously!

è Time complexity per node

è System-wide time complexity. Do nodes
execute fully concurrently?

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 26

Metrics
è Message complexity
è Number of messages (affects space complexity of
message overhead)
è Size of messages (affects space complexity of
message overhead + time component via increased
transmission time)
è Message time complexity: depends on number of
messages, size of messages, concurrency in sending
and receiving messages
è Other metrics: # send and # receive events; #
multicasts, and implementation related metrics?
è (Shared memory systems): size of shared memory; #
synchronization operations
DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 27
Summary
è Racap: Leader Election in Rings
è Leader Election Problem
è LCR algorithm / O(nlogn) algorithm using probes

è Topology Abstraction and Overlays

è Various Interconnection Topologies
è Abstraction - Basic Concepts
è Interconnection Patterns suitable for message
propagation
è Types of Algorithms and their executions
è Measures and Metrics

è Many more to come up … stay tuned in !!

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 28

Penalties
➤ Every Student is expected to strictly follow a
fair Academic Code of Conduct to avoid
penalties

➤ Penalties is heavy for those who involve in:

➤ Copy and Pasting the code
➤ Plagiarism (copied from your neighbor or
friend – in this case, both will get “0” marks for
that specific take home assignments)
➤ If the candidate is unable to explain his own
solution, it would be considered as a “copied
case”!!
➤ Any other unfair means of completing the
assignments
DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 29
Help among Yourselves?
• Perspective Students (having CGPA above 8.5
and above)
• Promising Students (having CGPA above 6.5
and less than 8.5)

• Needy Students (having CGPA less than 6.5)

• Can the above group help these students? (Your
work will also be rewarded)

• You may grow a culture of collaborative

learning by helping the needy students

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 30

How to reach me?
è Please leave me an email:
rajendra [DOT] prasath [AT] iiits [DOT] in

è Visit my homepage @
è https://www.iiits.ac.in/people/regular-
faculty/dr-rajendra-prasath/

(OR)

è http://rajendra.2power3.com

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 31

Assistance

➤ You may post your questions to me at any time

➤ You may meet me in person on available time or

with an appointment

➤ You may ask for one-to-one meeting

Best Approach

➤ You may leave me an email any time

(email is the best way to reach me faster)

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 32

 Questions
It’s Your Time

DC - Monsoon 2022 @ IIIT Sri City 28/09/2022 33