Parallel and

Distributed
Computing
Dr. Sulaiman Syed Mohamed
Professor
Alliance University, Bangalore
 Motivation for distributed computing

By distributing tasks across multiple

Efficiency machines, distributed computing can
significantly reduce processing time
Fault and improve overall performance,
tolerance especially for complex computations.

Scalability
If one machine in a distributed Distributed systems can easily
system fails, the others can handle increasing amounts of data
continue to operate, ensuring and processing needs by adding
higher reliability and more machines to the network. This is
availability. crucial for applications with growing
user bases or data volumes.
Applications of distributed systems
• Finance and Commerce: Amazon, eBay, Online Banking, E
Commerce websites.
• Information Society: Search Engines, Wikipedia, Social
Networking, Cloud Computing.
• Cloud Technologies: AWS, Salesforce, Microsoft Azure, SAP.
• Entertainment: Online Gaming, Music, youtube.
• Healthcare: Online patient records, Health Informatics.
• Education: E-learning.
• Transport and logistics: GPS, Google Maps.
• Environment Management: Sensor technologies.
Definition
• A distributed system is a collection of independent computers
that appear to users as a single coherent system.
• Autonomous processors communicating over a
communication network
• Some characteristics
• Components are autonomous (independent)
• Components communicate and coordinate through message
passing
• No common physical clock
• No shared memory
• Geographical separation
• Components can fail independently
 Message Passing Models

Synchronous Model Asynchronous Model

Fixed bounds on: No bounds on:
1.Time to execute each step 1.Process execution speeds
2.Message transmission delay 2.Message transmission
3.Clock drift rate delays
Easier to design algorithms but 3.Clock drift rates
less realistic No timing assumptions
More realistic but harder to
design algorithms
Time to execute each step
• In a synchronous model, we assume that every
computational step on any processor takes a known
maximum amount of time.
• We know the upper limit on how long a calculation will
take.
Message transmission
• It might be faster delay
sometimes, but it will never exceed
this bound.
• Similarly, the time it takes for a message to travel from one
computer to another is also bounded.
• We know the maximum amount of time a message will be in
transit.
• Again, it might arrive quicker, but there's a guaranteed upper
limit.
Clock drift rate
• Each computer in a distributed system has its own
clock.
• These clocks are rarely perfectly synchronized.
• Clock drift refers to how much these clocks can deviate
from each other over time.
• The synchronous model assumes a known maximum
rate at which these clocks can drift apart.
• This means we can predict the largest possible
difference between any two clocks in the system.
Why is it Easier to Design?
• These fixed bounds make algorithm design significantly
easier.
• Because you have guarantees about timing
• You can reason about the system's behavior more
precisely.
• You can, for example, guarantee that a particular
operation will complete within a certain timeframe, or
that all messages sent within a certain round will have
arrived.
• This simplifies proving correctness and analyzing the
performance of algorithms.
Why is it Less Realistic?
• These perfect timing guarantees rarely hold in the real
world.
• Network delays can be unpredictable
• Processor speeds can vary
• Clock drift is inevitable and often difficult to precisely
bound.

Real-world distributed systems are inherently

asynchronous, things don't happen at predictable,
coordinated times.
It is dangerous to assume they are synhronized
 Synchronous Message Passing
Synchronous message passing involves a tightly coordinated interaction
between the sender and receiver.
The key characteristics include:
• Timing Coordination: Before proceeding with execution, the sender waits
for the recipient to confirm receipt of the message or finish processing it.
• Request-Response Pattern: often use a request-response paradigm in
which the sender sends a message requesting something and then waits for
the recipient to react.
• Advantages:
• Ensures precise synchronization between communicating entities.
• Simplifies error handling as the sender knows when the message has
been successfully received or processed.
 Asynchronous Message Passing
Asynchronous message passing allows processes to operate
independently of each other in terms of timing.
Key features include:
• Decoupled Timing: The sender does not wait for an immediate response
from the receiver after sending a message. It continues its execution
without blocking.
• Event-Driven Model: Communication is often event-driven, where
processes respond to messages or events as they occur asynchronously.
Advantages:
• Enhances system responsiveness and throughput by allowing processes
to execute concurrently.
• Allows for interactions that are loosely connected, allowing processes to
process messages at their own speed.
 Unicast Messaging
• Unicast messaging is a one-to-one communication
where a message is sent from a single sender to a
specific receiver.
• The key characteristics include:
• Direct Communication: The message is targeted at a
single, specific node or endpoint.
• Efficiency for Point-to-Point: Since only one recipient
receives the message, resources are efficiently used for
direct, point-to-point communication.
Unicast Messaging
• Advantages:
• Optimized for targeted communication, as the message is only
sent to the intended recipient.
• Minimizes network load compared to group messaging, as it
doesn’t broadcast to unnecessary nodes.
• Disadvantages:
• Not scalable for group communications; sending multiple
unicast messages can strain the system in larger networks.
• managing multiple unicast connections in large-scale
applications can increase the complexity
Multicast Messaging
Multicast messaging enables one-to-many
communication, where a message is sent from one
sender to a specific group of receivers.

The key characteristics include:

• Group-Based Communication: Messages are delivered to a
subset of nodes that have joined the multicast group.
• Efficient for Groups: Saves bandwidth by sending the
message once to all nodes in the group instead of individually.
Broadcast Messaging
• Broadcast messaging involves sending a
message from one sender to all nodes within
the network.
• The key characteristics include:
• Wide Coverage: The message is sent to every node,
ensuring that all nodes in the network receive it.
• Network-Wide Reach: Suitable for announcements,
alerts, or updates intended for all nodes without
targeting specific ones.
Broadcast Messaging
• Advantages:
• Guarantees that every node in the network receives the
message, which is useful for critical notifications or status
updates.
• Simplifies dissemination of information when all nodes need to
be aware of an event or data change.
• Disadvantages:
• Consumes significant network resources since every node,
regardless of relevance, receives the message.
• Can lead to unnecessary processing at nodes that don’t need
the message, potentially causing inefficiency.
Distributed Models

1.Client-Server Model
1. Clear distinction between service providers (servers) and service users (clients)
2. Centralized control
3. Examples: Web servers, database servers
2.Peer-to-Peer Model
1. All processes have equal roles
2. Decentralized control
3. Examples: BitTorrent, blockchain networks
3.Hybrid Models (Multi-tier Architecture)
1. Combination of client-server and peer-to-peer
2. Examples: Modern distributed applications
 Client-Server Model
• It is a centralized approach in which the client
initiate requests for services and severs respond by
providing those services. It mainly works on the
request-response model where the client sends a
request to the server and the server processes it, and
responds to the client accordingly.
• It can be achieved by using TCP/IP, HTTP protocols
on the transport layer.

• This is mainly used in web services, cloud

computing, database management systems etc.
 Peer-to-Peer Model
It is a decentralised approach in which all the
distributed computing nodes, known as peers, are
all the same in terms of computing capabilities and
can both request as well as provide services to
other peers. It is a highly scalable model because
the peers can join and leave the system
dynamically, which makes it an ad-hoc form of
network.
•The resources are distributed and the peers need
to look out for the required resources as and when
required.

•The communication is directly done amongst the

peers without any intermediaries according to
some set rules and procedures defined in the P2P
networks.

•The best example of this type of computing is

BitTorrent.
 Hybrid Models (Multi-
tier Architecture)

•Tiered architectures are complementary to

layering. Whereas layering deals with the vertical
organization of services into layers of abstraction,
tiering is a technique to organize functionality of
a given layer and place this functionality into
appropriate servers and, as a secondary
consideration, on to physical nodes.
•This technique is most commonly associated
with the organization of applications and services
as in Figure 2.7 above, but it also applies to all
layers of a distributed systems architecture
Time and Global states
• Time is an important practical issue.
• But time is problematic in distributed systems.
• Each computer may have its own physical clock, but the
clocks typically deviate, and we cannot synchronize them
perfectly.
About Physical clocks
• Physical clocks measure the time in seconds.
• They include analogue/mechanical clocks based on pendulums or
similar mechanisms, and
• Digital clocks based e.g. on a vibrating quartz
• Quartz clocks are cheap, but they are not totally accurate.
• Due to manufacturing imperfections, some clocks run slightly
faster than others.
• Moreover, the oscillation frequency varies with the temperature.
• Typical quartz clocks are tuned to be quite stable around room
temperature, but significantly higher or lower temperatures slow
down the clock.
• The rate by which a clock runs fast or slow is called drift.
Quartz clocks
• Quartz crystal laser-
trimmed to
mechanically resonate
at a specific frequency
• Piezoelectric effect:
mechanical force ⇔
electric field
• Oscillator circuit
produces signal at
resonant frequency
• Count number of cycles
to measure elapsed
time
Quartz clock error: drift
• One clock runs slightly fast, another slightly slow
• Drift measured in parts per million (ppm)
• 1 ppm = 1 microsecond/second = 86 ms/day = 32
s/year
• Most computer clocks correct within ≈ 50 ppm
Atomic clocks
• When greater accuracy is required, atomic clocks are used.
• Based on quantum mechanical properties of certain atoms,
such as Caesium or rubidium.
• Caesium-133 has a resonance (“hyperfine transition”) at ≈ 9
GHz
• Tune an electronic oscillator to that resonant frequency
• 1 second = 9,192,631,770 periods of that signal
• Accuracy ≈ 1 in 10−14 (1 second in 3 million years)
• Price ≈ £20,000 (?) (can get cheaper rubidium clocks for ≈
£1,000)
Definitions
• Clock skew: difference between two clocks at a point
in time
• Clock drift: The rate by which a clock runs fast or slow
is called drift.
Synchronizing Physical Clocks
• External synchronization: refers to the process of
synchronizing the clocks of all the systems in the
distributed system with an external time source.
• External Time Source: This is a highly accurate time
reference that is independent of the distributed system
itself. Common examples include:
• Atomic Clocks: These are extremely precise clocks that use the
frequency of atomic oscillations to measure time. They are often
used as primary time standards.
• GPS (Global Positioning System): GPS satellites provide very
accurate time signals that can be used for synchronization.
• Radio Time Signals: Some radio stations broadcast time signals
that can be received and used for synchronization.
Techniques for External
Synchronization
• NTP (Network Time Protocol): NTP is a widely used
protocol for synchronizing clocks over a network. It can
be used for both internal and external synchronization.
In the case of external synchronization, NTP servers are
used to distribute time information from an external
time source.
• PTP (Precision Time Protocol): PTP is a more precise
protocol than NTP, often used in applications that
require very high accuracy. It can also be used for
external synchronization.
Synchronizing Physical Clocks
• Internal synchronization: refers to the process of keeping the clocks
of all the systems within the distributed system consistent with each
other. It doesn't necessarily mean the clocks are perfectly accurate
relative to an external, absolute time source, but rather that they agree
with each other.
• It ensures that if system A records an event before system B, all other
systems in the distributed system will also perceive that ordering.
• How it works (in principle): Internal synchronization algorithms
attempt to minimize the skew between clocks. Skew is the difference in
time between any two clocks in the system.
• These algorithms typically involve:
• Exchanging Timing Information: Systems periodically exchange messages
containing their current time.
• Clock Adjustment: Based on the exchanged information, systems adjust their
local clocks to bring them closer to the average or a designated reference clock
within the system.
Techniques for Internal
Synchronization
• Cristian's Algorithm: A classic algorithm for
synchronizing a client clock with a time server.
• Berkeley Algorithm: A decentralized algorithm where
a master node polls other nodes for their time and then
calculates an average to which all nodes synchronize.
• Network Time Protocol (NTP): While NTP can be
used for external synchronization, it also plays a crucial
role in internal synchronization by establishing a
hierarchy of time servers within a network.
How NTP Works
• Hierarchical Structure (Stratum Levels): NTP uses
a hierarchical system of time servers, organized into
"strata."
• Stratum 0 servers are the most accurate, typically
atomic clocks or GPS receivers.
• Stratum 1 servers synchronize directly with stratum 0
servers, stratum 2 servers with stratum 1, and so on.
• This hierarchy distributes time information efficiently
and prevents overloading the most accurate time
sources.
Image source: https://www.networkacademy.io/ccna/network-services/network-time-protocol-ntp
NTP Method
• Client-Server Model: Computers that need their clocks synchronized (clients)
connect to NTP servers. The client sends a request to the server, and the server
responds with its current time.
• Time Stamps and Round-Trip Delay: NTP uses precise timestamps to account
for network latency (the time it takes for messages to travel between client and
server). It measures the round-trip delay and uses this information to
compensate for the time it takes for the time information to arrive.
• Clock Adjustment: The client uses the received time information and the
calculated round-trip delay to adjust its local clock. It doesn't just jump to the
new time; it gradually slews its clock to avoid sudden changes that could disrupt
running applications.
• Multiple Servers: Clients typically connect to multiple NTP servers to improve
accuracy and reliability. By comparing time information from different servers,
the client can filter out any erroneous time readings and get a more accurate
overall time.
Estimating time with NTP

Round-trip network delay :

Estimated server time when client receives response:

Correcting clock skew
• Once the client has estimated the clock skew θ, it needs
to apply that correction to its clock.
• If |θ| < 125 ms, slew the clock:
slightly speed it up or slow it down by up to 500 ppm (brings
clocks in sync within ≈ 5 minutes)
• If 125 ms ≤ |θ| < 1,000 s, step the clock:
suddenly reset client clock to estimated server timestamp
• If |θ| ≥ 1,000 s, panic and do nothing
(leave the problem for a human operator to resolve)
Precision Time Protocol
• While NTP is widely used and effective for many
applications, PTP is designed for situations demanding
much greater accuracy, often down to the nanosecond
level. Think of it as the specialized timekeeping tool for
applications where even tiny discrepancies in time can
be critical.
• Master-Slave Hierarchy: PTP also uses a hierarchical
structure, but instead of strata, it uses a master-slave
relationship. A "grand master" clock is selected, and all
other clocks in the network synchronize to it. The grand
master is often a highly accurate clock, like an atomic
clock or a GPS-disciplined oscillator.
Method
• Hardware Timestamping: A key feature of PTP is its
use of hardware timestamping. This means that the
precise time a packet is sent or received is recorded
directly by the network interface card (NIC) hardware,
rather than by the operating system. This significantly
reduces the impact of software overhead and latency,
which are major sources of inaccuracy in NTP.
• Synchronization Messages: PTP uses specialized
messages, such as Sync and Delay_Req/Delay_Resp, to
exchange timing information between the master and
slave clocks.
• Two delay paths must be calculated, the Master
to Slave and the Slave to Master.
• Master to Slave difference = T2 - T1
• Slave to Master difference = T4 - T3
• One way delay = (Master to Slave difference +
Slave to Master difference) / 2
• The offset is used to correct the Slave clock:
• Offset = Master to Slave difference – One way delay
or
• Offset = ((T2 - T1) - (T4 - T3)) / 2
• Delay Measurement: PTP accurately measures the
path delay between the master and slave clocks. This is
crucial for compensating for the time it takes for
messages to travel across the network. It uses the
Delay_Req/Delay_Resp messages to determine this.
• Clock Adjustment: Slave clocks use the received
timing information and the calculated path delay to
adjust their local clocks with extremely high precision.
 PTP vs NTP
Feature PTP NTP
Accuracy Nanosecond level Millisecond level
Timestamping Hardware Software
Accurately measured Estimated and less
Network Delay
and compensated precise compensation
More complex to Simpler to configure
Complexity
configure and deploy and deploy
Generally more Generally less
Cost
expensive expensive
High-precision General-purpose time
Use Cases
applications synchronization
Thank You