UNIT 1

Introduction to Distributed Systems

A distributed system is a collection of independent computers that work together to appear as a
single system to the user. The computers communicate with each other over a network to share
resources, process data, and coordinate tasks. Examples of distributed systems include cloud
computing platforms, online banking systems, and e-commerce websites.

Examples of Distributed Systems

1. World Wide Web (WWW):
– A massive distributed system that allows users to access web pages and services
hosted on servers across the globe.
2. Online Banking Systems:
– Banks use distributed systems to handle customer transactions from different
branches or through ATMs and mobile apps.
3. E-commerce Platforms:
– Websites like Amazon handle millions of customers simultaneously, requiring a
distributed setup for inventory management, order processing, and payment
handling.
4. Cloud Computing Services:
– Services like AWS, Google Cloud, or Azure distribute tasks such as data storage,
computation, and machine learning across multiple servers.
5. Social Media Platforms:
– Platforms like Facebook or Instagram use distributed systems to manage user
profiles, photos, and posts across multiple servers.

Resource Sharing and the Web

Resource sharing is a primary goal of distributed systems. Examples include:
 • Hardware Resources: Printers, servers, or storage devices shared over a network.
• Data Sharing: Allowing multiple users to access shared files, databases, or services.
• Web Applications: The web enables resource sharing by providing access to online
resources like documents, media, and applications, e.g., Google Drive or YouTube.

Challenges in Distributed Systems

1. Coordination:
– Ensuring that all independent computers in the system work together
seamlessly.
2. Fault Tolerance:
– Managing hardware or software failures without disrupting the system.
3. Latency:
– Minimizing delays in communication between components of the system.
4. Security:
– Protecting sensitive data and ensuring secure communication between
computers.
5. Scalability:
– Ensuring the system can handle increasing users and tasks without degrading
performance.

Architectural Models
The architecture of a distributed system defines how its components interact. Common models
include:

1. Client-Server Model:
– The server provides services (e.g., a database server), and the client requests
them (e.g., a web browser).
2. Peer-to-Peer (P2P) Model:
– All nodes (computers) have equal roles and share resources directly without a
centralized server (e.g., torrent systems).
3. Layered Architecture:
– Organizes the system into layers (e.g., application layer, middleware layer,
network layer) for better manageability.

Fundamental Models
Fundamental models define the basic properties and behaviors of a distributed system:

1. Interaction Model:
– Describes how components communicate. For example, components may send
messages over a network.
2. Failure Model:
 – Describes possible failures like crash failures, omission failures (missing data), or
timing failures (delays).
3. Security Model:
– Defines security measures like encryption, authentication, and secure data access
to protect the system from attacks.

Limitations of Distributed Systems

1. Absence of Global Clock:
– Unlike centralized systems, there is no single clock in a distributed system to
synchronize operations. Instead, they rely on algorithms like logical clocks or
vector clocks to maintain a sequence of events.
2. Shared Memory:
– In distributed systems, computers have their own local memory, and there’s no
global shared memory. Communication between systems is done through
message passing or data replication.
3. Network Dependency:
– Distributed systems heavily rely on networks, so a failure in connectivity can
disrupt operations.
4. Complexity:
– Designing, managing, and debugging distributed systems is more challenging
due to the interactions between multiple independent systems.

By understanding these concepts, one can grasp the essence of distributed systems and their
real-world applications while appreciating the challenges involved in their development and
management.

Logical Clocks
Logical clocks are mechanisms to order events in a distributed system where there is no global
clock. They provide a way to determine the sequence of events (happened-before relationship)
in a distributed environment. Logical clocks are essential for achieving consistency in distributed
systems.

Lamport’s Logical Clocks

Lamport's Logical Clocks are based on the happens-before (→) relationship:

• Happens-before (→): If event ( A ) happens before ( B ), then ( A → B ).

– ( A → B ) if:
i. ( A ) and ( B ) occur on the same process and ( A ) happens before ( B ).
ii. A process sends a message representing ( A ), and another process
receives the message as ( B ).

Lamport introduced a logical clock (L) for each process to track events:
 1. Rule 1 (Internal Events): For every event ( e ) within a process ( P ), increment ( L(P) ) by 1.
– ( L(P) = L(P) + 1 ).
2. Rule 2 (Send Event): When ( P ) sends a message, it includes its current timestamp.
3. Rule 3 (Receive Event): When ( Q ) receives a message, update ( L(Q) ) to:
– ( L(Q) = \max(L(Q), L(P)) + 1 ).

This ensures that timestamps respect the happens-before relationship.

Vector Clocks
Vector clocks enhance Lamport's Logical Clocks by tracking causal relationships between
events. Each process maintains a vector of timestamps, one for each process in the system.

1. Structure:
– ( V[i] ) represents the logical clock of process ( i ).
2. Rules:
– Internal Event: Increment the local clock of the process:
• ( V[i] = V[i] + 1 ).
– Send Event: Before sending a message, increment the local clock and attach the
vector clock.
– Receive Event: When a process ( Q ) receives a message, update its vector clock:
• ( V[Q][j] = \max(V[Q][j], V[P][j]) ) for all ( j ), and increment ( V[Q][Q] ).

Vector clocks provide more precision than Lamport clocks by identifying causally related and
concurrent events.

Concepts in Message Passing Systems

1. Causal Order:
– Ensures that messages are delivered in an order respecting causal relationships
(if message ( M1 → M2 ), ( M1 ) must be delivered before ( M2 )).
2. Total Order:
– Ensures that all processes deliver messages in the same sequence, even if the
messages are independent.
3. Total Causal Order:
– Combines causal order and total order. It ensures that all processes deliver
messages in the same sequence, while maintaining causal relationships.
4. Message Ordering:
– Refers to how messages are delivered in a distributed system. Types include:
• FIFO (First-In-First-Out) Order: Messages sent from one process to
another are received in the order they were sent.
• Causal Order: Messages respect causal relationships.
• Global Order: All processes see the same order of messages.
Causal Ordering of Messages
This ensures that messages are delivered in the same causal order as they were sent. For
instance:

• If process ( P ) sends message ( M1 ) and then ( M2 ), and both are sent to ( Q ), ( Q ) must
deliver ( M1 ) before ( M2 ).

Vector clocks help achieve causal ordering by timestamping messages with their causal
dependencies.

Global State
The global state of a distributed system is the combined state of all processes and the
communication channels at a particular point in time. Capturing the global state is challenging
due to the lack of a global clock.

• Chandy-Lamport Algorithm: A famous algorithm for capturing a consistent global state

by using "marker" messages to divide events into "before marker" and "after marker"
phases.

Termination Detection
Termination detection determines when a distributed computation has completed. This is tricky
in distributed systems because processes operate independently and asynchronously.

1. Active and Passive States:

– A process is active if it is working on a task or sending messages.
– A process is passive if it is idle and waiting for incoming messages.
2. Distributed Termination Detection:
– Termination is reached when all processes are passive, and no messages are in
transit.
– Algorithms like Dijkstra-Scholten Algorithm are used to detect termination by
tracing dependencies between active processes.

Summary
• Logical Clocks (Lamport & Vector): Provide ways to order events in a distributed system.
• Message Passing Systems: Handle causal and total ordering of messages.
• Global State: Represents the system's snapshot at a given moment.
• Termination Detection: Ensures the system knows when all work is done.

Understanding these concepts is crucial for designing and analyzing distributed systems,
especially in applications like cloud computing and real-time systems.
UNIT 2

Distributed Mutual Exclusion

Distributed Mutual Exclusion is a mechanism that ensures only one process in a distributed
system can access a shared resource (critical section) at a time. This is crucial to prevent conflicts
and maintain consistency in distributed systems.

Classification of Distributed Mutual Exclusion

Distributed mutual exclusion algorithms can be classified based on how they coordinate access
to the critical section:

1. Token-Based Algorithms:
– A unique token is used to grant access to the critical section.
– Only the process holding the token can enter the critical section.
– Examples: Token ring algorithm.
2. Non-Token-Based Algorithms:
– Processes use a request and reply mechanism to ensure mutual exclusion.
– Messages are exchanged between processes to decide who gets access.
– Examples: Ricart-Agrawala algorithm.
3. Hybrid Algorithms:
– Combine features of token-based and non-token-based approaches for better
performance or fault tolerance.

Requirements of Mutual Exclusion (Theorem)

For any mutual exclusion algorithm, the following requirements must be met:

1. Safety:
– At most one process can be in the critical section at any time.
2. Liveness:
 – A process requesting access to the critical section will eventually be granted
access (no starvation).
3. Fairness:
– Requests are handled in the order they are made (FIFO order).
4. Fault Tolerance:
– The algorithm should handle failures (e.g., of processes or communication links)
gracefully.

Token-Based Algorithms
In token-based algorithms, a unique token circulates in the system, granting access to the
critical section.

Token Ring Algorithm:

1. Structure:
– Processes are organized in a logical ring structure.
– The token passes from one process to another in the ring.
2. Operation:
– A process can enter the critical section only if it holds the token.
– Once done, it passes the token to the next process in the ring.
3. Advantages:
– Low message overhead: A process only needs to forward the token.
– No starvation: The token circulates, ensuring every process gets a turn.
4. Disadvantages:
– Token loss: If the token is lost, it needs to be regenerated, adding complexity.

Non-Token-Based Algorithms
In non-token-based algorithms, processes communicate using messages to request and release
access to the critical section.

Ricart-Agrawala Algorithm:
1. Structure:
– Each process keeps a logical clock (Lamport or vector clock).
2. Operation:
– When a process wants to enter the critical section:
• It sends a request message (with its timestamp) to all other processes.
– Other processes:
• Reply immediately if they are not in the critical section or not requesting
it.
• Delay their reply if they are in the critical section or have a request with a
lower timestamp.
– The process enters the critical section once it receives replies from all other
processes.
 3. Advantages:
– No token is needed, so there's no risk of token loss.
– Fair ordering is ensured using timestamps.
4. Disadvantages:
– High message overhead: ( 2(n-1) ) messages for every critical section request,
where ( n ) is the number of processes.

Performance Metrics
To evaluate and compare distributed mutual exclusion algorithms, the following performance
metrics are considered:

1. Message Complexity:
– The number of messages exchanged per critical section access.
– Token-based: Low (1 message to pass the token).
– Non-token-based: High (at least ( 2(n-1) )).
2. Synchronization Delay:
– The time taken to grant access to the critical section after a request.
– Token-based: Low, as the token is directly passed.
– Non-token-based: Higher, as it requires replies from all processes.
3. Fault Tolerance:
– Ability to handle failures.
– Token-based: Token loss requires regeneration, which is complex.
– Non-token-based: More resilient, as no token is used.
4. Fairness:
– Ensures processes are granted access in the order of their requests.
– Both token-based and non-token-based algorithms can maintain fairness with
proper design.
5. Throughput:
– The number of critical section accesses completed in a given time.
– Token-based algorithms generally have higher throughput due to low overhead.

Comparison Between Token-Based and Non-Token-Based

Algorithms
Feature Token-Based Non-Token-Based
Message Overhead Low (1 message to pass the token) High (( 2(n-1) ) messages)
Fairness Ensured through token circulation Ensured via timestamps
Fault Tolerance Token loss requires recovery Resilient to token loss
Synchronization Delay Low High
Complexity Medium (token regeneration High (many messages
needed) exchanged)
Summary
Distributed mutual exclusion is vital for ensuring that multiple processes in a distributed system
access shared resources safely and fairly. Token-based algorithms are simple and efficient but
vulnerable to token loss, while non-token-based algorithms are robust but incur high message
overhead. The choice of algorithm depends on the system's requirements, such as message
efficiency, fault tolerance, and fairness.

Distributed Deadlock Detection

In distributed systems, deadlock occurs when processes are stuck waiting for resources or
messages held by other processes indefinitely. Deadlock detection in such systems is more
complex due to the lack of a centralized control and the asynchronous nature of communication.

System Model
The system can be represented as a Resource Allocation Graph (RAG):

1. Nodes:
– Processes (P1, P2, ...).
– Resources (R1, R2, ...).
2. Edges:
– Request edges: ( P1 \rightarrow R1 ) (process requests resource).
– Assignment edges: ( R1 \rightarrow P1 ) (resource assigned to process).

In distributed systems:

• Processes and resources are distributed across nodes.

• Communication occurs through message passing.

Resource Deadlocks vs. Communication Deadlocks

1. Resource Deadlocks:
– Processes are waiting for resources held by others.
– Example: Process P1 is waiting for resource R1, held by P2, while P2 waits for R2,
held by P1.
2. Communication Deadlocks:
– Processes are waiting for messages or responses from other processes, creating
a cyclic dependency.
– Example: Process P1 waits for a message from P2, while P2 waits for a message
from P1.

Both types can be modeled using a Wait-For Graph (WFG), where:

• Nodes represent processes.

• Edges represent dependencies (e.g., P1 → P2 indicates P1 is waiting for P2).
Deadlock Prevention
Deadlock prevention ensures that the system never enters a deadlock state by enforcing
constraints:

1. Avoid circular wait:

– Impose a strict ordering on resource acquisition.
2. Hold-and-wait:
– Disallow processes from holding resources while waiting for others.
3. Preemption:
– Allow preemption of resources held by other processes.
4. Avoid mutual exclusion:
– Avoid exclusive access to resources (if possible).

While effective, prevention can be too restrictive and lead to low resource utilization.

Deadlock Avoidance
Deadlock avoidance ensures the system remains in a safe state where deadlock cannot occur:

1. Banker’s Algorithm (adapted for distributed systems):

– The system evaluates resource requests to ensure granting them will not lead to
a deadlock.
– Works well for resource deadlocks but is computationally intensive in distributed
systems.

Deadlock Detection & Resolution

1. Deadlock Detection:
– Detect cycles in the Wait-For Graph (WFG).
– Techniques include:
• Centralized detection.
• Distributed detection.
• Hierarchical detection.
2. Resolution:
– Terminate or roll back one or more processes in the cycle.
– Preempt resources held by one or more processes.

Centralized Deadlock Detection

In centralized detection:

1. A single coordinator collects the WFG from all processes.

 2. It periodically checks for cycles in the graph.
3. If a cycle is found, it resolves the deadlock by terminating or rolling back one or more
processes.

Advantages:

• Simplicity.
• Easier to implement and debug.

Disadvantages:

• Single point of failure.

• Communication overhead as all nodes send information to the central coordinator.

Distributed Deadlock Detection

In distributed detection:

1. Each node maintains its local WFG.

2. Nodes communicate and share WFG information to detect global cycles.
3. Algorithms are used to identify deadlocks collectively.

Advantages:

• No single point of failure.

• Scalable for large systems.

Disadvantages:

• Complex algorithms.
• Communication overhead due to message exchanges.

Path Pushing Algorithms

Path pushing algorithms detect deadlocks by sharing dependency information (paths) between
processes:

1. Procedure:
– Each node sends its dependency paths to other nodes.
– When a node receives a path, it extends it with its dependencies and propagates
it further.
– If a path returns to the originating node, a cycle (deadlock) is detected.
2. Example:
– Process P1 → P2 → P3 → P1 indicates a deadlock.
3. Advantages:
– Simple conceptually.
– Effective for detecting cycles.
4. Disadvantages:
 – High communication overhead due to long paths.

Edge Chasing Algorithms

Edge chasing algorithms use special probe messages to detect cycles:

1. Procedure:
– When a process ( P1 ) is waiting on ( P2 ), it sends a probe message to ( P2 ).
– The message includes:
• Information about the initiator (e.g., ( P1 )).
• The dependency chain so far.
– If the probe returns to the initiator ( P1 ), a deadlock is detected.
2. Advantages:
– Low communication overhead compared to path pushing.
– More efficient in detecting cycles.
3. Disadvantages:
– Requires careful handling of probe messages.
– More complex to implement.

Comparison: Path Pushing vs. Edge Chasing

Feature Path Pushing Edge Chasing
Communication High (paths shared) Lower (probes only)
Complexity Moderate Higher (probe handling)
Efficiency Effective for large systems Efficient for smaller cycles
Scalability Can handle large systems Better for localized deadlocks

Summary
Distributed deadlock detection is a challenging but essential aspect of managing resources in
distributed systems. It involves identifying and resolving deadlocks caused by resource or
communication dependencies. Centralized methods are simpler but less robust, while
distributed methods, such as path pushing and edge chasing, offer scalability and resilience at
the cost of complexity. Choosing the right approach depends on the system's size,
requirements, and fault tolerance needs.
UNIT 3

Agreement Protocols in Distributed Systems

Agreement protocols are essential for ensuring that distributed systems operate reliably and
consistently, even in the presence of faults. They define the rules and mechanisms through
which nodes in a distributed system reach a common decision.

Introduction
Distributed systems consist of multiple nodes that must often agree on a shared value or
decision, such as committing a transaction, synchronizing clocks, or electing a leader. However,
achieving agreement in such systems is challenging due to:

1. Network failures: Messages might be lost, delayed, or corrupted.

2. Node failures: Nodes may crash or behave maliciously (Byzantine faults).
System Models
1. Synchronous System:
– Known message delivery times.
– Nodes operate at the same clock rate.
– Easier to design agreement protocols but less realistic.
2. Asynchronous System:
– No guarantee on message delivery time or execution speed.
– More realistic but harder to achieve agreement.
3. Fault Tolerance:
– Nodes may exhibit Crash Faults (stop functioning) or Byzantine Faults
(arbitrary/malicious behavior).
– Protocols aim to tolerate a certain number of faulty nodes.

Classification of Agreement Problems

1. Consensus Problem:
– All non-faulty nodes must agree on a single value.
– Example: Committing a transaction in a database.
2. Byzantine Agreement Problem:
– Nodes must agree on a value, even if some behave maliciously.
– Example: Secure communication in a military setting.
3. Interactive Consistency Problem:
– All non-faulty nodes must agree on a set of values proposed by all nodes.
– Ensures consistency across all nodes.
4. Atomic Commitment Problem:
– Ensures that a transaction is either fully committed or fully aborted across all
nodes.
– Example: Distributed database systems.

Byzantine Agreement Problem

Definition:
The Byzantine Agreement problem addresses the challenge of achieving agreement in the
presence of Byzantine faults, where some nodes behave arbitrarily or maliciously.

Requirements:
1. Agreement: All non-faulty nodes agree on the same value.
2. Validity: If all non-faulty nodes propose the same value, they must agree on that value.
3. Termination: All nodes must reach a decision in a finite time.

Challenges:
• Difficult to distinguish between a slow node and a faulty one.
• Malicious nodes can send conflicting information to different nodes.
Solution:
The Byzantine Agreement problem can be solved if:

1. The total number of nodes ( N ) satisfies ( N \geq 3f + 1 ), where ( f ) is the number of

Byzantine nodes.
2. Protocols use techniques such as message exchange, voting, and verification.

Example Protocol: Byzantine Generals Problem

• A group of generals must agree on a common battle plan.
• Some generals may be traitors, sending conflicting messages.
• The solution involves multiple rounds of message exchanges and voting to ensure all
loyal generals agree on a plan.

Consensus Problem
Definition:
Consensus involves all nodes in a distributed system agreeing on a single value, despite failures.

Requirements:
1. Agreement: All non-faulty nodes agree on the same value.
2. Validity: If a value is proposed by a non-faulty node, the agreed-upon value must be one
of the proposed values.
3. Termination: All nodes decide within a finite time.

Solutions:
• Paxos Algorithm: Ensures consensus in an asynchronous system with crash faults.
• Raft Algorithm: Simplified version of Paxos, widely used in distributed systems.

Interactive Consistency Problem

Definition:
Interactive consistency ensures that all non-faulty nodes agree on a consistent set of values
proposed by all nodes, even in the presence of faults.

Requirements:
• Each node must agree on the same value for every other node, ensuring consistency.

Example:
In a flight control system, interactive consistency ensures all controllers have the same state of
the aircraft's position.
Solution to Byzantine Agreement Problem
Key Techniques:
1. Message Authentication:
– Use cryptographic methods to detect tampering or malicious messages.
2. Redundancy:
– Exchange messages multiple times to verify consistency.
3. Majority Voting:
– Rely on the majority to determine the correct value.

Algorithm Example:
Oral Messages Algorithm:

1. Each node sends its value to all other nodes.

2. Nodes exchange received values and apply majority voting to filter out inconsistent
values.

Applications of Agreement Protocols

1. Distributed Databases:
– Ensures consistency in transactions across multiple nodes.
2. Blockchain:
– Consensus protocols like Proof of Work or Proof of Stake rely on agreement.
3. Fault-Tolerant Systems:
– Ensures system reliability in the presence of faults.
4. Leader Election:
– Nodes agree on a single leader in distributed systems.
5. Secure Communication:
– Byzantine agreement ensures data integrity and reliability.

Atomic Commit in Distributed Database Systems

Definition:
Atomic Commit ensures that a transaction is either:

1. Committed: Successfully applied to all nodes.

2. Aborted: Discarded completely from all nodes.

Techniques:
1. Two-Phase Commit (2PC):
– Phase 1: Coordinator asks all nodes if they can commit.
– Phase 2: Based on responses, the coordinator sends a commit or abort decision.
– Drawback: Blocking nature if the coordinator crashes.
 2. Three-Phase Commit (3PC):
– Adds an intermediate phase to avoid blocking.
– Ensures safety in case of coordinator failure.

Summary
Agreement protocols play a critical role in distributed systems by enabling nodes to reach
consistent decisions despite failures or malicious behavior. These protocols ensure reliability
and consistency across diverse applications, such as distributed databases, fault-tolerant
systems, and blockchain networks. Techniques like Byzantine Agreement, Consensus, and
Atomic Commit provide robust solutions tailored to specific system models and requirements.

Distributed Resource Management: Issues in Distributed File

Systems
In a distributed system, resources like files, storage, and computational power are spread
across different machines or nodes in the network. Distributed Resource Management refers to
managing these resources efficiently to ensure that they are accessible, consistent, and used
optimally by all the components in the distributed system.

A Distributed File System (DFS) is a system that manages and provides access to files stored on
multiple servers or machines, which can be located in different geographical locations. The goal
of a DFS is to provide users with a unified interface to access these files, despite the underlying
distribution of data.

1. Issues in Distributed File Systems

Distributed file systems face several challenges that must be addressed to ensure that files can
be accessed efficiently, securely, and reliably across a distributed network.

a. Transparency:
Transparency refers to hiding the complexity of the distributed system from the user. In the
context of distributed file systems, there are several types of transparency:

• Access Transparency: Users should not need to know whether they are accessing a file
stored locally or on a remote machine.
• Location Transparency: Users should not need to know where the files are physically
stored. The system should handle the details of file location automatically.
• Replication Transparency: Users should not need to know whether a file has multiple
replicas, or if some replicas are unavailable.
• Concurrency Transparency: The system should manage concurrent access to files so
that users can read/write data without conflicts or data corruption.
b. Fault Tolerance and Reliability:
In a distributed file system, nodes and communication links can fail. A robust DFS must handle
failures without compromising data integrity and availability. Issues related to fault tolerance
include:

• Crash Recovery: Ensuring that the system can recover from crashes without losing data.
• Replication: Files should be replicated across multiple servers to ensure that even if one
server fails, the data is still available.
• Consistency: The system must maintain consistency across replicas, ensuring that
changes made to one replica are reflected in others.

c. Consistency and Concurrency Control:

When multiple users or processes access the same file simultaneously, the DFS must ensure
that data is not corrupted or inconsistent. There are several issues related to concurrency:

• File Locking: To prevent conflicting operations, the system must provide mechanisms
for locking files or parts of files.
• Write Propagation: Changes made to a file must be propagated to all replicas in a
consistent manner.
• Consistency Models: The DFS must decide whether to use strong consistency (where all
nodes see the same data at the same time) or eventual consistency (where updates are
propagated asynchronously, leading to temporary discrepancies).

d. Performance:
Performance is a critical issue in distributed file systems. The system should be able to handle
large-scale data access with low latency and high throughput. Key factors that affect
performance include:

• Data Locality: Storing files near the users or processes that access them most
frequently.
• Caching: Using caches to store frequently accessed files or parts of files locally, reducing
the need for repeated access over the network.
• Load Balancing: Distributing the file access load evenly across the system’s resources to
avoid bottlenecks.

e. Security and Access Control:

In a distributed system, files are accessed by different users or machines, which can pose
security risks. A DFS needs to ensure that:

• Authentication: Only authorized users or processes can access files.

• Authorization: Different levels of access (read, write, execute) should be enforced
depending on user roles.
• Encryption: Files may need to be encrypted both at rest (on disk) and in transit (over the
network).
f. Scalability:
As the system grows, a DFS must handle an increasing amount of data and requests without
significant degradation in performance. Scalability issues include:

• Data Distribution: The system should be able to scale horizontally, adding more nodes
and distributing the data evenly across them.
• Directory Management: The system must efficiently manage metadata (file locations,
access permissions, etc.) even as the number of files and users grows.

2. Mechanisms for Building Distributed File Systems

There are several techniques and mechanisms used to build and implement a Distributed File
System. These mechanisms address issues related to transparency, fault tolerance,
performance, and scalability.

a. File Naming and Location Management:

File naming and location management ensure that users can access files without knowing their
physical location. The mechanisms for this include:

• Global Naming Space: A common naming scheme used across all machines in the
distributed system, which ensures that file names are unique and globally recognized.
• Directory Services: The system must maintain a global directory that maps file names to
their physical locations (usually a directory server or metadata server).

b. Replication and Fault Tolerance:

Replication and fault tolerance mechanisms ensure that data remains available even in the event
of a failure. Common approaches include:

• Data Replication: Storing multiple copies of data on different nodes so that if one node
fails, another can take over.
• Primary-Backup Replication: One replica is the primary copy, and other replicas are
backups. If the primary fails, a backup is promoted.
• Quorum-based Systems: A minimum number of replicas must agree on a read or write
operation to ensure consistency.
• Erasure Coding: Data is split into pieces and encoded with redundancy. This reduces the
storage overhead compared to full replication.

c. Caching:
Caching improves performance by storing frequently accessed data locally. The system should
have mechanisms to manage cached data, such as:

• Write-through Cache: Changes to the cache are immediately written to the underlying
storage.
• Lazy Write Cache: Changes are written to the storage at a later time, which improves
performance but may introduce consistency issues.
 • Cache Invalidation: When a file is updated, all cached copies must be invalidated or
updated to maintain consistency.

d. Distributed Locking and Concurrency Control:

Concurrency control ensures that multiple clients can access the same file simultaneously
without causing data corruption. This is typically done through:

• Distributed Locking: A mechanism to ensure that only one client or process can access a
file or portion of a file at a time. Locks can be applied at various granularity levels (e.g.,
file-level, block-level).
• Leases: A lease is a time-limited lock granted to a client for exclusive access to a file.
After the lease expires, the lock is released, allowing other clients to access the file.
• Versioning: Each update to a file can be stored as a new version, allowing clients to work
on different versions and later merge them.

e. Consistency Models:
The DFS must decide which consistency model it will implement. Some common models
include:

• Strict Consistency: All replicas are updated instantly and seen consistently by all clients.
• Sequential Consistency: Operations on the file system appear to happen in some
sequential order, but not necessarily instantaneously across replicas.
• Eventual Consistency: Updates to the system are propagated asynchronously, meaning
that at any given point, different replicas might be inconsistent, but they will eventually
converge to the same value.

f. Fault Detection and Recovery:

Fault detection and recovery mechanisms ensure the system can handle node failures and
continue operating. Key mechanisms include:

• Heartbeat Messages: Regular messages are exchanged between nodes to detect

failures.
• Logging: Operations on files are logged so that if a failure occurs, the system can recover
to a consistent state.
• Checkpointing: Periodically saving the state of the system so that recovery can occur
from a known consistent state.

Summary
• Distributed File Systems (DFS) are essential for providing access to data stored across
multiple machines in a distributed environment.
• Key issues in DFS include ensuring transparency, fault tolerance, consistency,
concurrency control, and performance.
• Mechanisms for building DFS include file naming and location management, data
replication, caching, distributed locking, concurrency control, and consistency models.
 • Effective DFS design focuses on achieving scalability, availability, and resilience to
failures, ensuring that users can access files reliably even in large-scale, distributed
systems.

By addressing these issues and employing these mechanisms, a distributed file system can
provide a robust and efficient infrastructure for managing and sharing files across a distributed
network.

Distributed Shared Memory (DSM)

Distributed Shared Memory (DSM) is a concept in distributed systems where a common
memory space is virtually shared among multiple nodes. It allows processes on different nodes
to communicate and share data as if they are accessing a single shared memory, simplifying
programming by abstracting the complexities of explicit message passing.

Design Issues in Distributed Shared Memory

Designing DSM involves addressing several challenges to ensure efficiency, consistency, and
fault tolerance. The main design issues are:

1. Structure of Shared Memory

• Memory Model: Deciding whether the shared memory is flat (global) or segmented
(divided into pages or objects).
• Granularity: Determining the size of the memory blocks (e.g., byte-level, page-level, or
object-level sharing).

2. Data Consistency
• Consistency Model: Ensuring that all nodes have a consistent view of the shared
memory.
– Strict Consistency: All memory operations appear to execute instantaneously at a
single point in time.
– Sequential Consistency: Operations appear in the same order across all nodes
but may not be instantaneous.
– Causal Consistency: Operations maintain cause-and-effect relationships.
 – Eventual Consistency: Changes are eventually propagated to all nodes but not
immediately.

3. Synchronization
• Mechanisms like locks, semaphores, or barriers are required to synchronize access to the
shared memory.
• Minimizing contention when multiple processes try to access the same memory
simultaneously is crucial.

4. Memory Coherence
• Ensuring that if multiple copies of the same data exist in different nodes, they are
updated consistently when changes occur.

5. Fault Tolerance
• Handling node or network failures while maintaining data integrity and availability.

6. Communication Overheads
• DSM requires communication between nodes to synchronize and update data.
Minimizing the frequency and size of these communications is critical for performance.

7. Scalability
• DSM should work efficiently as the number of nodes increases.

8. Transparency
• DSM should provide:
– Access Transparency: Abstract details of data location.
– Replication Transparency: Abstract how data copies are managed.
– Concurrency Transparency: Abstract the complexity of simultaneous access.

Algorithms for Implementing Distributed Shared Memory

Various algorithms have been proposed to implement DSM, each catering to specific
requirements like consistency and performance. The two primary approaches are centralized
and distributed algorithms.

1. Centralized Algorithm
• A single node (or server) is responsible for managing the shared memory.
• Advantages:
– Simple design.
– Easier to maintain consistency.
• Disadvantages:
– Single point of failure.
– Scalability issues due to bottlenecks.

2. Distributed Algorithm
• Memory management responsibilities are distributed across all nodes.
 • Advantages:
– Better scalability.
– No single point of failure.
• Disadvantages:
– Increased complexity in maintaining consistency and coherence.

Implementation Algorithm: Page-Based DSM

A common implementation of DSM uses a page-based approach, where memory is divided into
fixed-size pages.

Steps of the Algorithm:

1. Initialization:
– Divide the virtual memory into pages.
– Assign ownership of each page to a specific node initially.
2. Access Detection:
– When a process accesses a page, it triggers a page fault if the page is not present
locally.
3. Page Request:
– The local node sends a request to the owner of the page.
4. Page Transfer:
– The owner node transfers the required page to the requesting node.
– Pages can be sent as a copy (replicated) or transferred entirely (migrated).
5. Memory Update:
– If the page is replicated:
• Updates are propagated to all replicas based on the consistency model.
– If the page is migrated:
• Ownership is transferred to the requesting node.
6. Synchronization:
– Locks or barriers are used to synchronize access to shared pages to prevent race
conditions.

Example Algorithm: IVY

The IVY algorithm is an early implementation of page-based DSM:

• Pages are replicated on multiple nodes.

• Writes are propagated immediately to ensure consistency (write-invalidate or write-
update policies).

Example: Object-Based DSM

Instead of pages, memory is shared at the object level.

• Each object has a manager node that tracks its location and ownership.
 • Nodes requesting access to the object communicate with the manager to retrieve or
update it.

Advantages of DSM
1. Simplifies programming by abstracting inter-process communication.
2. Provides a unified view of memory across distributed nodes.
3. Reduces development time for distributed applications.

Disadvantages of DSM
1. Overhead in maintaining consistency and coherence.
2. Potential for increased latency due to network communication.
3. Scalability issues if not designed efficiently.

Conclusion
DSM is a powerful abstraction for distributed systems, enabling nodes to share memory
seamlessly. However, its design involves addressing challenges related to consistency,
synchronization, fault tolerance, and performance. Algorithms like page-based or object-based
DSM play a key role in implementing these systems efficiently.

Failure Recovery in Distributed Systems

Failure recovery in distributed systems involves restoring the system to a correct state after
failures, ensuring minimal data loss and system downtime. Failures can arise from hardware
crashes, software bugs, network issues, or power outages. Recovery techniques are designed to
maintain system integrity, consistency, and availability.

Key Concepts in Recovery

1. Backward Recovery
• Definition: The system is restored to a previously saved consistent state (checkpoint) and
then re-executes operations from that point onward.
• Use Case: Commonly used when the error’s effects are difficult to isolate.
• Steps:
a. Identify a previous consistent checkpoint.
b. Roll back the system state to that checkpoint.
c. Re-execute operations, ensuring the error doesn’t recur.

2. Forward Recovery
• Definition: The system moves forward by correcting the errors without rolling back to a
previous state.
• Use Case: Used when the error can be isolated and corrected directly.
• Steps:
 a. Analyze and identify the impact of the failure.
b. Apply corrective actions to bring the system to a consistent state.
c. Continue operations without significant downtime.

Recovery in Concurrent Systems

Concurrent systems involve multiple processes or threads running simultaneously, which
complicates recovery due to interdependencies.

Challenges:
1. Synchronization Issues: Concurrent processes often share resources, leading to
difficulties in maintaining a consistent state during recovery.
2. Order of Execution: Ensuring that operations maintain their intended order after
recovery.
3. Deadlocks: Recovery might introduce deadlocks that need resolution.

Solutions:
• Consistent Checkpoints: Ensure all processes save their states at a globally consistent
point.
• Message Logging: Record communication between processes to replay messages during
recovery.

Obtaining Consistent Checkpoints

A consistent checkpoint ensures that the system can resume execution without violating
correctness after a failure.

Challenges:
1. Process Dependencies: States of interdependent processes may become inconsistent.
2. Non-Deterministic Events: Random events or failures may disrupt checkpointing.

Techniques:
1. Coordinated Checkpointing:
– All processes coordinate to save their states simultaneously.
– Ensures a global consistent checkpoint.
– Disadvantage: May cause delays as all processes must synchronize.
2. Uncoordinated Checkpointing:
– Each process independently saves its state.
– Relies on message logging to resolve inconsistencies.
– Disadvantage: May lead to the domino effect, requiring multiple rollbacks.
3. Incremental Checkpointing:
– Only changes since the last checkpoint are saved.
– Reduces overhead but requires additional mechanisms to manage partial state
updates.
 4. Consistent Snapshot Algorithm (e.g., Chandy-Lamport Algorithm):
– Captures the global state of the system.
– Achieved by marking the state of processes and the messages in transit.

Recovery in Distributed Database Systems

Distributed databases are particularly sensitive to failures due to the need to maintain ACID
properties (Atomicity, Consistency, Isolation, Durability) across multiple nodes.

Challenges:
1. Network Partitions: Nodes might lose communication temporarily.
2. Transaction Dependencies: Transactions running across multiple nodes might become
inconsistent.
3. Concurrency Control: Coordinating recovery among multiple concurrent transactions.

Techniques:
1. Logging:
– Maintain logs (undo logs, redo logs) of all database operations.
– Use these logs to roll back or reapply operations during recovery.
2. Checkpointing:
– Save consistent snapshots of the database at regular intervals.
– Combine checkpointing with logs for faster recovery.
3. Two-Phase Commit (2PC):
– Ensures atomicity across distributed transactions.
– Phases:
i. Prepare Phase: All nodes agree to commit or abort.
ii. Commit Phase: Commit is executed only after all nodes confirm
readiness.
– Disadvantage: Can block progress during failures.
4. Three-Phase Commit (3PC):
– Enhances 2PC by adding a third phase to reduce blocking.
– Ensures system remains consistent even in case of node failures.
5. Replication:
– Maintain copies of data on multiple nodes.
– Use replication protocols (e.g., Paxos, Raft) to ensure consistency during
recovery.

Conclusion
Failure recovery in distributed systems is essential to maintain reliability and consistency.
Techniques like backward and forward recovery, checkpointing, and specialized database
recovery mechanisms address different challenges posed by distributed environments. Proper
recovery strategies depend on the system’s design goals, performance requirements, and the
types of failures anticipated.
Fault Tolerance in Distributed Systems
Fault tolerance refers to the ability of a distributed system to continue functioning correctly in
the presence of hardware, software, or network failures. It ensures system reliability, availability,
and consistency despite failures.

Issues in Fault Tolerance

1. Failure Models:
– Different types of failures need distinct handling strategies, such as crash
failures, omission failures, timing failures, and Byzantine failures (arbitrary or
malicious behavior).
2. Redundancy:
– Fault tolerance often relies on redundancy (replicating components, data, or
operations), which increases cost and complexity.
3. Consistency:
– Ensuring data consistency across replicas during failures is challenging.
4. Detection of Failures:
– Identifying failures, especially in asynchronous systems, is complex due to
network delays or partitions.
5. Recovery:
– Bringing the system back to a consistent state after a failure requires
checkpoints, logging, or replication techniques.
6. Performance Overhead:
– Fault-tolerant mechanisms like replication, consensus protocols, or logging can
slow down normal operations.
7. Dynamic Changes:
– Systems must adapt to changes in topology, such as nodes joining or leaving,
during fault recovery.

Commit Protocols
Commit protocols are used to ensure that distributed transactions maintain atomicity (all-or-
nothing property) despite failures.

1. Two-Phase Commit (2PC)

• A widely used protocol for ensuring atomicity.
• Phases:
a. Prepare Phase:
• The coordinator asks all participants if they are ready to commit.
• Each participant replies with "Yes" (ready) or "No" (abort).
b. Commit Phase:
• If all participants reply "Yes," the coordinator sends a commit message.
• Otherwise, it sends an abort message.
 • Advantages:
– Simple and ensures atomicity.
• Disadvantages:
– Blocking: If the coordinator fails, participants remain blocked until it recovers.

2. Three-Phase Commit (3PC)

• An improvement over 2PC to reduce blocking.
• Phases:
a. CanCommit Phase: Coordinator asks participants if they can commit.
b. PreCommit Phase: Coordinator sends a pre-commit message if all participants
agree.
c. Commit Phase: Coordinator sends a commit message after ensuring all
participants are in a pre-commit state.
• Advantages:
– Non-blocking: Participants can independently decide to abort if the coordinator
fails.
• Disadvantages:
– Increased complexity and overhead.

Voting Protocols
Voting protocols ensure consistency and fault tolerance by requiring a majority or quorum of
nodes to agree on an operation or decision.

1. Static Voting Protocols:

• A fixed set of nodes participate in the voting process.
• A majority (or quorum) of votes is required to commit a decision.
• Advantages:
– Simple and predictable.
• Disadvantages:
– Not adaptive to changes in the system (e.g., node failures or additions).

2. Dynamic Voting Protocols:

• The quorum size and participants can change dynamically based on system state.
• Ensures that the system can make progress even as nodes fail or recover.
• Advantages:
– Flexible and more resilient to failures.
• Disadvantages:
– Increased complexity in managing dynamic quorums.

Dynamic Voting Protocols

Dynamic voting protocols adapt to changing system conditions, such as node failures or
recoveries, by recalculating quorums dynamically.
Key Concepts:
• Voting Power:
– Each node is assigned a voting power.
– Votes are counted to determine whether a quorum is achieved.
• Dynamic Adjustment:
– If a node fails, its voting power is redistributed among the remaining nodes.
– If a node recovers, it regains its voting power.

Example Algorithm:
1. A transaction or operation is proposed.
2. Nodes cast votes based on their current voting power.
3. If the quorum is met (e.g., majority of votes), the operation is committed.
4. If the quorum is not met, the operation is aborted or retried.

Advantages:
• Allows progress even in systems with frequent failures or recoveries.
• Reduces the risk of blocking.

Disadvantages:
• Complex to implement and manage.

Applications of Fault Tolerance

• Distributed Databases:
– Commit protocols like 2PC ensure atomicity of distributed transactions.
• Cloud Computing:
– Replication and fault-tolerant protocols provide high availability.
• Blockchain Systems:
– Consensus protocols (e.g., Proof of Work, Paxos, Raft) ensure fault tolerance.
• Critical Systems:
– Dynamic voting protocols are used in aerospace, banking, and
telecommunications to ensure continuous operation.

Conclusion
Fault tolerance is a critical aspect of distributed systems, ensuring reliability and consistency in
the presence of failures. Techniques like commit protocols, voting, and dynamic voting provide
mechanisms to handle failures effectively, each with its trade-offs in terms of complexity,
performance, and resilience.
Transactions and Concurrency Control in Distributed Systems
Distributed systems often involve multiple processes or users accessing shared data.
Transactions and Concurrency Control mechanisms are used to maintain consistency, isolation,
and atomicity in such systems, even when multiple transactions execute concurrently.

1. Transactions
A transaction is a sequence of operations performed as a single logical unit of work. It ensures
the ACID properties:

1. Atomicity: Either all operations of the transaction are completed, or none are.
2. Consistency: A transaction transforms the system from one consistent state to another.
3. Isolation: Transactions operate independently without interfering with each other.
4. Durability: Once a transaction is committed, its changes persist, even in case of a failure.

Types of Transactions
• Flat Transactions: A single transaction with no internal structure.
• Nested Transactions: A transaction that contains sub-transactions, providing a
hierarchical structure.

2. Nested Transactions
Nested transactions allow greater modularity and fault tolerance. They consist of a parent
transaction and multiple child transactions.

Characteristics:
• Child transactions can execute independently.
• If a child transaction fails, it can roll back without affecting other child transactions.
• The parent transaction only commits when all child transactions successfully commit.

Use Cases:
• Complex distributed systems where operations can be divided into subtasks.
• Database systems for distributed queries.
3. Concurrency Control
Concurrency control ensures the correctness of transactions that execute simultaneously. It
resolves issues like:

1. Lost Updates: Two transactions overwrite each other's updates.

2. Dirty Reads: A transaction reads uncommitted changes from another transaction.
3. Non-Repeatable Reads: A transaction reads the same data twice but gets different
results.
4. Phantom Reads: A transaction retrieves a set of records and finds additional ones on re-
execution.

4. Concurrency Control Methods

a. Lock-Based Concurrency Control
Locks restrict access to data to ensure consistency.

1. Types of Locks:
– Shared Lock (Read Lock): Allows multiple transactions to read but not write.
– Exclusive Lock (Write Lock): Prevents other transactions from reading or writing.
2. Two-Phase Locking (2PL):
– Growing Phase: A transaction acquires locks but cannot release them.
– Shrinking Phase: A transaction releases locks but cannot acquire new ones.
– Guarantees serializability but may cause deadlocks.
3. Deadlock Handling:
– Deadlock Prevention: Avoid situations where circular waiting occurs.
– Deadlock Detection: Periodically check for deadlocks and resolve them by
aborting transactions.

b. Optimistic Concurrency Control (OCC)

• Assumes conflicts are rare and allows transactions to execute without locking.
• Phases:
a. Read Phase: Transactions execute, reading data and making tentative changes.
b. Validation Phase: Check if the transaction conflicts with others.
c. Write Phase: Commit changes if validation succeeds, else abort.
• Advantages:
– No locks, so no deadlocks.
– Suitable for read-heavy systems.
• Disadvantages:
– High overhead in conflict detection.
– Risk of frequent rollbacks in write-heavy workloads.
c. Timestamp Ordering
• Assigns each transaction a unique timestamp to order operations.
• Rules:
– If a transaction’s operation conflicts with another transaction, the older
timestamp takes precedence.
– Operations are delayed or aborted if they violate the order.
• Advantages:
– Ensures serializability.
– Simple to implement.
• Disadvantages:
– Requires precise timestamp generation.
– Susceptible to frequent aborts in systems with high contention.

5. Comparison of Concurrency Control Methods

Optimistic Concurrency
Aspect Lock-Based Control (OCC) Timestamp Ordering
Conflict Prevents conflicts Detects conflicts during Resolves conflicts
Handling using locks. validation. using timestamps.
Overhead High due to lock High due to validation and Moderate due to
management. rollbacks. timestamp checks.
Deadlocks Possible. Not possible. Not possible.
Best Use Case Write-heavy Read-heavy systems with Systems requiring
systems. low contention. strict ordering.

6. Summary
• Transactions ensure atomicity, consistency, isolation, and durability in distributed
systems.
• Nested Transactions enhance fault tolerance by allowing partial rollbacks.
• Concurrency Control mechanisms like locking, OCC, and timestamp ordering resolve
conflicts and maintain correctness during concurrent executions.
• Each method has trade-offs and should be chosen based on system requirements such
as read-write patterns, conflict frequency, and the need for deadlock prevention.

Distributed Transactions
Distributed transactions occur across multiple systems or nodes in a distributed environment,
where each participating node (or database) may execute different parts of the transaction.
Ensuring that distributed transactions are atomic (all-or-nothing), consistent, isolated, and
durable (ACID properties) is more complex in distributed systems due to challenges like network
failures, system crashes, and concurrent access to shared data.
1. Flat and Nested Distributed Transactions
a. Flat Distributed Transactions
• In flat transactions, all operations involved in the transaction are executed in a single
sequence, typically across multiple nodes or systems.
• Characteristics:
– It has a single scope, where all sub-transactions are treated as a whole.
– Typically follows the Two-Phase Commit (2PC) protocol for atomic commit.
– If any operation fails at any node, the entire transaction is aborted.
• Example: A financial transaction that updates accounts in two different databases. If one
of the updates fails, the entire transaction is rolled back.

b. Nested Distributed Transactions

• A nested distributed transaction allows a transaction to have sub-transactions, meaning
a transaction can contain other smaller transactions within it.
• Characteristics:
– Parent transactions can commit or abort based on the success or failure of the
child (sub) transactions.
– Provides better modularity and flexibility, as the failure of one sub-transaction
can be isolated without affecting others.
– Nested transactions are often used in complex distributed systems where
different subsystems or databases are involved.
• Example: A global order system where a parent transaction handles placing an order,
and child transactions manage payment, shipping, and inventory updates.

2. Atomic Commit Protocols

The Atomic Commit Protocol ensures that a distributed transaction is either fully committed or
fully aborted, even in the presence of system failures.

a. Two-Phase Commit (2PC)

• Phase 1 (Prepare Phase):
– The coordinator sends a "prepare" request to all participants.
– Each participant responds with either "Yes" (ready to commit) or "No" (abort).
• Phase 2 (Commit Phase):
– If all participants reply "Yes," the coordinator sends a "commit" request to all
participants.
– If any participant replies "No," the coordinator sends an "abort" message to all
participants.
• Advantages:
– Simple to understand and implement.
– Ensures atomicity and consistency.
• Disadvantages:
– Blocking: If the coordinator crashes, participants wait indefinitely.
 – Single Point of Failure: If the coordinator fails, the transaction might be left in an
uncertain state.

b. Three-Phase Commit (3PC)

• A more fault-tolerant version of 2PC that reduces the blocking problem.

• Involves an additional phase called Pre-Commit between the Prepare and Commit
phases.

• Phases:
a. CanCommit: The coordinator checks if all participants are ready to commit.
b. PreCommit: The coordinator sends a pre-commit signal to participants.
c. Commit: The coordinator sends a final commit signal after all participants are
pre-committed.
• Advantages:
– Non-blocking in case of coordinator failure.
• Disadvantages:
– More complex and overhead than 2PC.

3. Concurrency Control in Distributed Transactions

Concurrency control in distributed transactions ensures that concurrent transactions do not lead
to inconsistent states. Distributed systems use various methods to manage conflicts and ensure
serializability:

a. Lock-Based Concurrency Control

• Locks are used to control access to shared resources. A distributed lock manager
coordinates lock acquisition and release across nodes.
• Problems:
– Deadlocks: Multiple transactions waiting for each other to release locks,
preventing progress.
– Latency: Distributed locks may suffer from delays due to network
communication.

b. Optimistic Concurrency Control (OCC)

• In OCC, transactions are executed without locks, assuming that conflicts are rare.
• Phases:
a. Read Phase: Transactions execute and make tentative changes.
b. Validation Phase: The system checks if the transaction conflicts with other
transactions.
c. Write Phase: If validation is successful, the transaction commits; otherwise, it
aborts.
• Advantages:
– No locking, thus preventing deadlocks.
 – High performance in read-heavy systems.
• Disadvantages:
– High overhead due to conflict validation.
– Transactions may be rolled back frequently in write-heavy systems.

c. Timestamp Ordering
• Each transaction is assigned a timestamp. Transactions are ordered based on their
timestamps, and their operations are executed in timestamp order.
• Advantages:
– Simple and ensures serializability.
• Disadvantages:
– Risk of transaction aborts due to timestamp violations.

4. Distributed Deadlocks
Deadlocks in distributed systems occur when transactions are stuck in a waiting state due to
resource contention, and no transaction can make progress.

a. Detection
• Distributed systems use algorithms to detect deadlocks by analyzing the dependencies
between transactions.
– Centralized Deadlock Detection: One node (centralized controller) monitors
transactions and detects deadlocks.
– Distributed Deadlock Detection: Multiple nodes coordinate to detect deadlocks,
typically using algorithms like path pushing or edge chasing.

b. Resolution
• Once deadlock is detected, the system must resolve it, typically by aborting one or more
transactions to break the cycle.

5. Transaction Recovery
Transaction recovery ensures that distributed transactions can be restored to a consistent state
after a failure, without violating ACID properties.

a. Checkpoints
• A checkpoint is a snapshot of the system at a particular point in time, recording the state
of the transactions and data.
• When a failure occurs, the system can roll back to the last checkpoint, ensuring that no
partially completed transactions are left behind.

b. Log-Based Recovery
• The system maintains a log of all operations performed by the transactions.
• In case of failure, the system can redo or undo operations based on the log:
– Redo: Reapplies committed transactions.
 – Undo: Rolls back uncommitted transactions.

c. Distributed Database Recovery**

• In distributed databases, recovery involves coordinating the recovery of all involved
nodes.
• Techniques like write-ahead logging ensure that changes are logged before they are
committed, enabling recovery even if the system crashes.

Summary
• Distributed Transactions involve coordinating multiple systems to ensure the ACID
properties are maintained across various nodes.
• Flat transactions are simple, while nested transactions allow more modular,
hierarchical operations.
• Atomic Commit Protocols, like 2PC and 3PC, are essential for ensuring atomicity and
consistency across nodes.
• Concurrency Control ensures transactions execute without conflicts, using techniques
like locking, optimistic concurrency control, and timestamp ordering.
• Distributed Deadlocks are handled by detection algorithms and resolution strategies to
prevent transactions from being stuck.
• Transaction Recovery uses techniques like checkpoints and log-based recovery to
restore consistency after a failure.

Replication in Distributed Systems

Replication in distributed systems involves creating copies (replicas) of data across different
nodes to improve availability, fault tolerance, and performance. By maintaining multiple copies
of data, systems can continue to function even when some of the nodes fail. This is crucial for
ensuring high availability and resilience in large-scale distributed environments.

1. System Model for Replication

A system model for replication in distributed systems typically includes the following
components:

• Nodes: These are the machines or servers that store replicas of data.
• Replica: A copy of a piece of data that is stored on multiple nodes.
• Primary (Master) and Replica (Slave) Nodes: A primary node holds the original version
of the data, while replica nodes hold copies of the data.
• Quorum: In some systems, a minimum number of nodes must agree on a change before
it is committed, ensuring consistency.
• Consistency Level: Defines how many replicas need to be updated before a transaction is
considered committed. Different consistency levels include:
– Strong Consistency: All replicas are identical after each update (e.g.,
linearizability).
 – Eventual Consistency: Updates propagate over time, but replicas might
temporarily differ (e.g., used in systems like Cassandra).

Types of Replication Models:

• Synchronous Replication:
– Changes to the data are applied to all replicas at the same time.
– Ensures consistency but can incur high latency, especially in geographically
distributed systems.
– Example: Paxos and Raft are popular protocols used to ensure synchronous
replication.
• Asynchronous Replication:
– Updates are made to the primary node first, and the changes are propagated to
replicas after a delay.
– Less costly in terms of latency but can result in temporary inconsistencies.
– Example: Cassandra uses an asynchronous replication model, making it more
efficient for write-heavy workloads.
• Hybrid Replication:
– A combination of synchronous and asynchronous replication, used to balance
consistency and performance.

2. Group Communication
Group communication is the foundation of replication in distributed systems. It refers to the
coordination of messages between multiple processes or nodes in a group, ensuring that
messages are delivered to all members in the group in a consistent and reliable manner.

Key Concepts in Group Communication:

• Broadcasting: Sending a message to all members of a group.
• Multicast: Efficiently sending messages to a selected group of nodes (not necessarily all
nodes).
• Reliability: Ensures that messages are not lost and are delivered to all members.
• Ordering: Ensures that messages are delivered in a specific order, which is crucial for
maintaining consistency across replicas.

Types of Ordering in Group Communication:

• Total Order: Ensures that all members of the group receive messages in the same order.
• Causal Order: Guarantees that if one message causally influences another, they are
delivered in the correct order.
• FIFO (First In First Out) Order: Messages from a single sender are delivered in the order
they were sent, but there’s no guarantee across different senders.

Group Communication Protocols:

• X.500: A directory service protocol that can support group communication in distributed
systems.
 • Reliable Multicast: Ensures message delivery to all nodes, even if some messages are
lost.
• Virtual Synchrony: Guarantees that all processes in the group receive messages in the
same order, ensuring consistency in replicated systems.

3. Fault Tolerance in Replication

Fault tolerance in replicated systems ensures that the system can continue to function properly
even in the presence of failures. Replication enhances fault tolerance by providing backup
copies of data across different nodes or systems, which increases the likelihood that at least one
copy will remain available in case of failure.

Techniques for Fault Tolerance:

• Redundancy: Having multiple copies of data on different nodes or servers to ensure
availability.

• Failure Detection: Mechanisms to detect when a node or replica is unavailable,

allowing the system to adjust its operations accordingly.

• Recovery: The ability to restore a system to a consistent state after a failure. In the
case of replication, this could mean restoring the primary data from a replica or
reintegrating a failed replica.
– Primary-backup replication: One replica is designated as the primary, and others
as backups. If the primary node fails, one of the backups is promoted to primary.
– Quorum-based Replication: A quorum defines a minimum number of replicas
that must participate in any read or write operation. A quorum helps to maintain
data consistency and ensures that a majority of nodes can make decisions
independently, even if some nodes are down.

Fault Tolerant Protocols:

• Paxos: A consensus algorithm used for fault-tolerant replication, ensuring that even if a
majority of nodes fail, the system can continue to function and agree on a consistent
value.
• Raft: Another consensus algorithm that is simpler and easier to implement than Paxos,
ensuring fault tolerance and consistency across replicated nodes.

4. Challenges in Replication and Fault Tolerance

While replication significantly improves fault tolerance and availability, it introduces a number of
challenges in terms of consistency, performance, and system complexity. Here are some of the
main challenges:

a. Consistency vs. Availability (CAP Theorem):

• The CAP theorem states that a distributed system can only provide at most two of the
following three guarantees at any time:
 – Consistency: All replicas are identical at all times.

– Availability: Every request to the system will receive a response.

– Partition Tolerance: The system can continue to operate even if

communication between nodes is interrupted.

– Trade-off: When a network partition occurs, systems may need to choose

between consistency and availability. Some systems favor consistency (e.g.,
HBase, Zookeeper), while others favor availability (e.g., Cassandra,
DynamoDB).

b. Latency and Performance:

• Synchronous replication can introduce significant latency because each update must be
confirmed by all replicas before it is considered complete. This can be problematic in
systems where low-latency response times are required.
• Asynchronous replication improves performance but at the cost of consistency, as
replicas might be temporarily out of sync.

c. Handling Failures and Data Recovery:

• When nodes fail, ensuring that the system recovers without data loss or inconsistency is
a key challenge. This is particularly important in high-availability systems like distributed
databases.
• Recovery mechanisms (e.g., checkpointing, log-based recovery) must be efficient
enough to ensure the system can recover quickly without compromising data integrity.

d. Split-Brain Problem:
• The split-brain scenario occurs when there is a network partition, and multiple replicas
think they are the primary replica. This leads to inconsistent data and can cause major
issues in the system. Protocols like Quorum-based replication and Raft can help avoid
this problem by ensuring that only one replica is allowed to be the primary at any given
time.

5. Applications of Replication
• Distributed Databases: Replication helps maintain consistency and availability in
distributed databases like MongoDB, Cassandra, and HBase.
• Cloud Services: Cloud providers use replication to ensure that data is highly available,
even in case of failures. Services like Amazon S3 and Google Cloud Storage rely on
replication.
• Distributed File Systems: Systems like HDFS (Hadoop Distributed File System)
replicate data across multiple nodes to ensure that files are available even if one or more
nodes fail.
• Content Delivery Networks (CDNs): CDNs use replication to store copies of data at
multiple locations, reducing latency and improving access speed for users across the
globe.
Summary
• Replication in distributed systems involves storing multiple copies of data across
different nodes to improve availability, fault tolerance, and performance.
• The system model for replication ensures that data is consistent and reliable, and group
communication ensures that updates to replicas are coordinated across all nodes.
• Fault tolerance in replicated systems is achieved through redundancy, failure detection,
and recovery mechanisms, and protocols like Paxos and Raft ensure consistency despite
failures.
• Challenges in replication include ensuring consistency (CAP Theorem), handling network
partitions, and managing the performance overhead of maintaining multiple replicas.