Unit 2 (Distributed

systems)

Dr. Ritu Gothwal

IILM, University
Distributed Debugging

Distributed debugging is the process of debugging multiple related processes that run in different environments within
a distributed application. Instead of debugging a single program or process, you can track and debug interactions
between multiple components—such as a client application and a remote server—within the same debugging session.

• It involves tracking the flow of operations across multiple nodes, which requires tools and techniques like
logging, tracing, and monitoring to capture and analyze system behavior.

• Issues such as synchronization errors, concurrency bugs, and network failures are common challenges in
distributed systems. Debugging aims to ensure that all parts of the system work correctly and efficiently
together, maintaining overall system reliability and performance.
 Common Sources of Errors and Failures in Distributed Systems
 Network Issues: Problems such as latency, packet loss, jitter, and disconnections can disrupt communication between
nodes, causing data inconsistency and system downtime.
 Concurrency Problems: Simultaneous operations on shared resources can lead to race conditions, deadlocks, and
livelocks, which are difficult to detect and resolve.
 Data Consistency Errors: Ensuring data consistency across multiple nodes can be challenging, leading to replication
errors, and partition tolerance issues.
 Faulty Hardware: Failures in physical components like servers, storage devices, and network infrastructure can introduce
errors that are difficult to trace back to their source.
 Software Bugs: Logical errors, memory leaks, improper error handling, and bugs in the code can cause unpredictable
behavior and system crashes.
 Configuration Mistakes: Misconfigured settings across different nodes can lead to inconsistencies, miscommunications,
and failures in the system's operation.
 Security Vulnerabilities: Unauthorized access and attacks, such as Distributed Denial of Service (DDoS), can disrupt
services and compromise system integrity.
 Resource Contention: Competing demands for CPU, memory, or storage resources can cause nodes to become
unresponsive or degrade in performance.
 Time Synchronization Issues: Discrepancies in system clocks across nodes can lead to coordination problems, causing
errors in data processing and transaction handling.
 Logging

Logging involves capturing detailed records of events, actions, and state changes within the system. Key aspects include:
 Centralized Logging: Collect logs from all nodes in a centralized location to facilitate easier analysis and correlation of
events across the system.
 Log Levels: Use different log levels (e.g., DEBUG, INFO, WARN, ERROR) to control the verbosity of log messages,
allowing for fine-grained control over the information captured.
 Structured Logging: Use structured formats (e.g., JSON) for log messages to enable better parsing and searching.
 Contextual Information: Include contextual details like timestamps, request IDs, and node identifiers to provide a clear
picture of where and when events occurred.
 Error and Exception Logging: Capture stack traces and error messages to understand the root causes of failures.
 Log Rotation and Retention: Implement log rotation and retention policies to manage log file sizes and storage
requirements.
 Tools for Logging: ELK Stack, Fluentd, Loki, Google Cloud Logging
 Monitoring

 Monitoring involves continuously observing the system's performance and health to detect anomalies and
potential issues. Key aspects include:
 Metrics Collection: Collect various performance metrics (e.g., CPU usage, memory usage, disk I/O,
network latency) from all nodes.
 Health Checks: Implement regular health checks for all components to ensure they are functioning
correctly.
 Alerting: Set up alerts for critical metrics and events to notify administrators of potential issues in real-
time.
 Visualization: Use dashboards to visualize metrics and logs, making it easier to spot trends, patterns, and
anomalies.
 Tracing: Implement distributed tracing to follow the flow of requests across different services and nodes,
helping to pinpoint where delays or errors occur.
 Anomaly Detection: Use machine learning and statistical techniques to automatically detect unusual
patterns or behaviors that may indicate underlying issues.
 Tools for Monitoring: Prometheus, Grafana, Datadog, New Relic
 Distributed tracing

 Distributed tracing is a technique used to track and visualize the flow of requests as they move
through different services in a distributed system. It helps developers to understand how a request
travels across microservices, detect bottlenecks, and troubleshoot issues efficiently.
 Trace Propagation: Passing trace context (e.g., trace ID and span ID) along with requests to maintain
continuity as they move through the system.
 End-to-End Visibility: Capturing traces across all services and components to get a comprehensive
view of the entire request lifecycle.
 Latency Analysis: Measuring the time spent in each service or component to identify where delays or
performance issues occur.
 Error Diagnosis: Pinpointing where errors happen and understanding their impact on the overall
request.
 Remote Debugging in Distributed Systems
 Remote debugging is a critical technique in debugging distributed systems, where developers need to diagnose and fix issues on systems
that are not physically accessible. It involves connecting to remote nodes or services to investigate and resolve problems. This technique is
essential due to the distributed nature of these systems, where components often run on different machines, sometimes across various
geographic locations.

 Remote Debugging Tools: Utilize specialized tools that support remote connections to debug applications running on distant servers.
 GDB (GNU Debugger): Supports remote debugging through gdbserver.
 Eclipse: Offers remote debugging capabilities through its Java Debug Wire Protocol (JDWP).
 Visual Studio: Provides remote debugging features for .NET applications.
 IntelliJ IDEA: Supports remote debugging for Java applications.
 Secure Connections: Establish secure connections using SSH, VPNs, or other secure channels to protect data and maintain confidentiality
during the debugging session.
 Configuration: Properly configure the remote environment to allow debugging. This may involve:
 Opening necessary ports in firewalls.
 Setting appropriate permissions.
 Installing and configuring debugging agents or servers.
 Breakpoints and Watchpoints: Set breakpoints and watchpoints in the code to pause execution and inspect the state of the application at
specific points.
 Logging and Monitoring: Use enhanced logging and monitoring to gather additional context and support remote debugging efforts. This
includes real-time log streaming and metric collection.
 Steps for Remote Debugging

 Ensure the remote machine is prepared for debugging. This includes installing the necessary
debugging tools and ensuring the application is running with debug symbols or in debug mode.
 Step 1: Configure Local Debugger: Configure the local debugger to connect to the remote machine.
This typically involves specifying the remote machine's address, port, and any necessary
authentication credentials.
 Step 2: Establish Connection: Use secure methods to establish a connection between the local
debugger and the remote machine.
 Step 3: Set Breakpoints: Identify and set breakpoints in the application code where you suspect
issues may be occurring.
 Step 4: Debug: Start the debugging session, and use the debugger's features to step through code,
inspect variables, and evaluate expressions.
 Step 5: Analyze and Fix: Analyze the gathered data to identify the root cause of the issue and apply
necessary fixes.
 Distributed Mutual exclusion
 In single computer system, memory and other resources are shared between different processes. The status of
shared resources and the status of users is easily available in the shared memory so with the help of shared
variable (For example: Semaphores) mutual exclusion problem can be easily solved.
 In Distributed systems, we neither have shared memory nor a common physical clock and therefore we can not
solve mutual exclusion problem using shared variables. To eliminate the mutual exclusion problem in
distributed system approach based on message passing is used. A site in distributed system do not have
complete information of state of the system due to lack of shared memory and a common physical clock.
Requirements of Mutual exclusion Algorithm:
 No Deadlock: Two or more site should not endlessly wait for any message that will never arrive.
 No Starvation: Every site who wants to execute critical section should get an opportunity to execute it in
finite time. Any site should not wait indefinitely to execute critical section while other site are repeatedly
executing critical section
 Fairness: Each site should get a fair chance to execute critical section. Any request to execute critical section
must be executed in the order they are made i.e Critical section execution requests should be executed in the
order of their arrival in the system.
 Fault Tolerance: In case of failure, it should be able to recognize it by itself in order to continue functioning
without any disruption.
 Solution to distributed mutual exclusion:
 Message passing is a way to implement mutual exclusion. Below are the three approaches based on message passing to implement mutual exclusion in distributed
systems.

1. Token Based Algorithm:

 A unique token is shared among all the sites.
 If a site possesses the unique token, it is allowed to enter its critical section
 This approach uses sequence number to order requests for the critical section.
 Each requests for critical section contains a sequence number. This sequence number is used to distinguish old and current requests.
 This approach insures Mutual exclusion as the token is unique
 Example: Suzuki–Kasami algorithm

2. Non-token based approach:

 A site communicates with other sites in order to determine which sites should execute critical section next. This requires exchange of two or more successive
round of messages among sites.
 This approach use timestamps instead of sequence number to order requests for the critical section.
 When ever a site make request for critical section, it gets a timestamp. Timestamp is also used to resolve any conflict between critical section requests.
 All algorithm which follows non-token based approach maintains a logical clock. Logical clocks get updated according to Lamport’s scheme.
 Example: Ricart–Agrawala algorithm

3. Quorum based approach:

 Instead of requesting permission to execute the critical section from all other sites, Each site requests only a subset of sites which is called a quorum.
 Any two subsets of sites or Quorum contains a common site.
 This common site is responsible to ensure mutual exclusion, Example: Maekawa’s Algorithm
 Distributed consensus

 Distributed consensus in distributed systems refers to the process by which multiple nodes or components in a network
agree on a single value or a course of action despite potential failures or differences in their initial states or inputs. It is
crucial for ensuring consistency and reliability in decentralized environments where nodes may operate independently and
may experience delays or failures. Popular algorithms like Paxos and Raft are designed to achieve distributed consensus
effectively. Importance of Distributed Consensus in Distributed Systems is discussed below .
 Consistency and Reliability: Distributed consensus ensures that all nodes in a distributed system agree on a common
state or decision. This consistency is crucial for maintaining data integrity and preventing conflicting updates.
 Fault Tolerance: Distributed consensus mechanisms enable systems to continue functioning correctly even if some
nodes experience failures or network partitions. By agreeing on a consistent state, the system can recover and
continue operations smoothly.
 Decentralization: In decentralized networks, where nodes may operate autonomously, distributed consensus
allows for coordinated actions and ensures that decisions are made collectively rather than centrally. This is essential
for scalability and resilience.
 Concurrency Control: Consensus protocols help manage concurrent access to shared resources or data across
distributed nodes. By agreeing on the order of operations or transactions, consensus ensures that conflicts are
avoided and data integrity is maintained.
 Blockchain and Distributed Ledgers: In blockchain technology and distributed ledgers, consensus algorithms (e.g.,
Proof of Work, Proof of Stake) are fundamental. They enable participants to agree on the validity of transactions and
maintain a decentralized, immutable record of transactions.
 Challenges of Achieving Consensus

 Achieving consensus in distributed systems presents several challenges due to the inherent
complexities and potential uncertainties in networked environments. Some of the key challenges
include:
 Network Partitions: Network partitions can occur due to communication failures or delays between nodes.
Consensus algorithms must ensure that even in the presence of partitions, nodes can eventually agree on a
consistent state or outcome.
 Node Failures: Nodes in a distributed system may fail or become unreachable, leading to potential
inconsistencies in the system state. Consensus protocols need to handle these failures gracefully and ensure
that the system remains operational.
 Asynchronous Communication: Nodes in distributed systems may communicate asynchronously, meaning
messages may be delayed, reordered, or lost. Consensus algorithms must account for such communication
challenges to ensure accurate and timely decision-making.
 Byzantine Faults: Byzantine faults occur when nodes exhibit arbitrary or malicious behavior, such as sending
incorrect information or intentionally disrupting communication. Byzantine fault-tolerant consensus
algorithms are needed to maintain correctness in the presence of such faults.