Latency and Throughput in System Design

Latency refers to the time it takes for a request to travel from its point of origin to its destination and receive a response. . It combines a number of delays - Response times, transmission, and processing time.

• Latency represents the delay between an action and its corresponding reaction.
• It can be measured in various units like seconds, milliseconds, and nanoseconds depending on the system and application.

What does it involve?

Latency involves so many things such as processing time, time to travel over the network between components, and queuing time.

• Round Trip Time: This includes the time taken for the request to travel to the server, processing time at the server, and the response time back to the sender.
• Different Components: Processing time, transmission time (over network or between components), queueing time (waiting in line for processing), and even human reaction time can all contribue to overall latency.

What is Throughput?

The rate at which a system, process, or network can move data or carry out operations in a particular period of time is referred to as throughput. Bits per second (bps), bytes per second, transactions per second, etc. are common units of measurement. It is computed by dividing the total number of operations or objects executed by the time taken.

For example, an ice-cream factory produces 50 ice-creams in an hour so the throughput of the factory is 50 ice-creams/hour.

Here are a few contexts in which throughput is commonly used:

1. Network Throughput: Throughput in networking is the quantity of data that can be sent via a network in a specific amount of time. When assessing the effectiveness of communication routes, this measure is important.
2. Disk Throughput: In storage systems, throughput measures how quickly data can be read from or written to a storage device, usually expressed in terms of bytes per second.
3. Processing Throughput: In computing, especially in the context of CPUs or processors, throughput is the number of operations completed in a unit of time. It could refer to the number of instructions executed per second.

Differences between Throughput and Latency (Throughput vs. Latency)

How does Latency works?

The time taken for each step—transmitting the action, server processing, transmitting the response, and updating your screen—contributes to the overall latency.

Example: Let see an example when player in an online game firing a weapon.

• If your latency is high: You press "fire."
• The command travels through the internet to the server, which takes time.
• The server processes the shot.
• The result travels back to your device.
• Your screen updates the result.

During this time, another player might have moved or shot you, but their actions haven't reached your device yet due to latency. This can result in what's called "shot registration delay." Your actions feel less immediate, and you might see inconsistencies between what you're seeing and what's happening in the game world.

The working of Latency can be understood by two ways:

1. Network Latency

In system architecture, network latency is a sort of latency that describes how long it takes for data to move between two points in a network. Using email as an example, we can consider it to be the time lag between sending an email and the recipient actually getting it. For real-time applications, it is measured in milliseconds or even microseconds, just like total latency.

2. System Latency

System latency refers to the overall time it takes for a request to go from its origin in the system to its destination and receive a response. Think of Latency as the "wait time" in a system. The time between clicking and seeing the updated webpage is the system latency. It includes processing time on both client and server, network transfers, and rendering delays.

Factors that causes High Latency

High latency can severely impact the performance and user experience of distributed systems. Here are key factors that contribute to high latency within this context:

1. Network Congestion: High traffic on a network can cause delays as data packets queue up for transmission.
2. Bandwidth Limitations: Limited bandwidth can cause delays in data transmission, particularly in data-intensive applications.
3. Geographical Distance: Data traveling long distances between distributed nodes can increase latency due to the inherent delays in transmission.
4. Server Load: Overloaded servers can take longer to process requests, contributing to high latency.
5. Latency in Database Queries: Complex or inefficient database queries can significantly increase response times.

How to measure Latency?

There are various ways to measure latency. Here are some common methods:

• Ping: This widely used tool sends data packets to a target server and measures the round-trip time (RTT), providing an estimate of network latency between two points. (RTT = 2 * one-way latency).
• Traceroute: This tool displays the path data packets take to reach a specific destination, revealing which network hops contribute the most to overall latency.
• MTR (traceroute with ping): Combines traceroute and ping functionality, showing both routing information and RTT at each hop along the path.
• Performance profiling tools: Specialized profiling tools track resource usage and execution times within a system, providing detailed insights into system latency contributors.
• Application performance monitoring (APM) tools: Similar to network monitoring tools for networks, APM tools monitor the performance of applications, including response times and latency across various components.

Example for calculating the Latency

Problem Statement:

Calculate the round-trip time (RTT) latency for a data packet traveling between a client in New York City and a server in London, UK, assuming a direct fiber-optic connection with a propagation speed of 200,000 km/s.

• Distance: Distance between NYC and London: 5570 km
• Propagation speed: 200,000 km/s 
• Constraints: Assume no network congestion or processing delays.
• Desired Output: RTT latency in milliseconds.

1. Calculate One-Way Latency: One-way latency is the time taken for the data to travel from the client to the server:

One-way latency = Distance / Propagation speed = 5570 KM / 200,000 Km/s = 27.85 ms

2. Calculate RTT: The RTT is twice the one-way latency:

RTT = 2 × 27.85ms = 55.7ms

Use Cases of Latency

Below are some of the important use cases of latency:

1. User Experience in Applications: Low latency ensures smooth experiences in apps like online banking, e-commerce, or streaming platforms.
2. Gaming and Virtual Reality (VR): Real-time interaction in multiplayer games or VR systems requires minimal latency for responsiveness.
3. Video Streaming: Platforms like YouTube and Netflix rely on low latency to deliver buffer-free streaming.
4. Online Meetings: Video conferencing tools (e.g., Zoom, Google Meet) depend on low latency for real-time communication.
5. Financial Transactions: In stock trading or payment systems, lower latency helps execute transactions faster and reduces risks.
6. IoT and Smart Devices: Devices like smart thermostats or autonomous cars need low latency for timely responses.
7. Healthcare: Applications like telemedicine or robotic surgeries demand low latency for real-time feedback and precision.

Factors affecting Throughput

• Network Congestion: High levels of traffic on a network can lead to congestion, reducing the available bandwidth and impacting throughput.
• Bandwidth Limitations: The maximum capacity of the network or communication channel can constrain throughput. Upgrading to higher bandwidth connections can address this limitation.
• Hardware Performance: The capabilities of routers, switches, and other networking equipment can influence throughput. Upgrading hardware or optimizing configurations may be necessary to improve performance.
• Software Efficiency: Inefficient software design or poorly optimized algorithms can contribute to reduced throughput.
• Latency: High latency can impact throughput, especially in applications where real-time data processing is crucial.

Methods to improve Throughput

1. Network Optimization:
   • Utilize efficient network protocols to minimize overhead.
   • Optimize routing algorithms to reduce latency and packet loss.
2. Load Balancing:
   • Distribute network traffic evenly across multiple servers or paths.
   • Prevents resource overutilization on specific nodes, improving overall throughput.
3. Hardware Upgrades:
   • Upgrade network devices, such as routers, switches, and NICs, to higher-performing models.
   • Ensure that servers and storage devices meet the demands of the workload.
4. Software Optimization:
   • Optimize algorithms and code to reduce processing time.
   • Minimize unnecessary computations and improve code efficiency.
5. Compression Techniques:
   • Use data compression to reduce the amount of data transmitted over the network.
   • Decreases the time required for data transfer, improving throughput.
6. Caching Strategies:
   • Implement caching mechanisms to store and retrieve frequently used data locally.
   • Reduces the need to fetch data from slower external sources, improving response times and throughput.