Parallel and Distributed

Computing
Communication in Networks
Today We Will learn,
• Communication Unit
• Communication Costs in Parallel Machines
• Message Routing Techniques
- Store-and-Forward Routing
- Packet Routing
- Cut-Through Routing
Network Layers Model (OSI) - Revisited
• Typical Model showing the Layers of the Network
Communication Unit
• Message:
- the unit of communication from the programmer’s perspective.
- Its size is limited only by the computer’s memory space

• Packet:
- Fixed-size small unit of communication containing routing information (e.g., a
destination address) and sequencing information in its header.
- Its size is of order hundred or thousands of bytes or words
- Data sent over computer networks, such as the Internet, is divided into packets.
- These packets are then recombined by the computer or device that receives them.
- It consists of header flits and data flits
Illustration of Communication Unit

• Flit: small unit of information at Link layer, of size of a few words

• Phit: the smallest physical unit of information at the Physical Layer, which is

transferred across one physical link in one cycle

Communication Unit
• Congestion:
- A state occurring in network layer when the message traffic is so heavy

that it slows down network response time.

- As delay increases, performance decreases.
- If delay increases, retransmission occurs, making situation worse.
Communication Unit
• Router
- A Router is a networking device that forwards data packets between computer
network.
- This device is usually connected to two or more different networks.
- When a data packet comes to a router port, the router reads address
information in packet to determine out which port the packet will be sent.
- For example, a router provides you with the internet access by connecting your
LAN with the Internet.
Communication Unit
• Routing Table
- Each router on the network supports a routing table in memory that may be simple
or complex. In the simplest form, the table consists of pairs of IP addresses.
- A routing table is a set of rules, often viewed in table format, that is used to
determine where data packets traveling over an Internet Protocol (IP) network will
be directed.
- All IP-enabled devices, including routers and switches, use routing tables. See
below a Routing Table:
Communication Costs in Parallel Machines
• One of the major overheads in the execution of parallel programs arises from
communication of information between processing elements.
• The cost of communication is dependent on a variety of features including
- Programming Model Semantics
- Network Topology
- Data Handling and Routing
- Communication Software Protocols
Major Parameters in Communication Costs (I)
• Time for communicating a message between two nodes is the sum of:
- Time to prepare a message for transmission
- Time taken by the message to traverse the network to its destination
• The principal parameters that determine the communication latency are as
follows:
- Startup time ( t)

- Per-hop Time (th )

- Per-word transfer time (tw)

Major Parameters in Communication Costs (I)
• Startup time ( t): time required to handle a message at the sending and
receiving nodes
- Prepare message (adding header, trailer, error correction information)
- Execute the routing algorithm
- Establish an interface between the local node and the router

• This latency is only incurred once for a single message transfer

Major Parameters in Communication Costs (I)
• Per-hop Time (th ):
- After a message leaves a node, it takes a finite amount of time to reach the
next node in its path.
- It is the time taken by the header of a message to travel between two
directly connected nodes in the network
- It is directly related to the latency within the routing switch for
determining which output buffer or channel the message should be
forwarded to
- The per-hop time is also called node latency
Major Parameters in Communication Costs (I)
• Per-word transfer time (tw):
- Time taken for one word to traverse a link
- This time includes network and buffering overheads
- Per-word transfer time is the reciprocal of the channel bandwidth.
- If the channel bandwidth is r words per second, then each word takes

time tw = 1/r to traverse the link.

Major Parameters in Communication Costs (I)
• Per-word transfer time (tw):

Example : if channel bandwidth is 50 words/Minute, then find

the per word transfer time
Major Parameters in Communication Costs (I)
• Per-word transfer time (tw):

Example : if channel bandwidth is 50 words/Minute, then find

the per word transfer time

Solution:
r = Channel bandwidth = 50 Words/Minutes
= 50/60 Words/Seconds
= 0.83 Words/Seconds

Tw = 1/r = 1/0.83
Message Routing Techniques
• Store-and-Forward Routing

• Packet Routing

• Cut-Through Routing
Message Routing Techniques
• Store-and-Forward Routing (I)
- Technique where the data packets are stored in each intermediate node, before
they are forwarded to the next node.
- The intermediate node checks whether the packet is error−free before
transmitting, thus ensuring integrity of the data packets.
- In general, the network layer operates in an environment that uses store and
forward packet switching.
- illustration
Message Routing Techniques
• Store-and-Forward Routing (I)
- Total communication cost for a message of size m to traverse a path of l
links

-Note that the lack of parallelism in utilizing communication resources

Message Routing Techniques
• Per-word transfer time (tw):

Example : Suppose there are 10 links, from where a message is

sent from source to destination, message took 10 seconds to
start it for sending from source to destination. Whereas, the
channel bandwidth is 50 words/Minute. Find total
communication cost using store-and-forward scheme.
Message Routing Techniques
Example : Ali send a Message and it passes through 10 intermediate links to destination
its destination, where each node took 5 Seconds to validate/verify the message. Ali took
took 10 seconds to compose/start message for forwarding to Sehar/destination. The
channel bandwidth was 50 words/Minute. Find total communication cost using store-
and-forward scheme.

Solution:
r = Channel bandwidth = 50 Words/Minutes = 50/60 Words/Seconds = 0.83 Words/Seconds
Tw = 1/r = 1/0.83

Using the equation, the result is

Tcomm = 10+ ( 50 X 1.20 + 5) X 10
Message Routing Techniques
• Store-and-Forward Routing (II)
- For present generation parallel computers, th is very small

- The quantity mtw is much larger. So we usually ignore the per-hop time

in store-and-forward routing scheme

- The simplified formula is:
Message Routing Techniques
• Packet Routing
- Connectionless network switching technique.
- A long message is cut into pieces
- Here, the message is divided and grouped into a number of units called
packets that are individually routed from the source to the destination.
- Message pieces are sent through the network one followed by another
- Store-and-Forward routing makes poor use of communication
resources.
- A message is sent from one node to the next only after the entire
message has been received
Message Routing Techniques
• Packet Routing
- Advantages:

 Better utilization of communication resources

 Lower overhead from packet loss (errors)

Packets may take different paths

Better error correction capability

Message Routing Techniques
• Packet Routing
Message Routing Techniques
• Packet Routing
- Packet routing is suited to networks with highly dynamic states and higher
error rates, such as local- and wide-area networks.
- This is because individual packets may take different routes and
retransmissions can be localized to lost packets.
Message Routing Techniques
• Packet Routing Cost
Cut-Through Routing
• A special packet switching for parallel computers with the following
properties to reduce cost:
- All packets go through the same path
- In-sequence delivery
- Associate error information at message level
- Use lean error detection mechanisms
Flits Communication
• Cut-through routing uses flits

• A tracer is first sent from the source to the destination node to

establish a connection

• Flits are sent through the path one after the other

• An immediate node forwards the flit as soon as it is received

• Buffer space is not necessary at the immediate nodes

Communication Cost of CT Routing
• Consider a message that is traversing such a network.

• If the message traverses l links, and th is the per-hop time, then the header of

the message takes time lth to reach the destination. If the message is m words

long, then the entire message arrives in time twm after the arrival of the
header of the message. Therefore, the total communication time for cut-
through routing is
Communication Cost of CT Routing

• Note that it no longer contains the product of message size and number of
links
• Cut-through routing is fast for large size message and long distance
communication
• For nearest neighbor communication, store-and forward routing and cut-
through routing are similar
Message Routing Techniques
Message Routing Techniques
Message Routing Techniques
Routing Mechanisms for Interconnection Networks
• Efficient algorithms for routing a message to its destination are critical to the performance
of parallel computers.

• A routing mechanism determines the path a message takes through the network to get
from the source to the destination node.
- It takes as input a message's source and destination nodes. It may also use information
about the state of the network.
- It returns one or more paths through the network from the source to the destination node.
Two routing mechanisms:

- Minimal and
- Non-minimal
Routing Mechanisms for Static Networks
• Minimal routing
- always selects one of the shortest paths between the source and
destination node
- In a minimal routing scheme, each link brings a message closer to its
destination
- This scheme could lead to congestion in parts of the network

• Non-minimal routing
- may route the message along a longer path to avoid network congestion
Routing Mechanisms for Static Networks
• Routing mechanisms can also be classified on the basis of how they use
information regarding the state of the network

• Two possible classes are

- Deterministic routing
- Adaptive routing
Routing Mechanisms for Static Networks
• Deterministic routing
- A deterministic routing scheme determines a unique path for a message,
based on its source and destination.
- It does not use any information regarding the state of the network
- Results in uneven use of the communication resources in a network.

• Adaptive routing
- In contrast, an adaptive routing scheme uses information regarding the
current state of the network to determine the path of the message.
- Adaptive routing detects congestion in the network and routes messages
around it.
Routing Mechanisms for Static Networks
• One commonly used deterministic minimal routing technique is called dimension-
ordered routing.

• Dimension-ordered routing assigns successive channels for traversal by a message

based on a numbering scheme determined by the dimension of the channel.

• The dimension ordered routing technique for a two-dimensional mesh is called

XY-routing and that for a hypercube is called E-cube routing.
- Both are deterministic and minimal routing techniques

- Both routing schemes are dead-lock free

Routing Mechanisms for Static Networks
• XY-Routing Scheme
- Message is first sent along the X dimension until it reaches the column
of the destination node
- Then it is routed along the Y dimension until it reaches the destination
- The length of the path is
| Sx − Dx | + | Sy − Dy |

- Source node: Psx, Sy

- Destination node : PDx, Dy

Routing Mechanisms for Static Networks
• XY-Routing Scheme
Routing Mechanisms for Static Networks
• E-Cube Routing Scheme
- The number of links in the path is determined by the binary labels of
the source and destination nodes (Hamming distance)
- Each step corrects one bit difference, starting from the least significant
digit (exclusive-or operation)
Embedding and Graph Mapping

• Embedding one network into another is important for porting

algorithms

• No need to develop algorithms for every network topology

• Process-processor mapping is not controlled by the programmer

• Mapping can be used to determine the degradation in the performance

of an algorithm
Graph Mapping Terminologies

• Congestion: Maximum number of edges of E mapped on to any edge in E‘

• Dilation: Maximum number of edges of E‘ that any one edge of E is mapped

onto That is how much we stretch an edge of E

• Expansion: |V' |/| V|

• Load: Maximum number of vertices of E that are mapped to a single vertex of

E'
Routing Mechanisms for Static Networks
• Routing mechanisms can also be classified on the basis of how they use
information regarding the state of the network.

• A deterministic routing scheme determines a unique path for a message,

based on its source and destination does not use network status

• An adaptive routing scheme uses information regarding the current status of

network to determine the path of message may detect a congestion spot and
route message around it