Lecture 3.2.

4 (Various interconnection networks)

Interconnection Networks
 What holds our parallel machines together - at the core of parallel computer
architecture
 Shares basic concept with LAN/WAN, but very different trade-offs due to very
different time scale/requirements

Different Scales of Networks

 On-Chip Networks – Interconnect within a single chip
 Devices are micro-architectural elements: caches, directories, processor cores
 Currently, designs with 10s of devices are common
 Ex: IBM Cell, Intel multicores, Tile processors
 Projected systems with 100s of devices on the horizon • Proximity: millimeters
 System-Area Networks

– Interconnects within one “machine”

 Interconnect in a multi-processor system
 Interconnect in a supercomputer
 Hundreds to thousands of devices interconnected
– IBM Blue Gene/L supercomputer (64K nodes, each with 2 processors)
 Maximum interconnect distance
– Fraction to tens of meters (typical)
– a few hundred meters (some)
 InfiniBand: 120 Gbps over a distance of 300m
 Local-Area Networks
– Interconnect autonomous computer systems
– Machine room or throughout a building or campus
– Hundreds of devices interconnected (1,000s with bridging)
– Maximum interconnect distance
 few metres to tens of kilometers
– Example (most popular): Ethernet, with 10 Gbps over 40Km
 Wide-Area Networks
– Interconnect systems distributed across the globe
– Internetworking support is required
– Many millions of devices interconnected
– Maximum interconnect distance
 many thousands of kilometers
 ICN Design Considerations
 Application requirements
– Number of terminals or ports to support
– Peak bandwidth of each terminal
– Average bandwidth of each terminal
– Latency requirements
– Message size distribution
– Expected traffic patterns
– Required quality of service
– Required reliability and availability
 – Job of an interconnection network is to transfer information from source node
to dest. node in support of network transactions that realize the application
– latency as small as possible
– as many concurrent transfers as possible
– cost as low as possible
– ICN Design Considerations
– Example requirements for a coherent processor-memory interconnect
– Processor ports 1-2048
– Memory ports 1-4096
– Peak BW 8 GB/s
– Average BW 400 MB/s
– Message Latency 100 ns
– Message size 64 or 576 bits
– Traffic pattern arbitrary
– Quality of service none
– Reliability no message loss
– Availability 0.999 to 0.99999
– Technology constraints
– Signaling rate
– Chip pin count (if off-chip networking)
– Area constraints (typically for on-chip networking)
– Chip cost – Circuit board cost (if backplane boards needed)
– Signals per circuit board
– Signals per cable
– Cable cost
– Cable length
– Channel and switch power constraints
An interconnection network in a parallel machine transfers information from any source node
to any desired destination node. This task should be completed with as small latency as
possible. It should allow a large number of such transfers to take place concurrently.
Moreover, it should be inexpensive as compared to the cost of the rest of the machine.
The network is composed of links and switches, which helps to send the information from the
source node to the destination node. A network is specified by its topology, routing
algorithm, switching strategy, and flow control mechanism.
Organizational Structure
Interconnection networks are composed of following three basic components −

 Links − A link is a cable of one or more optical fibers or electrical wires with a
connector at each end attached to a switch or network interface port. Through this, an
analog signal is transmitted from one end, received at the other to obtain the original
digital information stream.
 Switches − A switch is composed of a set of input and output ports, an internal “cross-
bar” connecting all input to all output, internal buffering, and control logic to effect
the input-output connection at each point in time. Generally, the number of input ports
is equal to the number of output ports.
 Network Interfaces − The network interface behaves quite differently than switch
nodes and may be connected via special links. The network interface formats the
packets and constructs the routing and control information. It may have input and
output buffering, compared to a switch. It may perform end-to-end error checking and
 flow control. Hence, its cost is influenced by its processing complexity, storage
capacity, and number of ports.
Interconnection Network
Interconnection networks are composed of switching elements. Topology is the pattern to
connect the individual switches to other elements, like processors, memories and other
switches. A network allows exchange of data between processors in the parallel system.

 Direct connection networks − Direct networks have point-to-point connections

between neighboring nodes. These networks are static, which means that the point-to-
point connections are fixed. Some examples of direct networks are rings, meshes and
cubes.
 Indirect connection networks − Indirect networks have no fixed neighbors. The
communication topology can be changed dynamically based on the application
demands. Indirect networks can be subdivided into three parts: bus networks,
multistage networks and crossbar switches.
 Bus networks − A bus network is composed of a number of bit lines onto which a
number of resources are attached. When busses use the same physical lines for data
and addresses, the data and the address lines are time multiplexed. When there are
multiple bus-masters attached to the bus, an arbiter is required.
 Multistage networks − A multistage network consists of multiple stages of switches. It
is composed of ‘axb’ switches which are connected using a particular interstage
connection pattern (ISC). Small 2x2 switch elements are a common choice for many
multistage networks. The number of stages determine the delay of the network. By
choosing different interstage connection patterns, various types of multistage network
can be created.
 Crossbar switches − A crossbar switch contains a matrix of simple switch elements
that can switch on and off to create or break a connection. Turning on a switch
element in the matrix, a connection between a processor and a memory can be made.
Crossbar switches are non-blocking, that is all communication permutations can be
performed without blocking.
Evaluating Design Trade-offs in Network Topology
If the main concern is the routing distance, then the dimension has to be maximized and a
hypercube made. In store-and-forward routing, assuming that the degree of the switch and the
number of links were not a significant cost factor, and the numbers of links or the switch
degree are the main costs, the dimension has to be minimized and a mesh built.
In worst case traffic pattern for each network, it is preferred to have high dimensional
networks where all the paths are short. In patterns where each node is communicating with
only one or two nearby neighbors, it is preferred to have low dimensional networks, since
only a few of the dimensions are actually used.
Routing
The routing algorithm of a network determines which of the possible paths from source to
destination is used as routes and how the route followed by each particular packet is
determined. Dimension order routing limits the set of legal paths so that there is exactly one
route from each source to each destination. The one obtained by first traveling the correct
distance in the high-order dimension, then the next dimension and so on.
Routing Mechanisms
Arithmetic, source-based port select, and table look-up are three mechanisms that high-speed
switches use to determine the output channel from information in the packet header. All of
these mechanisms are simpler than the kind of general routing computations implemented in
traditional LAN and WAN routers. In parallel computer networks, the switch needs to make
the routing decision for all its inputs in every cycle, so the mechanism needs to be simple and
fast.
Deterministic Routing
A routing algorithm is deterministic if the route taken by a message is determined exclusively
by its source and destination, and not by other traffic in the network. If a routing algorithm
only selects shortest paths toward the destination, it is minimal, otherwise it is non-minimal.
Deadlock Freedom
Deadlock can occur in a various situations. When two nodes attempt to send data to each
other and each begins sending before either receives, a ‘head-on’ deadlock may occur.
Another case of deadlock occurs, when there are multiple messages competing for resources
within the network.
The basic technique for proving a network is deadlock free, is to clear the dependencies that
can occur between channels as a result of messages moving through the networks and to
show that there are no cycles in the overall channel dependency graph; hence there is no
traffic patterns that can lead to a deadlock. The common way of doing this is to number the
channel resources such that all routes follow a particular increasing or decreasing sequences,
so that no dependency cycles arise.
Switch Design
Design of a network depends on the design of the switch and how the switches are wired
together. The degree of the switch, its internal routing mechanisms, and its internal buffering
decides what topologies can be supported and what routing algorithms can be implemented.
Like any other hardware component of a computer system, a network switch contains data
path, control, and storage.
Ports
The total number of pins is actually the total number of input and output ports times the
channel width. As the perimeter of the chip grows slowly compared to the area, switches tend
to be pin limited.
Internal Data path
The data path is the connectivity between each of the set of input ports and every output port.
It is generally referred to as the internal cross-bar. A non-blocking cross-bar is one where
each input port can be connected to a distinct output in any permutation simultaneously.
Channel Buffers
The organization of the buffer storage within the switch has an important impact on the
switch performance. Traditional routers and switches tend to have large SRAM or DRAM
buffers external to the switch fabric, while in VLSI switches the buffering is internal to the
switch and comes out of the same silicon budget as the data path and the control section. As
the chip size and density increases, more buffering is available and the network designer has
more options, but still the buffer real-estate comes at a prime choice and its organization is
important.
Flow Control
When multiple data flows in the network attempt to use the same shared network resources at
the same time, some action must be taken to control these flows. If we don’t want to lose any
data, some of the flows must be blocked while others proceed.

References

Suggested Book References

•      J.P. Hayes, “Computer Architecture and Organization”, Third Edition.
•      Mano, M., “Computer System Architecture”, Third Edition, Prentice Hall.
•      Stallings, W., “Computer Organization and Architecture”, Eighth Edition.
Other References
•     http://cva.stanford.edu/classes/ee382c/ee482b/scribes01/lect04/ourlect_final.pdf
•     https://www.tutorialspoint.com/parallel_computer_architecture/
parallel_computer_architecture_interconnection_network_design.htm