Program & network properties

• Introduction
• Condition of parallelism
o Data dependence and resource dependence
o Hardware and software dependence
o The role of compiler
• Program partitioning and scheduling
o Grain size and latency
o Grain packing and scheduling
• Program flow mechanism
• System interconnect architecture.
o Network properties and routing
o Static connection network
o Dynamic connection network
• Summary
• Keywords
2.0 Objective
In this lesson we will study about fundamental properties of programs how parallelism
can be introduced in program. We will study about the granularity, partitioning of
programs, and program flow mechanism and compilation support for parallelism.
Interconnection architecture both static and dynamic type will be discussed.
2.1 Introduction
The advantage of multiprocessors lays when parallelism in the program is popularly
exploited and implemented using multiple processors. Thus in order to implement the
parallelism we should understand the various conditions of parallelism.
What are various bottlenecks in implementing parallelism? Thus for full implementation
of parallelism there are three significant areas to be understood namely computation
models for parallel computing, interprocessor communication in parallel architecture and
system integration for incorporating parallel systems. Thus multiprocessor system poses a
number of problems that are not encountered in sequential processing such as designing a
parallel algorithm for the application, partitioning of the application into tasks,
coordinating communication and synchronization, and scheduling of the tasks onto the
machine.
2.2 Condition of parallelism
The ability to execute several program segments in parallel requires each segment to be
independent of the other segments. We use a dependence graph to describe the relations.
The nodes of a dependence graph correspond to the program statement (instructions), and
directed edges with different labels are used to represent the ordered relations among the
statements. The analysis of dependence graphs shows where opportunity exists for
parallelization and vectorization.
2.2.1 Data and resource Dependence
Data dependence: The ordering relationship between statements is indicated by the data
dependence. Five type of data dependence are defined below:
1. Flow dependence: A statement S2 is flow dependent on S1 if an execution path exists
from s1 to S2 and if at least one output (variables assigned) of S1feeds in as input

(operands to be used) to S2 also called RAW hazard and denoted as

2. Antidependence: Statement S2 is antidependent on the statement S1 if S2 follows S1 in
the program order and if the output of S2 overlaps the input to S1 also called RAW

hazard and denoted as

3. Output dependence : two statements are output dependent if they produce (write) the

same output variable. Also called WAW hazard and denoted as

4. I/O dependence: Read and write are I/O statements. I/O dependence occurs not
because the same variable is involved but because the same file referenced by both I/O
statement.
5. Unknown dependence: The dependence relation between two statements cannot be
determined in the following situations:
• The subscript of a variable is itself subscribed( indirect addressing)
• The subscript does not contain the loop index variable.
• A variable appears more than once with subscripts having different coefficients of
the loop variable.
• The subscript is non linear in the loop index variable.
Parallel execution of program segments which do not have total data independence can
produce non-deterministic results.
Consider the following fragment of any
program: S1 Load R1, A
S2 Add R2, R1
S3 Move R1, R3
S4 Store B, R1
• here the Forward dependency S1to S2, S3 to S4, S2 to S2
• Anti-dependency from S2to S3
• Output dependency S1 toS3

Figure 2.1 Dependence graph

Control Dependence: This refers to the situation where the order of the execution of
statements cannot be determined before run time. For example all condition statement,
where the flow of statement depends on the output. Different paths taken after a
conditional branch may depend on the data hence we need to eliminate this data
dependence among the instructions. This dependence also exists between operations
performed in successive iterations of looping procedure. Control dependence often
prohibits parallelism from being exploited.
Control-independent
example: for (i=0;i<n;i++) {
a[i] = c[i];
if (a[i] < 0) a[i] = 1;
}
Control-dependent example:
for (i=1;i<n;i++) {
if (a[i-1] < 0) a[i] = 1;
}
Control dependence also avoids parallelism to being exploited. Compilers are used to
eliminate this control dependence and exploit the parallelism.
Resource dependence:
Data and control dependencies are based on the independence of the work to be done.
Resource independence is concerned with conflicts in using shared resources, such as
registers, integer and floating point ALUs, etc. ALU conflicts are called ALU
dependence. Memory (storage) conflicts are called storage dependence.
Bernstein’s Conditions - 1
Bernstein’s conditions are a set of conditions which must exist if two processes can
execute in parallel.
Notation
Ii is the set of all input variables for a process Pi . Ii is also called the read set or domain
of Pi. Oi is the set of all output variables for a process Pi .Oi is also called write set
If P1 and P2 can execute in parallel (which is written as P1 || P2), then:

Bernstein’s Conditions - 2

33
In terms of data dependencies, Bernstein’s conditions imply that two processes can
execute in parallel if they are flow-independent, antiindependent, and output-
independent. The parallelism relation || is commutative (Pi || Pj implies Pj || Pi ), but not
transitive (Pi || Pj and Pj || Pk does not imply Pi || Pk ) . Therefore, || is not an equivalence
relation. Intersection of the input sets is allowed.
2.2.2 Hardware and software parallelism
Hardware parallelism is defined by machine architecture and hardware multiplicity i.e.,
functional parallelism times the processor parallelism .It can be characterized by the
number of instructions that can be issued per machine cycle. If a processor issues k
instructions per machine cycle, it is called a k-issue processor. Conventional processors
are one-issue machines. This provide the user the information about peak attainable
performance. Examples. Intel i960CA is a three-issue processor (arithmetic, memory
access, branch). IBM RS -6000 is a four-issue processor (arithmetic, floating-point,
memory access, branch).A machine with n k-issue processors should be able to handle a
maximum of nk threads simultaneously.
Software Parallelism
Software parallelism is defined by the control and data dependence of programs, and is
revealed in the program’s flow graph i.e., it is defined by dependencies with in the code
and is a function of algorithm, programming style, and compiler optimization.
2.2.3 The Role of Compilers
Compilers used to exploit hardware features to improve performance. Interaction
between compiler and architecture design is a necessity in modern computer
development. It is not necessarily the case that more software parallelism will improve
performance in conventional scalar processors. The hardware and compiler should be
designed at the same time.
2.3Program Partitioning & Scheduling
2.3.1 Grain size and latency
The size of the parts or pieces of a program that can be considered for parallel execution
can vary. The sizes are roughly classified using the term “granule size,” or simply
“granularity.” The simplest measure, for example, is the number of instructions in a

34
program part. Grain sizes are usually described as fine, medium or coarse, depending on
the level of parallelism involved.
Latency
Latency is the time required for communication between different subsystems in a
computer. Memory latency, for example, is the time required by a processor to access
memory. Synchronization latency is the time required for two processes to synchronize
their execution. Computational granularity and communication latency are closely
related. Latency and grain size are interrelated and some general observation are
• As grain size decreases, potential parallelism increases, and overhead also
increases.
• Overhead is the cost of parallelizing a task. The principle overhead is
communication latency.
• As grain size is reduced, there are fewer operations between communication, and
hence the impact of latency increases.
• Surface to volume: inter to intra-node comm.
Levels of Parallelism
Instruction Level Parallelism
This fine-grained, or smallest granularity level typically involves less than 20 instructions
per grain. The number of candidates for parallel execution varies from 2 to thousands,
with about five instructions or statements (on the average) being the average level of
parallelism.
Advantages:
There are usually many candidates for parallel execution
Compilers can usually do a reasonable job of finding this parallelism
Loop-level Parallelism
Typical loop has less than 500 instructions. If a loop operation is independent between
iterations, it can be handled by a pipeline, or by a SIMD machine. Most optimized
program construct to execute on a parallel or vector machine. Some loops (e.g. recursive)
are difficult to handle. Loop-level parallelism is still considered fine grain computation.
Procedure-level Parallelism

35
Medium-sized grain; usually less than 2000 instructions. Detection of parallelism is more
difficult than with smaller grains; interprocedural dependence analysis is difficult and
history-sensitive. Communication requirement less than instruction level SPMD (single
procedure multiple data) is a special case Multitasking belongs to this level.
Subprogram-level Parallelism
Job step level; grain typically has thousands of instructions; medium- or coarse-grain
level. Job steps can overlap across different jobs. Multiprograming conducted at this level
No compilers available to exploit medium- or coarse-grain parallelism at present.
Job or Program-Level Parallelism
Corresponds to execution of essentially independent jobs or programs on a parallel
computer. This is practical for a machine with a small number of powerful processors,
but impractical for a machine with a large number of simple processors (since each
processor would take too long to process a single job).
Communication Latency
Balancing granularity and latency can yield better performance. Various latencies
attributed to machine architecture, technology, and communication patterns used.
Latency imposes a limiting factor on machine scalability. Ex. Memory latency increases
as memory capacity increases, limiting the amount of memory that can be used with a
given tolerance for communication latency.
Interprocessor Communication Latency
• Needs to be minimized by system designer
• Affected by signal delays and communication patterns Ex. n communicating tasks
may require n (n - 1)/2 communication links, and the complexity grows
quadratically, effectively limiting the number of processors in the system.
Communication Patterns
• Determined by algorithms used and architectural support provided
• Patterns include permutations broadcast multicast conference
• Tradeoffs often exist between granularity of parallelism and communication
demand.
2.3.2 Grain Packing and Scheduling
Two questions:

36
How can I partition a program into parallel “pieces” to yield the shortest execution time?
What is the optimal size of parallel grains?
There is an obvious tradeoff between the time spent scheduling and synchronizing
parallel grains and the speedup obtained by parallel execution.
One approach to the problem is called “grain packing.”
Program Graphs and Packing
A program graph is similar to a dependence graph Nodes = { (n,s) }, where n = node
name, s = size (larger s = larger grain size).
Edges = { (v,d) }, where v = variable being “communicated,” and d = communication
delay.
Packing two (or more) nodes produces a node with a larger grain size and possibly more
edges to other nodes. Packing is done to eliminate unnecessary communication delays or
reduce overall scheduling overhead.
Scheduling
A schedule is a mapping of nodes to processors and start times such that communication
delay requirements are observed, and no two nodes are executing on the same processor
at the same time. Some general scheduling goals
• Schedule all fine-grain activities in a node to the same processor to minimize
communication delays.
• Select grain sizes for packing to achieve better schedules for a particular parallel
machine.
Node Duplication
Grain packing may potentially eliminate interprocessor communication, but it may not
always produce a shorter schedule. By duplicating nodes (that is, executing some
instructions on multiple processors), we may eliminate some interprocessor
communication, and thus produce a shorter schedule.
Program partitioning and scheduling
Scheduling and allocation is a highly important issue since an inappropriate scheduling of
tasks can fail to exploit the true potential of the system and can offset the gain from
parallelization. In this paper we focus on the scheduling aspect. The objective of
scheduling is to minimize the completion time of a parallel application by properly

37
allocating the tasks to the processors. In a broad sense, the scheduling problem exists in
two forms: static and dynamic. In static scheduling, which is usually done at compile
time, the characteristics of a parallel program (such as task processing times,
communication, data dependencies, and synchronization requirements) are known before
program execution
A parallel program, therefore, can be represented by a node- and edge-weighted directed
acyclic graph (DAG), in which the node weights represent task processing times and the
edge weights represent data dependencies as well as the communication times between
tasks. In dynamic scheduling only, a few assumptions about the parallel program can be
made before execution, and thus, scheduling decisions have to be made on-the-fly. The
goal of a dynamic scheduling algorithm as such includes not only the minimization of the
program completion time but also the minimization of the scheduling overhead which
constitutes a significant portion of the cost paid for running the scheduler. In general
dynamic scheduling is an NP hard problem.
2.5 System interconnects architecture.
Various types of interconnection networks have been suggested for SIMD computers.
These are basically classified have been classified on network topologies into two
categories namely
Static Networks
Dynamic Networks
Static versus Dynamic Networks
The topological structure of an SIMD array processor is mainly characterized by the data
routing network used in interconnecting the processing elements.
The topological structure of an SIMD array processor is mainly characterized by the data
routing network used in the interconnecting the processing elements. To execute the
communication the routing function f is executed and via the interconnection network the
PEi copies the content of its Ri register into the Rf(i) register of PEf(i). The f(i) the
processor identified by the mapping function f. The data routing operation occurs in all
active PEs simultaneously.
2.5.1 Network properties and routing
The goals of an interconnection network are to provide low-latency high data transfer rate
wide communication bandwidth. Analysis includes latency bisection bandwidth data-
routing functions scalability of parallel architecture
These Network usually represented by a graph with a finite number of nodes linked by
directed or undirected edges.
Number of nodes in graph = network size .
Number of edges (links or channels) incident on a node = node degree d (also note in and
out degrees when edges are directed).

Node degree reflects number of I/O ports associated with a node, and should ideally be
small and constant.
Network is symmetric if the topology is the same looking from any node; these are easier
to implement or to program.
Diameter : The maximum distance between any two processors in the network or in other
words we can say Diameter, is the maximum number of (routing) processors through
which a message must pass on its way from source to reach destination. Thus diameter
measures the maximum delay for transmitting a message from one processor to another
as it determines communication time hence smaller the diameter better will be the
network topology.
Connectivity: How many paths are possible between any two processors i.e., the
multiplicity of paths between two processors. Higher connectivity is desirable as it
minimizes contention.
Arch connectivity of the network: the minimum number of arcs that must be removed for
the network to break it into two disconnected networks. The arch connectivity of various
network are as follows
• 1 for linear arrays and binary trees
• 2 for rings and 2-d meshes
• 4 for 2-d torus
• d for d-dimensional hypercubes
Larger the arch connectivity lesser the conjunctions and better will be network topology.
Channel width : The channel width is the number of bits that can communicated
simultaneously by a interconnection bus connecting two processors
Bisection Width and Bandwidth: In order divide the network into equal halves we require
the remove some communication links. The minimum number of such communication
links that have to be removed are called the Bisection Width. Bisection width basically
provide us the information about the largest number of messages which can be sent
simultaneously (without needing to use the same wire or routing processor at the same
time and so delaying one another), no matter which processors are sending to which other
processors. Thus larger the bisection width is the better the network topology is
considered. Bisection Bandwidth is the minimum volume of communication allowed
between two halves of the network with equal numbers of processors This is important
for the networks with weighted arcs where the weights correspond to the link width i.e.,
(how much data it can transfer). The Larger bisection width the better network topology
is considered.
Cost the cost of networking can be estimated on variety of criteria where we consider the
the number of communication links or wires used to design the network as the basis of
cost estimation. Smaller the better the cost
Data Routing Functions: A data routing network is used for inter –PE data exchange. It
can be static as in case of hypercube routing network or dynamic such as multistage
network. Various type of data routing functions are Shifting, Rotating, Permutation (one
to one), Broadcast (one to all), Multicast (many to many), Personalized broadcast (one to
many), Shuffle, Exchange Etc.
Permutations
Given n objects, there are n ! ways in which they can be reordered (one of which is no
reordering). A permutation can be specified by giving the rule for reordering a group of
objects. Permutations can be implemented using crossbar switches, multistage networks,
shifting, and broadcast operations. The time required to perform permutations of the
connections between nodes often dominates the network performance when n is large.
Perfect Shuffle and Exchange
Stone suggested the special permutation that entries according to the mapping of the k-bit
binary number a b … k to b c … k a (that is, shifting 1 bit to the left and wrapping it
around to the least significant bit position). The inverse perfect shuffle reverses the effect
of the perfect shuffle.
Hypercube Routing Functions
If the vertices of a n-dimensional cube are labeled with n-bit numbers so that only one bit
differs between each pair of adjacent vertices, then n routing functions are defined by the
bits in the node (vertex) address. For example, with a 3-dimensional cube, we can easily
identify routing functions that exchange data between nodes with addresses that differ in
the least significant, most significant, or middle bit.
Factors Affecting Performance
Functionality – how the network supports data routing, interrupt handling,
synchronization, request/message combining, and coherence
Network latency – worst-case time for a unit message to be transferred
Bandwidth – maximum data rate
Hardware complexity – implementation costs for wire, logic, switches, connectors, etc.
Scalability – how easily does the scheme adapt to an increasing number of processors,
memories, etc.?
2.5.2 Static connection Networks
In static network the interconnection network is fixed and permanent interconnection
path between two processing elements and data communication has to follow a fixed
route to reach the destination processing element. Thus it Consist of a number of point-
to-point links. Topologies in the static networks can be classified according to the
dimension required for layout i.e., it can be 1-D, 2-D, 3-D or hypercube.
One dimensional topologies include Linear array as shown in figure 2.2 (a) used in some
pipeline architecture.
Various 2-D topologies are
• The ring (figure 2.2(b))
• Star (figure 2.2(c))
• Tree (figure 2.2(d))
• Mesh (figure 2.2(e))
• Systolic Array (figure
2.2(f)) 3-D topologies include
• Completely connected chordal ring (figure 2.2(g))
• Chordal ring (figure 2.2(h))
• 3 cube (figure 2.2(i))
Figure 2.2 Static interconnection network topologies.
Torus architecture is also one of popular network topology it is extension of the mesh by
having wraparound connections Figure below is a 2D Torus This architecture of torus is a
symmetric topology unlike mesh which is not. The wraparound connections reduce the
torus diameter and at the same time restore the symmetry. It can be
o 1-D torus
2-D torus
3-D torus
The torus topology is used in Cray T3E
Figure 2.3 Torus technology
We can have further higher dimension circuits for example 3-cube connected cycle. A D-
dimension W-wide hypercube contains W nodes in each dimension and there is a
connection to a node in each dimension. The mesh and the cube architecture are actually
2-D and 3-D hypercube respectively. The below figure we have hypercube with
dimension 4.

Figure 2.4 4-D hypercube.

2.5.3 Dynamic connection Networks
The dynamic networks are those networks where the route through which data move
from one PE to another is established at the time communication has to be performed.
Usually all processing elements are equidistant and an interconnection path is established
when two processing element want to communicate by use of switches. Such systems are
more difficult to expand as compared to static network. Examples: Bus-based, Crossbar,
Multistage Networks. Here the Routing is done by comparing the bit-level representation
of source and destination addresses. If there is a match goes to next stage via pass-
through else in case of it mismatch goes via cross-over using the switch.
There are two classes of dynamic networks namely
• single stage network
 • multi stage
2.5.3.1 Single Stage Networks
A single stage switching network with N input selectors (IS) and N output selectors (OS).
Here at each network stage there is a 1- to-D demultiplexer corresponding to each IS such
that 1<D<N and each OS is an M-to-1 multiplexer such that 1<M<=N. Cross bar network
is a single stage network with D=M=N. In order to establish a desired connecting path
different path control signals will be applied to all IS and OS selectors. The single stage
network is also called as recirculating network as in this network connection the single
data items may have to recirculate several time through the single stage before reaching
their final destinations. The number of recirculation depends on the connectivity in the
single stage network. In general higher the hardware connectivity the lesser is the number
of recirculation. In cross bar network only one circulation is needed to establish the
connection path. The cost of completed connected cross bar network is O(N2) which is
very high as compared to other most recirculating networks which have cost O(N log N)
or lower hence are more cost effective for large value of N.
2.5.3.2 Multistage Networks
Many stages of interconnected switches form a multistage SIMD network. It is basicaaly
consist of three characteristic features
• The switch box,
• The network topology
• The control structure
Many stages of interconnected switches form a multistage SIMD networks. Eachbox is
essentially an interchange device with two inputs and two outputs. The four possible
states of a switch box are which are shown in figure 3.6
• Straight
• Exchange
• Upper Broadcast
• Lower broadcast.
A two function switch can assume only two possible state namely state or exchange
states. However a four function switch box can be any of four possible states. A
multistage network is capable of connecting any input terminal to any output terminal.
Multi-stage networks are basically constructed by so called shuffle-exchange switching
element, which is basically a 2 x 2 crossbar. Multiple layers of these elements are
connected and form the network.
Figure 2.5 A two-by-two switching box and its four interconnection states
A multistage network is capable of connecting an arbitrary input terminal to an arbitrary
n
output terminal. Generally it is consist of n stages where N = 2 is the number of input
and output lines. And each stage use N/2 switch boxes. The interconnection patterns from
one stage to another stage is determined by network topology. Each stage is connected to
the next stage by at least N paths. The total wait time is proportional to the number stages
i.e., n and the total cost depends on the total number of switches used and that is Nlog2N.
The control structure can be individual stage control i.e., the same control signal is used
to set all switch boxes in the same stages thus we need n control signal. The second
control structure is individual box control where a separate control signal is used to set
the state of each switch box. This provide flexibility at the same time require n2/2 control
signal which increases the complexity of the control circuit. In between path is use of
partial stage control.
Examples of Multistage Networks
Banyan
Baseline
Cube
Delta
Flip
Indirect cube
Omega
Multistage network can be of two types
• One side networks : also called full switch having input output port on the same
side
• Two sided multistage network : which have an input side and an output side. It
can be further divided into three class
o Blocking: In Blocking networks, simultaneous connections of more than
one terminal pair may result conflicts in the use of network
communication links. Examples of blocking network are the Data
Manipulator, Flip, N cube, omega, baseline. All multistage networks that
are based on shuffle-exchange elements, are based on the concept of
blocking network because not all possible here to make the input-output
connections at the same time as one path might block another. The figure
2.6 (a) show an omega network.
o Rearrangeable : In rearrangeable network, a network can perform all
possible connections between inputs and outputs by rearranging its
existing connections so that a connection path for a new input-output pair
can always be established. An example of this network topology is Benes
Network ( see figure 2.6 (b) showing a 8** Benes network)which support
synchronous data permutation and a synchronous interprocessor
communication.
o Non blocking : A non –blocking network is the network which can handle
all possible connections without blocking. There two possible cases first
one is the Clos network ( see figure 2.6(c)) where a one to one connection
 is made between input and output. Another case of one to many
connections can be obtained by using crossbars instead of the shuffle-
exchange elements. The cross bar switch network can connect every input
port to a free output port without blocking.

Figure 2.6 Several Multistage Interconnection

Networks Mesh-Connected Illiac Networks
A single stage recirculating network has been implemented in the ILLiac –IV array with
N= 64 PEs. Here in mesh network nodes are arranged as a q-dimensional lattice. The
neighboring nodes are only allowed to communicate the data in one step i.e., each PEi is
allowed to send the data to any one of PE(i+1) , PE (i-1), Pe(i+r) and PE(i-r) where r=
square root N( in case of Iliac r=8). In a periodic mesh, nodes on the edge of the mesh
have wrap-around connections to nodes on the other side this is also called a toroidal
mesh.
Mesh Metrics
For a q-dimensional non-periodic lattice with kq nodes:
• Network connectivity = q
• Network diameter = q(k-1)
• Network narrowness = k/2
• Bisection width = kq-1
• Expansion Increment = kq-1
• Edges per node = 2q
Thus we observe the output of IS k is connected to inputs of OSj where j = k-1,K+1,k-
r,k+r as shown in figure below.

Figure2.7 routing function of mesh Topology

Similarly the OSj gets input from ISk for K= j-1, j+1,j-r,j+r. The topology is formerly
described by the four routing functions:
• R+1(i)= (i+1) mod N => (0,1,2…,14,15)
• R-1(i)= (i-1) mod N => (15,14,…,2,1,0)
• R+r(i)= (i+r) mod N => (0,4,8,12)(1,5,9,13)(2,6,10,14)(3,7,11,15)
• R-r(i)= (i-r) mod N => (15,11,7,3)(14,10,6,2)(13,9,5,1)(12,8,4,0)
The figure given below show how each PEi is connected to its four nearest neighbors in
the mesh network. It is same as that used for IILiac –IV except that w had reduced it for
N=16 and r=4. The index are calculated as module N.
Figure 2.8 Mesh Connections
Thus the permutation cycle according to routing function will be as follows:
Horizontally, all PEs of all rows form a linear circular list as governed by the following
two permutations, each with a single cycle of order N. The permutation cycles (a b c) (d
e) stands for permutation a->b, b->c, c->a and d->e, e->d in a circular fashion with each
pair of parentheses.
R+1 = (0 1 2 ….N-1)
R–1 = (N-1 ….. 2 1 0).
Similarly we have vertical permutation also and now by combining the two permutation
each with four cycles of order four each the shift distance for example for a network of N
= 16 and r = square root(16) = 4, is given as follows:
R +4 = (0 4 8 12)(1 5 9 13)(2 6 10 14)(3 7 11 15)
R –4 = (12 8 4 0)(13 9 5 1)(14 10 6 2)(15 11 7 3)

Figure 4.9 Mesh Redrawn

Each PEi is directly connected to its four neighbors in the mesh network. The graph
shows that in one step a PE can reach to four PEs, seven PEs in two step and eleven PEs
in three steps. In general it takes I steps ( recirculations) to route data from PEi to another
PEj for a network of size N where I is upper –bound given by
I<= square root(N) -1
Thus in above example for N=16 it will require at most 3 steps to route data from one PE
to another PE and for Illiac –IV network with 64 PE need maximum of 7 steps for routing
data from one PE to Another.
Cube Interconnection Networks
The cube network can be implemented as either a recirculating network or as a multistage
network for SIMD machine. It can be 1-D i.e., a single line with two pE each at end of a
line, a square with four PEs at the corner in case of 2-D, a cube for 3-D and hypercube in
4-D. in case of n-dimension hypercube each processor connects to 2n neighbors. This can
be also visualized as the unit (hyper) cube embedded in d-dimensional Euclidean space,
with one corner at 0 and lying in the positive orthant. The processors can be thought of as
lying at the corners of the cube, with their (x1,x2,...,xd) coordinates identical to their
processor numbers, and connected to their nearest neighbors on the cube. The popular
examples where cube topology is used are : iPSC, nCUBE, SGI O2K.
Vertical lines connect vertices (PEs) whose address differ in the most significant
bit position. Vertices at both ends of the diagonal lines differ in the middle bit position.
Horizontal lines differ in the least significant bit position. The unit – cube concept can be
extended to an n- dimensional unit space called an n cube with n bits per vertex. A cube
network for an SIMD machine with N PEs corresponds to an n cube where n = log2 N.
We use binary sequence to represent the vertex (PE) address of the cube. Two processors
are neighbors if and only if their binary address differs only in one digit place
For an n-dimensional cube network of N PEs is specified by the following n routing
functions
Ci (An-1 …. A1 A0)= An-1…Ai+1 A’i Ai-1……A0 for i =0,1,2,…,n-1
A n- dimension cube each PE located at the corner is directly connected to n neighbors.
The addresses of neighboring PE differ in exactly one bit position. Pease’s binary n cube
the flip flop network used in staran and programmable switching network proposed for
Phoenix are examples of cube networks.
In a recirculating cube network each ISa for 0<=A+< N-1 is connected to n OSs whose
addresses are An-1…Ai+1 A’i Ai-1……A0 . When the PE addresses are considered as
the corners of an m-dimensional cube this network connects each PE to its m neighbors.
The interconnections of the PEs corresponding to the three routing function C0, C1 and
C2 are shown separately in below figure.
• Examples
Figure 2.10 The recirculating Network
It takes n<= log2 N steps to rotate data from any PE to another.
Example: N=8 => n=3

Figure 2.11 Possible routing in multistage Cube network for N = 8

Figure 2.12 A multistage Cube network for N = 8

The same set of cube routing functions i.e., C0,C1, C2 can also be implemented by three
stage network. Two functions switch box is used which can provide either straight and
exchange routing is used for constructing multistage cube networks. The stages are
numbered as 0 at input end and increased to n-1 at the output stage i.e., the stage I
implements the Ci routing function or we can say at ith stage connect the input line to the
output line that differ from it only at the ith bit position.
This connection was used in the early series of Intel Hypercubes, and in the CM-2.
Suppose there are 8 process ring elements so 3 bits are required for there address. and
that processor 000 is the root. The children of the root are gotten by toggling the first
address bit, and so are 000 and 100 (so 000 doubles as root and left child). The children
of the children are gotten by toggling the next address bit, and so are 000, 010, 100 and
110. Note that each node also plays the role of the left child. Finally, the leaves are gotten
by toggling the third bit. Having one child identified with the parent causes no problems
as long as algorithms use just one row of the tree at a time. Here is a picture.

Figure 2.13 A tree embedded in 3-D hypercube

Shuffle-Exchange Omega Networks
k
A shuffle-exchange network consists of n=2 nodes and it is based on two routing
functions shuffle (S) and exchange (E). Let A= An-1…A1A0be the address of a PE than
a shuffle function is given by:
S(A)=S(An-1…A1A0)=A.n-2…A1A0An-1, 0<A<1
The cyclic shifting of the bits in A to the left for one bit osition is performed by the S
function. Which is effectively like shuffling the bottom half of a card deck into the top
half as shown in figure below.
Figure 2.14 Perfect shuffle and inverse perfect shuffle
There are two type of shuffle the perfect shuffle cuts the deck into two halves from the
centre and intermix them evenly. Perfect shuffle provide the routing connections of node
i with node 2i mod(n-1), except for node n-1 which is connected to itself. The inverse
perfect shuffle does the opposite to restore the original order it is denoted as exchange
routing function E and is defined as :
E(An-1…A1A0)= (An-1…A1A0’)
This obtained by complementing the least significant digit means data exchange
between two PEs with adjacent addresses. The E(A) is same as the cube routing function
as described earlier. Exchange routing function connects nodes whose numbers differ in
their lowest bit.
The shuffle exchange function can be implemented as either a recirculating network or
multistage network. The implementation of shuffle and exchange network through
recirculating network is shown below. Use of shuffle and exchange topology for parallel
processing was proposed by Stone. It is used for solving many parallel algorithms
efficiently. The example where it is used include FFT (fast Fourier transform), sorting,
matrix transposition , polynomial evaluations etc.
Figure2.15 shuffle and exchange recirculating network for N=8
The shuffle –exchange function have been implemented as multistage Omega network by
LAwrie. An N by N omega network, consists of n identical stages. Between two adjacent
column there is a perfect shuffle interconnection. Thus after each stage there is a N/2
four-function interchange boxes under independent box control. The four functions are
namely straight exchange upper broadcast and lower broadcast. The shuffle connects
output P n-l...Pl P0 of stage i to input P n-2...PlP0Pn-l of stage i-1. Each interchange box
in an omega network is controlled by the n-bit destination tags associated with the data
on its input lines.

Figure 2.16
The diameter is m=log_2 p, since all message must traverse m stages. The bisection
width is p. This network was used in the IBM RP3, BBN Butterfly, and NYU
Ultracomputer. If we compare the omega network with cube network we find Omega
network can perform one to many connections while n-cube cannot. However as far as
bisections connections n-cube and Omega network they perform more or less same.