CSL718

CSL718 :: Multiprocessors
Multiprocessors

Interconnection Mechanisms
Performance Models
20th April, 2006

Anshul Kumar, CSE IITD

Connecting
Connecting Processors
Processors and
and Memories
Memories
• Shared Buses
• Interconnection Networks
– Static Networks
– Dynamic Networks
M M M M M M M M
P P P P P P P P
Interconnection Network Interconnection Network

M M M M M M
Global Interconnection Network
M M M
Anshul Kumar, CSE IITD slide 2
 Shared
Shared Bus
Bus
each processor sees this picture:
processing

bus access
bus transaction time
bus utilization   
processing time  bus transaction time
prob of a processor using the bus = 
prob of a processor not using the bus = 1 – 
prob of none of the n processors using the bus = (1 – )n
prob of at least one processor using the bus = 1 – (1 – )n
achieved BW on a relative scale = 1 – (1 – )n
required BW = n  available BW = 1
Anshul Kumar, CSE IITD slide 3
Effect
Effect of
of re-submitted
re-submitted requests
requests
1-  + PA  (1-PA ) 1-PA

A W
prob = qA PA prob = qW
PA PA
qA   qW 1  q A
PA   1  PA    PA 1   
actual request rate a   q A  q w
 PA PA 
 1  
  PA 1      PA 1      PA 1   
a 
BW n a 1  1  a 
n
also PA   a
 a
 1   

Anshul Kumar, CSE IITD slide 4
 Shared Bus : BW per proc

0.600
0.500 n=2
0.400 n=3
0.300 n=4
BW achieved

0.200 n=2
0.100 n=3
0.000 n=4
-0.100 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
BW required (req probability)
 Shared Bus : utilization

1.200
1.000 n=2
0.800 n=3
0.600 n=4
utilization

0.400 n=2
0.200 n=3
0.000 n=4
-0.200 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
req probability
 Waiting
Waiting time
time
waiting time  i Tbus
if request is rejected i times and accepted on (i  1 )th attempt
probability of this (1  PA ) i PA

Expected value of waiting time Tw  i Tbus (1  PA ) i PA
i 1

1  PA
Tbus PA  i (1  PA ) Tbus PA 
i

i 1 1  (1  PA )
2

1  PA   a
 Tbus  Tbus
PA a

Anshul Kumar, CSE IITD slide 7

 Switched
Switched Networks
Networks
BUS Switched Network
• Shared media • Switched paths
• Lower Cost • Higher cost
• Lower throughput • Higher throughput
• Scalability poor • Scalability better

Anshul Kumar, CSE IITD slide 8

 Interconnection
Interconnection Networks
Networks
• Topology : who is connected to whom
• Direct / Indirect : where is switching done
• Static / Dynamic : when is switching done
• Circuit switching / packet switching : how are
connections established
• Store & forward / worm hole routing : how is the path
determined
• Centralized / distributed : how is switching controlled
• Synchronous/asynchronous : mode of operation

Anshul Kumar, CSE IITD slide 9

 Direct
Direct and
and Indirect
Indirect Networks
Networks
P link
node
M
P P
node
M node
M P link
S S node

SWITCH
link M
link link
link P link
S S node
M
node
M node
M
P P P link
node
M
DIRECT
INDIRECT
Anshul Kumar, CSE IITD slide 10
 Static
Static and
and Dynamic
Dynamic Networks
Networks
• Static Networks
– fixed point to point connections
– usually direct
– each node pair may not have a direct connection
– routing through nodes
• Dynamic Networks
– connections established as per need
– usually indirect
– path can be established between any pair of nodes
– routing through switches
Anshul Kumar, CSE IITD slide 11
 Static
Static Network
Network Topologies
Topologies
Non-uniform connectivity

Linear 2D-Mesh

Tree
Star

Anshul Kumar, CSE IITD slide 12

Static
Static Networks
Networks Topologies-
Topologies- contd.
contd.
Uniform connectivity

Ring

Torus
Fully Connected

Anshul Kumar, CSE IITD slide 13

 Illiac
Illiac IV
IV Mesh
Mesh Network
Network

0
0 1 2 8 1

3 4 5 7 2

6 7 8
6 3

neighbors of node r : 5 4
(r  1) mod 9 and
(r  3) mod 9 Chordal Ring
Anshul Kumar, CSE IITD slide 14
 Fat
Fat Tree
Tree Network
Network

Anshul Kumar, CSE IITD slide 15

 Dynamic
Dynamic Networks
Networks
kk building block for multi-stage
cross -bar
switch
dynamic networks

simplest 22
cross-bar switch

straight exchange upper lower

broadcast broadcast
Anshul Kumar, CSE IITD slide 16
 Baseline
Baseline Network
Network

000 000
001 001
010 010
011 011
100 100
101 101
110 110
111 111

blocking can occur

Anshul Kumar, CSE IITD slide 17
 Benes
Benes Network
Network

non-blocking
Anshul Kumar, CSE IITD slide 18
 Switching
Switching Mechanism
Mechanism
• Circuit Switching (connection oriented
communication)
– A circuit is established between the source and
the destination
• Packet Switching (connectionless
communication)
– Information is divided into packets and each
packet is sent independently from node to node

Anshul Kumar, CSE IITD slide 19

 Routing
Routing in
in Networks
Networks
node
incoming outgoing
message message

header payload/data
store & forward
routing H l
BW BW
 H l 
latency n   
 BW BW 
time
worm hole  H  l
latency n  
routing  BW  BW

Anshul Kumar, CSE IITD slide 20

Routing
Routing in
in presence
presence of
of congestion
congestion
• Worm hole routing
– When message header is blocked, many links
get blocked with the message
• Solution: cut-through routing
– When message header is blocked, tail is
allowed to move, compressing the message into
a single node

Anshul Kumar, CSE IITD slide 21

 Routing
Routing Options
Options
• Deterministic routing: always same path
followed
• Adaptive routing: best path selected to
minimize congestion

• Source based routing: message specifies

path to destination
• Destination based routing: message
specifies only destination address
Anshul Kumar, CSE IITD slide 22
 Some
Some Performance
Performance Parameters
Parameters

overhead Tx time=bytes/BW
sender
time of flight
Tx time=bytes/BW overhead
receiver

transport latency
total latency

time
Anshul Kumar, CSE IITD slide 23
 Other
Other Parameters
Parameters
• Throughput  Bandwidth (no credit for header)
• Bisection bandwidth = BW across a bisection
• Node degree
• Network Diameter
• Cost
• Fault Tolerance

Anshul Kumar, CSE IITD slide 24

 Multidimensional
Multidimensional Grid/Mesh
Grid/Mesh

k-ary n-cube Size

=k  k  ….  k (n times)
=kn
Diameter
= (k-1)  n without end around
connections
= k  n /2 with end around
connections
for (Binary) Hypercube : k = 2
Anshul Kumar, CSE IITD slide 25
 Grid/Mesh
Grid/Mesh Performance
Performance -- 11

Message arrival rate 

 r n k d

n is number of dimensions
k d is av. no. of hops
along one dimension
kd
r is prob of message req in a cycle

Anshul Kumar, CSE IITD slide 26

 Grid/Mesh
Grid/Mesh Performance
Performance -- 22
2n
Service rate  
Ts
 r k d Ts
Server Occupancy   
 2
Probability of request

along a link p 
2n

Anshul Kumar, CSE IITD slide 27

 Grid/Mesh
Grid/Mesh Performance
Performance -- 33

k-ary n-cube

Tw waiting time at a node

use M B / D / 1 open queue model

1   (   p)   p
Tw   
  Ts
  2(1   )  2(1   )

Anshul Kumar, CSE IITD slide 28

 Switch
Switch Performance
Performance
Let prob of request at an input
km
port during one service cycle r cross -bar
Here it is assumed that each mesage switch

(or packet) requires same service time T

prob of i simultaneous requests on k ports q(i)  k C i r i (1  r ) k  i
expected no. of requests accepted out of i requests E(i)
fraction of address patterns including a specific output port
num of output ports
i
m  (m  1) i   m  1 
i

 i
m  1    m
m   m  
Anshul Kumar, CSE IITD slide 29
 Switch
Switch Performance
Performance –– contd.
contd.
k
Expected BW (on relative scale)  E (i )q (i )
i 0

k   m  1
i
 k i
 1     m C i r (1  r ) k i

i 0   m  
k   m k
1  k i
i

 m Ci r (1  r )    
k i k i
  m Ci r (1  r )
k i

i 0 i 0   m  

i
k
 m 1 
k
m k Ci r i (1  r ) k  i  m k Ci  r  (1  r ) k  i
i 0 i 0  m 
k k
 m 1   r
m  m r  (1  r )  m  m 1  
 m   m
Anshul Kumar, CSE IITD slide 30
 Switch
Switch Performance
Performance –– contd.
contd.
Requested bandwidth r k
Expected BW (if there were no output port conflicts) m
k
 r 
Expected BW (because of output port conflicts) m  m 1  
 m 
this is less than m as well as r k (assuming that r  1)
BW
prob of acceptance of requests PA 
rk
We now consider effect of request re - submission due to conflicts.
We need to compute revised request rate due to
re - submission and also compute delays because of waiting.

Anshul Kumar, CSE IITD slide 31

Effect
Effect of
of re-submitted
re-submitted requests
requests
actual request rate r ' r q A  qw
(using Markov graph with states q A and qw )
r
 r' 
r  PA 1  r 
k
 r'  BW
BW m  m 1   PA 
 m r' k
1  PA
waiting time  T
PA
H l
T cycle time 
BW of link
Anshul Kumar, CSE IITD slide 32
 Effect
Effect of
of buffering
buffering
There are two possibilities
• Buffering before switching (k buffers, one
at each input port)
• Buffering after switching (m buffers, one at
each output port)

Anshul Kumar, CSE IITD slide 33

 Switch
Switch with
with input
input buffers
buffers
Rate of messages at input and output of each queue
is same in steady state - r per cycle
Service time includes delays due to conflicts,
1  PA
calculated as T earlier. This has an
PA
exponential distribution – recall the analysis for a
shared bus.
M/M/1 open queue model can be used to calculate
queuing delay. Details are omitted.

Anshul Kumar, CSE IITD slide 34

 Switch
Switch with
with output
output buffers
buffers
Here we assume that all the messages destined for same
output are queued in the same buffer, in some order. That
is no rejections and no re-submissions.
For each queue,
rk
Messages arriving per service cycle =  =
m
Prob of a request coming from one of
r
the k sources = p =
m
Apply MB/D/1 model for finding queuing delay Tw
 p
Tw  T
2(1   )
Anshul Kumar, CSE IITD slide 35
 References
References
• D. Sima, T. Fountain, P. Kacsuk,
"Advanced Computer Architectures : A
Design Space Approach", Addison Wesley,
1997.
• K. Hwang, "Advanced Computer
Architecture : Parallelism, Scalability,
Programmability", McGraw Hill, 1993.

Anshul Kumar, CSE IITD slide 36