Parallel and Distributed Computing

CS3006 (BCS-6E/6F)
Lecture 17
Instructor: Dr. Syed Mohammad Irteza
Assistant Professor, FAST School of Computing
NUCES Lahore
24 March, 2025
Previous Lecture
• Basic Communication Operations
• Scatter
• Gather
• Circular q-shift
• All Operations
• Across hypercube
• Across 2-D ring
• Across 4*4 mesh
• Across balanced binary tree

2
A more in-depth look
• Algorithms of these operations

3
1.
2.
procedure ONE_TO_ALL_BC(d, my_id, X)
begin One-to-All Broadcast
3. mask := 2d - 1; /* Set all d bits of mask to 1 */
4. for i := d - 1 downto 0 do /* Outer loop */
5. mask := mask XOR 2i; /* Set bit i of mask to 0 */
6. if (my_id AND mask) = 0 then /* If lower i bits of my_id are 0 */
7. if (my_id AND 2i) = 0 then
8. msg_destination := my_id XOR 2i;
9. send X to msg_destination;
10. else
(6) my_id AND mask==0 → if I will be in
11. msg_source := my_id XOR 2i; communication in this step (if my i bits are zero)
12. receive X from msg_source;
13. endelse; (7) my_id AND 2i == 0 → if I’m the sender of
message (if my i+1th bit is 0)
14. endif;
15. endfor; (8) XOR-ing my_id with 2i gives my partner in the ith
16. end ONE_TO_ALL_BC phase of the algorithm

Algorithm 4.1 One-to-all broadcast of a message X from node 0 of a d-dimensional p-node hypercube
(d = log p). AND and XOR are bitwise logical-and and exclusive-or operations, respectively.
4
1.
2.
procedure GENERAL_ONE_TO_ALL_BC(d, my_id, source, X)
begin
One-to-All Broadcast:
3.
4.
my_virtual id := my_id XOR source;
mask := 2d - 1;
Generic Form
5. for i := d - 1 downto 0 do /* Outer loop */
6. mask := mask XOR 2i; /* Set bit i of mask to 0 */ source XOR source makes source→ zero and
7. if (my_virtual_id AND mask) = 0 then similarly maps other according to the source
8. if (my_virtual_id AND 2i) = 0 then
9. virtual_dest := my_virtual_id XOR 2i;
10. send X to (virtual_dest XOR source);
/* Convert virtual_dest to the label of the physical destination */
11. else
12. virtual_source := my_virtual_id XOR 2i;
13. receive X from (virtual_source XOR source);
/* Convert virtual_source to the label of the physical source */
14. endelse;
15. endfor;
16. end GENERAL_ONE_TO_ALL_BC

Algorithm 4.2 One-to-all broadcast of a message X initiated by source on a d-dimensional hypothetical

hypercube. The AND and XOR operations are bitwise logical operations.
5
1. procedure ALL_TO_ONE_REDUCE(d, my_id, m, X, sum)
2. begin
3.
4.
for j := 0 to m - 1 do sum[j] := X[j];
mask := 0;
All-to-One Reduce
5. for i := 0 to d - 1 do
/* Select nodes whose lower i bits are 0 */
6. if (my_id AND mask) = 0 then
7. if (my_id AND 2i) ≠ 0 then
8. msg_destination := my_id XOR 2i;
9. send sum to msg_destination;
10. else
11. msg_source := my_id XOR 2i;
12. receive X from msg_source;
13. for j := 0 to m - 1 do
14. sum[j] :=sum[j] + X[j];
15. endelse;
16. mask := mask XOR 2i; /* Set bit i of mask to 1 */
17. endfor;
18. end ALL_TO_ONE_REDUCE

Algorithm 4.3 Single-node accumulation on a d-dimensional hypercube. Each node contributes a message X
containing m words, and node 0 is the destination of the sum. The AND and XOR operations are bitwise
6
logical operations.
Basic Communication Operations
(One-to-All Broadcast and All-to-One Reduction)
• Cost Estimation
• Broadcast needs log(p) point-to-point simple message transfer steps.
• Message size of each transfer is m
• Time for each of the transfers is: 𝒕𝒔 + 𝒎𝒕𝒘

Hence cost for log(p) transfers→ T = (𝒕𝒔 +𝒎𝒕𝒘 ) log 𝒑

7
 All-to-All Broadcast and All-to-All Reduction
8

All-to-All Broadcast
• A generalization of one-to-all broadcast.
• Every process broadcasts an m-word message.
• The broadcast-message for each of the processes can be different from others

All-to-All Reduction
• Dual of all-to-all broadcast
• Each node is the destination of an all-to-one reduction out of total P
reductions.
 All-to-All Broadcast and All-to-All Reduction
9

A naïve Broadcast method may be

Reduction: let’s say every node has generated different P random numbers.
performing P one-to-all broadcasts. The problem is to sum the respective indices and store it to respective process

This will result 𝑝(log 𝑝 (𝑡𝑠 + 𝑚𝑡𝑤 )) Solution: It is possible to use the communication links in the interconnection
network more efficiently by performing all p one-to-all broadcasts
communication time.
simultaneously so that all messages traversing the same path at the same
time are concatenated into a single message whose size is the sum of the sizes
Solution? of individual messages.
 All-to-All Broadcast and
All-to-All Reduction

Figure 4.9 All-to-all broadcast on an eight-node

ring. The label of each arrow shows the time step
and, within parentheses, the label of the node
that owned the current message being
transferred before the beginning of the
broadcast.

The number(s) in parentheses next to each node

are the labels of nodes from which data has been
received prior to the current communication
step.

Only the first, second, and last communication

steps are shown.
 Linear Ring Broadcast
11
 All-to-All Reduction
12
Linear Array or Ring
• Reduction
• Draw an All-to-All Broadcast on a P-node linear ring
• Reverse the directions in each foreach of the step without changing message
• After each communication step, combine messages having same broadcast
destination with associative operator.

• Now, Its your turn to draw?

• Draw an All-to-All Broadcast on a 4-node linear ring
• Reverse the directions and combine the results using ‘SUM’
 14
All-to-All
Broadcast on 2D
Mesh
All-to-All15Broadcast
on 2D Mesh
Algorithm
 All-to-All Broadcast
on Hypercube

Log p steps

→communicating nodes exchange data in each dimension

→Message size doubles in every iteration

→Reduction is performed by simply reversing the steps.

→For example in first step: P1 will send (4,5,6,7) to P5 and

P5 will send (0,1,2,3) to P1..Both reduce their contents
by half
 All-to-All Broadcast on Hypercube. Algorithm
17

XOR-ing an id with 2^(dimension-1) gives partner in that dimension.

References
1. Slides from Dr. Rana Asif Rehman & Dr. Haroon Mahmood
2. Kumar, V., Grama, A., Gupta, A., & Karypis, G. (1994). Introduction to parallel computing (Vol. 110).
Redwood City, CA: Benjamin/Cummings.
3. https://phyweb.physics.nus.edu.sg/~phytaysc/cz4102_07/cz4102_le6.pdf

CS3006 - Spring 2025 18