PARALLEL
PROCESSING
UNIT
1
1
UNDERSTANDING
PARALLEL
2 ENVIRONMENT
QUIZ
What are 3 traditional ways HW Designers make
computers r u n faster?
Faster Clocks
Longer Clock Period
More Work per Clock Cycle
Larger Hard Disk
More Processors
Reduce amount of memory
3
SEYMOUR CRAY (SUPER COMPUTER
DESIGNER)
Ifyou are plowing a field, which would
you rat h er use?
⚫ Two strong oxen.
⚫ 1024 chickens
4
PARALLEL COMPUTING
It was intended to be used by super computing.
Now all computers/mobiles are using parallel
computing.
Modern GPUs
⚫ Hundred of processors
⚫ Thousand of ALUs (3,000)
⚫ Ten or thousands of concurrent threads.
This requires a different way of programming
t h a n a single scalar processor
General purpose programmability over GPU
(GPGPU.) 5
TRANSISTORS CONTINUE ON
MOORE’S PATH . . . FOR NOW
6
CLOCK SPEED (NO MORE
SPEED)
7
QUIZ
Are processing today getting faster Because
We are clocking their transistors faster
We have more transistors available
for computation.
o Why don’t we keep increasing clock speed of a
single processor instead of multiprocessors with a
less clock speed?
o No, we can’t because of power (heat)
8
WHAT KIND OF PROCESSORS
WILL WE BUILD?
Assume major design constraint is Power
Why are traditional CPU-like processors are not
the most energy efficient processors?
⚫ It has complex control hardware
⚫ This increase flexibility and performance
⚫ And increase power consumption and design
complexity as well
How to increase power efficiency (GPU-
like)?
⚫ Build simple control structure.
⚫ Take those transistors and devote them to support
9
more computation on the data path
⚫ The challenge becomes how to program?
MORE TO UNDERSTAND
10
Less speed with
M ORE TO UNDERSTAND (CONT.) simple
structure
More speed
with
complex
structure
Less
Power Power
11
QUIZ
Which techniques are computer designer using
today to build more power-efficient chips?
Fewer, more complex processors
More, Simpler processors
Maximizing the speed of the processor clock
Increasing the complexity of the control
hardware
12
ANOTHER FACTOR FOR POWER
EFFICIENCY
Power Efficiency
Decrease latency Increase Throughput
(Amount of time to (Task completed per
complete a task) unit time)
“Time” “Number”
The two goals are not
⚫ CPU-like: design to decrease latency
aligned
⚫ GPU-like: design to increase throughput
13
The choice depends on the application (Image processing
prefer to increase the throughput)
SUPER QUIZ
Why do I say GPU-like and not saying Multi-core
CPU? Is there a deference ?!
⚫ They both build for parallel programming. However,
Multi-core CPUs can be used for sequential and
parallel programming as well (provides branches and
interrupts ). On the other hand GPU build for
parallel programming from scratch.
14
GPU DESIGN BELIEVES
Lots of simple compute units
Explicitly parallel programming model
⚫ We know there are many processors and we didn’t
depend on the complier for example to parallel the
task for us.
Optimized for throughput not latency
15
INTRO TO
PARALLEL
16 PROGRAMMING
IMPORTANCE OF PARALLEL
PROGRAMMING
Intel 8 core Ivy bridge
8-wide AVX vector operations/core
2 threads core (hyper threading)
This means the processor has 128 way of
parallelism
Parallel programming is more complex however
Running sequential C program means using less
t h a n 1% of this processor power
17
CUDA PLATFORM
CUDA Program
W
ith
Ex
te
ns
io
C
ns
CPU GPU
"Host" Co-processor "Device "
Memory Memory
CUDA compiler generate two separated program one
for CPU (Host) and another for GPU (Device).
CPU in charge and control the GPU
⚫ Moves data between memories (cudaMemcpy)
⚫ Allocates memory on GPU (cudaMalloc)
⚫ Invokes programs (kernels) on the GPU: ”Host 18
lunches kernels on the Device”
QUIZ
The GPU can do the following:
Initiate dat a send from GPU to CPU
Respond to CPU request to send data from GPU
to CPU
Initiate dat a request from CPU to GPU
Respond to CPU request to receive data from
CPU to GPU
Compute a kernel lunched by CPU
Compute a kernel lunched by GPU
19
TYPICAL GPU PROGRAM
CPU allocate storage on GPU
CPU copy input data from CPU to GPU
CPU lunches the kernels on the GPU to process
the data
CPU copies results back to the CPU from the
GPU
If you need to move data many times between
CPU and GPU, CUDA is not good for your
program because it takes many steps to do so as
showing above 20
MAIN ISSUE
Defining the GPU computation
⚫ Write a Kernel like serial program
⚫ When lunching the kernel tell the GPU how
many threads to lunch
21
QUIZ
What is the GPU good at?
Lunching a small number of threads
efficiently
Lunching a large number of threads efficiently
Running one thread very Quickly
Respond to CPU request to receive data from
CPU to GPU
Running one thread t h a t does lots of work
in parallel 22
Running a large number of threads in
GPU P OWER
Example:
⚫ In : [1, 2, 3, …., 64]
⚫ Out: [02 , 1 2 , 2 2 , …., 642 ]
Sequential solution:
for(int i=0;i<64;i++)
Out[i]=in[i]*in[i];
⚫ here we have 1 thread do 64 multiplications
each takes 2 ns.
23
GPU P OWER (CONT.)
Example:
⚫ In : [1, 2, 3, …., 64]
⚫ Out: [02 , 1 2 , 2 2 , …., 642 ]
CPU GPU
Allocate memory out= in * in
Copy data to/from GPU
launch kernel
Parallel solution: j
⚫ CPU code: square kernel <<<64>>>(out, in)
⚫ here we have 64 thread each do 1 multiplication which
take 10 ns. 24
EXAMP
25 LE
start
THREADS AND
26 BLOCKS
THREADS
A single execution units t h a t r u n kernels on the
GPU. Similar to CPU threads but there's usually
many more of them. They are sometimes drawn as
arrows
27
BLOCKS
Thread blocks are a virtual collection of threads.
All the threads in any single thread block can
communicate
28
GRID
A kernel is launched as a collection of thread
blocks called the grid.
29
MAXIMUMS
You can launch up to 1024 threads per block (or
512 if your card is compute capability 1.3 or less).
You can launch 2 32 -1 blocks in a single launch(or
2 16 -1 if your card is compute capability 2.0 or
less).
So my relatively inexpensive GeForce GT 440 can
launch a rat her ridiculous 67,108,864 threads.
30
WHY BLOCKS AND THREADS?
You may be wondering why not just say “launch 67 million
threads” instead of organizing them into blocks.
Suppose you wrote a program for a GPU can which can
r u n 2000 threads concurrently. Then you want to execute
the same code on a higher GPU with 6000 threads. Are you
going to change the whole code fore each GPU?
Each GPU h as a limit on the number of threads per block
but (almost) no limit on the number of blocks. Each GPU
can r u n some number of blocks concurrently, executing
some number of threads simultaneously.
By adding the extra level of abstraction, higher
performance GPU's can simply r u n more blocks
concurrently and chew through the workload quicker with
absolutely no change to the code.
nVidia h as done this to allow automatic performance gains
when your code is r u n on different higher performance
GPU's. 31
DIM3
32
DIM3 DATA TYPE
Dim3 is a 3d structure or vector type with three
integers, x, y and z. You can initialize as many of
the three coordinates as you like:
⚫ dim3 threads(256); // Initialize with x as 256, y and z
// will both be 1
⚫ dim3 blocks(100, 100); // Initialize x and y, z will be 1
dim3 anotherOne(10, 54, 32); // Initialises all
three values, x
⚫ // will be 10, y gets 54 and z
⚫ // will be the 32.
33
THREAD ACCESS PARAMETERS
Each of the running threads is individual, they
know the following:
threadIdx ← Thread index within the block
blockIdx ← Block index within the grid
blockDim ← Number of threads in the block
gridDim ← Number of blocks in the grid
Each of these are dim3 structures and can be
read in the kernel to assign particular workloads
to any thread. 34
THREAD ACCESS PATTERN
Its common to have threads calculate a unique id
within the kernel to process some specific data. If we
launch a kernel with:
SomeKernel<<<100, 25>>>(...);
Inside the kernel, each thread can calculate a unique
id with:
⚫ int id = blockIdx.x * blockDim.x + threadIdx.x;
So the 5th thread of the 4th block would calculate:
⚫ int id = 4 * 25 + 5 = 105
The 14th thread of the 76th block would calculate:
⚫ int id = 76 * 25 + 14 = 1914 35
MAPPIN
36 G
MAP
Set of elements to process [64 floats]
Function to r u n on each element [square]
Map(element, function)
37
QUIZ
Which programs can be solved using Map
Sort a n input array
Add one to each element of a n input array
Sum up all elements of a n input array
Compute the average of and input array
38
