ECE498AL

Lecture 4: CUDA Threads

– Part 2

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 1

ECE498AL, University of Illinois, Urbana-Champaign
 CUDA Thread Block
• All threads in a block execute the same
kernel program (SPMD)
• Programmer declares block: CUDA Thread Block
– Block size 1 to 512 concurrent threads
– Block shape 1D, 2D, or 3D Thread Id #:
– Block dimensions in threads 0123… m
• Threads have thread id numbers within block
– Thread program uses thread id to select
work and address shared data Thread program

• Threads in the same block share data and

synchronize while doing their share of the
work
• Threads in different blocks cannot cooperate
Courtesy: John Nickolls, NVIDIA
– Each block can execute in any order relative
to other blocs!

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 2

ECE498AL, University of Illinois, Urbana-Champaign
 Transparent Scalability
• Hardware is free to assigns blocks to any
processor at any time
– A kernel scales across any number of
parallel processors
Device Kernel grid
Device
Block 0 Block 1

Block 2 Block 3

Block 0 Block 1 Block 4 Block 5

Block 0 Block 1 Block 2 Block 3
Block 6 Block 7
Block 2 Block 3
time
Block 4 Block 5 Block 6 Block 7

Block 4 Block 5
Each block can execute in any order relative to
Block 6 Block 7 other blocks.
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 3
ECE498AL, University of Illinois, Urbana-Champaign
 G80 CUDA mode – A Review
• Processors execute computing threads
• New operating mode/HW interface for computing
Host

Input Assembler

Thread Execution Manager

Parallel Data Parallel Data Parallel Data Parallel Data Parallel Data Parallel Data Parallel Data Parallel Data
Cache Cache Cache Cache Cache Cache Cache Cache

Texture
Texture Texture Texture Texture Texture Texture Texture Texture

Load/store Load/store Load/store Load/store Load/store Load/store

Global Memory
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 4
ECE498AL, University of Illinois, Urbana-Champaign
 G80 Example: Executing Thread Blocks
t0 t1 t2 … tm SM 0 SM 1 t0 t1 t2 … tm

MT IU MT IU
Blocks
SP SP

Blocks • Threads are assigned to Streaming

Multiprocessors in block granularity
– Up to 8 blocks to each SM as
Shared Shared resource allows
Memory Memory
– SM in G80 can take up to 768 threads
• Could be 256 (threads/block) * 3
Flexible resource allocation blocks
• Or 128 (threads/block) * 6 blocks, etc.
• Threads run concurrently
– SM maintains thread/block id #s
– SM manages/schedules thread
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 execution 5
ECE498AL, University of Illinois, Urbana-Champaign
 G80 Example: Thread Scheduling
• Each Block is executed as Block 1 Warps
… …Block 2 Warps Block 1 Warps
…
32-thread Warps t0 t1 t2 … t31 t0 t1 t2 … t31 t0 t1 t2 … t31
– An implementation decision, … … …
not part of the CUDA
programming model
– Warps are scheduling units Streaming Multiprocessor
in SM Instruction L1

• If 3 blocks are assigned to an Instruction Fetch/Dispatch

SM and each block has 256 Shared Memory

threads, how many Warps are SP SP

there in an SM? SP SP
SFU SFU
– Each Block is divided into SP SP
256/32 = 8 Warps
SP SP
– There are 8 * 3 = 24 Warps

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 6

ECE498AL, University of Illinois, Urbana-Champaign
 G80 Example: Thread Scheduling
(Cont.)

• SM implements zero-overhead warp scheduling

– At any time, only one of the warps is executed by SM
– Warps whose next instruction has its operands ready for
consumption are eligible for execution
– Eligible Warps are selected for execution on a prioritized
scheduling policy
– All threads in a warp execute the same instruction when selected

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 7

ECE498AL, University of Illinois, Urbana-Champaign
 G80 Block Granularity Considerations
• For Matrix Multiplication using multiple blocks, should I
use 8X8, 16X16 or 32X32 blocks?

– For 8X8, we have 64 threads per Block. Since each SM can take
up to 768 threads, there are 12 Blocks. However, each SM can
only take up to 8 Blocks, only 512 threads will go into each SM!

– For 16X16, we have 256 threads per Block. Since each SM can
take up to 768 threads, it can take up to 3 Blocks and achieve full
capacity unless other resource considerations overrule.

– For 32X32, we have 1024 threads per Block. Not even one can fit
into an SM!

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 8

ECE498AL, University of Illinois, Urbana-Champaign
 More Details of API Features

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 9

ECE498AL, University of Illinois, Urbana-Champaign
 Application Programming Interface
• The API is an extension to the C programming
language
• It consists of:
– Language extensions
• To target portions of the code for execution on the device
– A runtime library split into:
• A common component providing built-in vector types and a
subset of the C runtime library in both host and device
codes
• A host component to control and access one or more
devices from the host
• A device component providing device-specific functions
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 10
ECE498AL, University of Illinois, Urbana-Champaign
 Language Extensions:
Built-in Variables

• dim3 gridDim;
– Dimensions of the grid in blocks (gridDim.z
unused)
• dim3 blockDim;
– Dimensions of the block in threads
• dim3 blockIdx;
– Block index within the grid
• dim3 threadIdx;
– Thread index within the block
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 11
ECE498AL, University of Illinois, Urbana-Champaign
 Common Runtime Component:
Mathematical Functions
• pow, sqrt, cbrt, hypot
• exp, exp2, expm1
• log, log2, log10, log1p
• sin, cos, tan, asin, acos, atan, atan2
• sinh, cosh, tanh, asinh, acosh, atanh
• ceil, floor, trunc, round
• Etc.
– When executed on the host, a given function uses
the C runtime implementation if available
– These functions are only supported for scalar types,
not vector types
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 12
ECE498AL, University of Illinois, Urbana-Champaign
 Device Runtime Component:
Mathematical Functions
• Some mathematical functions (e.g. sin(x))
have a less accurate, but faster device-only
version (e.g. __sin(x))
– __pow
– __log, __log2, __log10
– __exp
– __sin, __cos, __tan

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 13

ECE498AL, University of Illinois, Urbana-Champaign
 Host Runtime Component
• Provides functions to deal with:
– Device management (including multi-device systems)
– Memory management
– Error handling

• Initializes the first time a runtime function is called

• A host thread can invoke device code on only one

device
– Multiple host threads required to run on multiple
devices
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 14
ECE498AL, University of Illinois, Urbana-Champaign
 Device Runtime Component:
Synchronization Function
• void __syncthreads();
• Synchronizes all threads in a block
• Once all threads have reached this point,
execution resumes normally
• Used to avoid RAW / WAR / WAW hazards
when accessing shared or global memory
• Allowed in conditional constructs only if the
conditional is uniform across the entire thread
block
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 15
ECE498AL, University of Illinois, Urbana-Champaign