CUDA C/C++ BASICS

NVIDIA Corporation

CUDA PROGRAMMING MODEL

Anatomy of a CUDA C/C++ Application

Serial code executes in a Host (CPU) thread
Parallel code executes in many Device (GPU) threads
across multiple processing elements
CUDA C/C++ Application
Serial code

Host = CPU
Device = GPU

Parallel code

Serial code

Host = CPU
Device = GPU

Parallel code

...

Compiling CUDA C Applications

void serial_function( ) {
...
}
void other_function(int ... ) {
...
}
void saxpy_serial(float ... ) {
for (int i = 0; i < n; ++i)
y[i] = a*x[i] + y[i];
}
void main( ) {
float x;
saxpy_serial(..);
...
}

Modify into
Parallel
CUDA C code

CUDA C
Functions

Rest of C
Application

NVCC
(Open64/LLVM)

CPU Compiler

CUDA object
files

CPU object
files

Linker

CPU-GPU
Executable

CUDA C : C with a few keywords

void saxpy_serial(int n, float a, float *x, float *y)
{
for (int i = 0; i < n; ++i)
y[i] = a*x[i] + y[i];
Standard
}
// Invoke serial SAXPY kernel
saxpy_serial(n, 2.0, x, y);

C Code

__global__ void saxpy_parallel(int n, float a, float *x, float *y)

{
int i = blockIdx.x*blockDim.x + threadIdx.x;
if (i < n) y[i] = a*x[i] + y[i];
Parallel C Code
}
// Invoke parallel SAXPY kernel with 256 threads/block
int nblocks = (n + 255) / 256;
saxpy_parallel<<<nblocks, 256>>>(n, 2.0, x, y);

CUDA Kernels
Parallel portion of application: execute as a kernel
Entire GPU executes kernel, many threads

CUDA threads:
Lightweight
Fast switching
1000s execute simultaneously
CPU

Host

Executes functions

GPU

Device

Executes kernels

CUDA Kernels: Parallel Threads

A kernel is a function executed
on the GPU as an array of
threads in parallel
All threads execute the same
code, can take different paths
Each thread has an ID
Select input/output data
Control decisions

float x = input[threadIdx.x];
float y = func(x);
output[threadIdx.x] = y;

CUDA Kernels: Subdivide into Blocks

CUDA Kernels: Subdivide into Blocks

Threads are grouped into blocks

CUDA Kernels: Subdivide into Blocks

Threads are grouped into blocks

Blocks are grouped into a grid

CUDA Kernels: Subdivide into Blocks

Threads are grouped into blocks

Blocks are grouped into a grid
A kernel is executed as a grid of blocks of threads

CUDA Kernels: Subdivide into Blocks

GPU

Threads are grouped into blocks

Blocks are grouped into a grid
A kernel is executed as a grid of blocks of threads

Kernel Execution
CUDA thread

CUDA thread block

CUDA core

CUDA Streaming
Multiprocessor

Each thread is executed by a

core

Each block is executed by

one SM and does not migrate
Several concurrent blocks can
reside on one SM depending
on the blocks memory
requirements and the SMs
memory resources

CUDA-enabled GPU
CUDA kernel grid

...

Each kernel is executed on

one device
Multiple kernels can execute
on a device at one time

Thread blocks allow cooperation

Instruction Cache
Scheduler Scheduler
Dispatch

Dispatch

Register File

Threads may need to cooperate:

Core Core Core Core

Core Core Core Core

Cooperatively load/store blocks of

memory all will use
Share results with each other or
cooperate to produce a single result
Synchronize with each other

Core Core Core Core

Core Core Core Core

Core Core Core Core
Core Core Core Core
Core Core Core Core
Core Core Core Core
Load/Store Units x 16
Special Func Units x 4
Interconnect Network

64K Configurable
Cache/Shared Mem
Uniform Cache

Thread blocks allow scalability

Blocks can execute in any order, concurrently or sequentially
This independence between blocks gives scalability:
A kernel scales across any number of SMs
Kernel Grid
Launch
Block 0

Device with 2 SMs

SM 0
Block 0
Block 2

SM 1
Block 1
Block 3

Block 4

Block 5

Block 6

Block 7

Block 1
Block 2
Block 3
Block 4
Block 5
Block 6
Block 7

Device with 4 SMs

SM 0

SM 1

SM 2

SM 3

Block 0

Block 1

Block 2

Block 3

Block 4

Block 5

Block 6

Block 7

What is CUDA?
CUDA Architecture
Expose GPU parallelism for general-purpose computing
Retain performance

CUDA C/C++
Based on industry-standard C/C++
Small set of extensions to enable heterogeneous programming
Straightforward APIs to manage devices, memory etc.

This session introduces CUDA C/C++

Introduction to CUDA C/C++

What will you learn in this session?
Start from Hello World!
Write and launch CUDA C/C++ kernels
Manage GPU memory
Manage communication and synchronization

Prerequisites
You (probably) need experience with C or C++
You dont need GPU experience
You dont need parallel programming experience

You dont need graphics experience

Heterogeneous Computing
Blocks
Threads

CONCEPTS

Indexing
Shared memory

__syncthreads()
Asynchronous operation
Handling errors
Managing devices

CONCEPTS

Heterogeneous Computing
Blocks
Threads
Indexing

Shared memory
__syncthreads()
Asynchronous operation

HELLO WORLD!

Handling errors
Managing devices

Heterogeneous Computing
Terminology:

Host
The CPU and its memory (host memory)
Device The GPU and its memory (device memory)

Host

Device

Heterogeneous Computing
#include <iostream>
#include <algorithm>
using namespace std;
#define N
1024
#define RADIUS 3
#define BLOCK_SIZE 16
__global__ void stencil_1d(int *in, int *out) {
__shared__ int temp[BLOCK_SIZE + 2 * RADIUS];
int gindex = threadIdx.x + blockIdx.x * blockDim.x;
int lindex = threadIdx.x + RADIUS;
// Read input elements into shared memory
temp[lindex] = in[gindex];
if (threadIdx.x < RADIUS) {
temp[lindex - RADIUS] = in[gindex - RADIUS];
temp[lindex + BLOCK_SIZE] = in[gindex + BLOCK_SIZE];
}
// Synchronize (ensure all the data is available)
__syncthreads();

parallel fn

// Apply the stencil

int result = 0;
for (int offset = -RADIUS ; offset <= RADIUS ; offset++)
result += temp[lindex + offset];
// Store the result
out[gindex] = result;
}
void fill_ints(int *x, int n) {
fill_n(x, n, 1);
}
int main(void) {
int *in, *out;
// host copies of a, b, c
int *d_in, *d_out;
// device copies of a, b, c
int size = (N + 2*RADIUS) * sizeof(int);
// Alloc space for host copies and setup values
in = (int *)malloc(size); fill_ints(in, N + 2*RADIUS);
out = (int *)malloc(size); fill_ints(out, N + 2*RADIUS);
// Alloc space for device copies
cudaMalloc((void **)&d_in, size);
cudaMalloc((void **)&d_out, size);
// Copy to device
cudaMemcpy(d_in, in, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_out, out, size, cudaMemcpyHostToDevice);
// Launch stencil_1d() kernel on GPU
stencil_1d<<<N/BLOCK_SIZE,BLOCK_SIZE>>>(d_in + RADIUS, d_out + RADIUS);
// Copy result back to host
cudaMemcpy(out, d_out, size, cudaMemcpyDeviceToHost);
// Cleanup
free(in); free(out);
cudaFree(d_in); cudaFree(d_out);
return 0;
}

serial code

parallel code
serial code

Simple Processing Flow

PCI Bus

1. Copy input data from CPU memory to GPU

memory

Simple Processing Flow

PCI Bus

1. Copy input data from CPU memory to GPU

memory
2. Load GPU program and execute,
caching data on chip for performance

Simple Processing Flow

PCI Bus

1. Copy input data from CPU memory to GPU

memory
2. Load GPU program and execute,
caching data on chip for performance
3. Copy results from GPU memory to CPU
memory

Hello World!
int main(void) {
printf("Hello World!\n");
return 0;
}

Output:
Standard C that runs on the host
NVIDIA compiler (nvcc) can be used to
compile programs with no device code

$ nvcc
hello_world.cu
$ a.out
Hello World!
$

Hello World! with Device Code

__global__ void mykernel(void) {
printf(Hello World from device!\n);
}

int main(void) {
mykernel<<<1,1>>>();
cudaDeviceSynchronize();
printf("Hello World from host!\n");
return 0;
}

Two new syntactic elements

Hello World! with Device Code

__global__ void mykernel(void) {
printf(Hello world from device!\n);
}

CUDA C/C++ keyword __global__ indicates a function that:

Runs on the device
Is called from host code

nvcc separates source code into host and device components

Device functions (e.g. mykernel()) processed by NVIDIA compiler
Host functions (e.g. main()) processed by standard host compiler
gcc, cl.exe

Hello World! with Device Code

mykernel<<<1,1>>>();

Triple angle brackets mark a call from host code to device code
Also called a kernel launch
Well return to the parameters (1,1) in a moment

Thats all that is required to execute a function on the GPU!

Access to BigRed2
ssh <username>@bigred2.uits.iu.edu
cp r /N/u/jbentz/BigRed2/oct2/cuda .
cd cuda
module load cudatoolkit
Use batch system for job submission
qsubsubmit a job to the queue
qstatshow all jobs in the queue
qdeldelete a job from the queue

Exercises: General Instructions (compiling)

Exercises are in cuda/exercises directory
Solutions are in cuda/exercise_solutions directory

To compile, use one of the provided makefiles

C:
> make

Exercises: General Instructions (running)

To run, just execute on the command line

> qsub runit

Hello world
Login to BigRed 2
Each coding project in a separate folder in the following dir
~cuda/exercises

cd cuda/exercises/hello_world
All dirs have Makefiles for you
Try building/running the code
make
Make sure youve loaded the cudatoolkit module!

qsub runit

Screenshot

Hello World! with Device Code

__global__ void mykernel(void) {
printf(Hello from device!\n);
}
int main(void) {
mykernel<<<1,1>>>();
printf("Hello from Host!\n");
return 0;
}

mykernel() does nothing interesting,

somewhat anticlimactic!

Output:

$ nvcc hello.cu
$ a.out
Hello from
device!
Hello from
Host!
$

Parallel Programming in CUDA C/C++

But wait GPU computing is about massive
parallelism!
We need a more interesting example
Well start by adding two integers and build up
to vector addition

Addition on the Device

A simple kernel to add two integers
__global__ void add(int *a, int *b, int *c) {
*c = *a + *b;
}

As before __global__ is a CUDA C/C++ keyword meaning

add() will execute on the device
add() will be called from the host

Addition on the Device

Note that we use pointers for the variables
__global__ void add(int *a, int *b, int *c) {
*c = *a + *b;
}
add() runs on the device, so a, b and c must point to device

memory
We need to allocate memory on the GPU

Memory Management
Host and device memory are separate entities
Device pointers point to GPU memory
May be passed to/from host code
May not be dereferenced in host code

Host pointers point to CPU memory

May be passed to/from device code
May not be dereferenced in device code

Simple CUDA API for handling device memory

cudaMalloc(), cudaFree(), cudaMemcpy()
Similar to the C equivalents malloc(), free(), memcpy()

Addition on the Device:

add()

Returning to our add() kernel

__global__ void add(int *a, int *b, int *c) {
*c = *a + *b;
}

Lets take a look at main()

Open exercises/simple_add/kernel.cu
Fill-in missing code as indicated.
Need to replace FIXME with code. Comments should help.
If something isnt clear, PLEASE ASK!

Addition on the Device:

int main(void) {
int a, b, c;
int *d_a, *d_b, *d_c;
int size = sizeof(int);

main()

// host copies of a, b, c
// device copies of a, b, c

// Allocate space for device copies of a, b, c

cudaMalloc((void **)&d_a, size);
cudaMalloc((void **)&d_b, size);
cudaMalloc((void **)&d_c, size);
// Setup input values
a = 2;
b = 7;

Addition on the Device:

main()

// Copy inputs to device

cudaMemcpy(d_a, &a, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_b, &b, size, cudaMemcpyHostToDevice);

// Launch add() kernel on GPU

add<<<1,1>>>(d_a, d_b, d_c);
// Copy result back to host
cudaMemcpy(&c, d_c, size, cudaMemcpyDeviceToHost);
// Cleanup
cudaFree(d_a); cudaFree(d_b); cudaFree(d_c);
return 0;
}

CONCEPTS

Heterogeneous Computing
Blocks
Threads
Indexing

Shared memory
__syncthreads()
Asynchronous operation

RUNNING IN PARALLEL

Handling errors
Managing devices

Moving to Parallel
GPU computing is about massive parallelism
So how do we run code in parallel on the device?
add<<< 1, 1 >>>();
add<<< N, 1 >>>();

Instead of executing add() once, execute N times in parallel

Vector Addition on the Device

With add() running in parallel we can do vector addition
Terminology: each parallel invocation of add() is referred to as a block
The set of blocks is referred to as a grid
Each invocation can refer to its block index using blockIdx.x
__global__ void add(int *a, int *b, int *c) {
c[blockIdx.x] = a[blockIdx.x] + b[blockIdx.x];
}

By using blockIdx.x to index into the array, each block handles a

different index

Vector Addition on the Device

__global__ void add(int *a, int *b, int *c) {
c[blockIdx.x] = a[blockIdx.x] + b[blockIdx.x];
}

On the device, each block can execute in parallel:

Block 0
c[0]

= a[0] + b[0];

Block 1
c[1]

= a[1] + b[1];

Block 2
c[2]

= a[2] + b[2];

Block 3
c[3]

= a[3] + b[3];

Vector Addition on the Device:

add()

Returning to our parallelized add() kernel

__global__ void add(int *a, int *b, int *c) {
c[blockIdx.x] = a[blockIdx.x] + b[blockIdx.x];
}

Lets take a look at main()

Open exercises/simple_add_blocks/kernel.cu
Fill-in missing code as indicated.
Should be clear from comments where you need to add some code
Need to replace FIXME with the proper piece of code.

Vector Addition on the Device:

main()

#define N 512
int main(void) {
int *a, *b, *c;
// host copies of a, b, c
int *d_a, *d_b, *d_c;
// device copies of a, b, c
int size = N * sizeof(int);
// Alloc space for device
cudaMalloc((void **)&d_a,
cudaMalloc((void **)&d_b,
cudaMalloc((void **)&d_c,
// Alloc
a = (int
b = (int
c = (int

copies of a, b, c
size);
size);
size);

space for host copies of a, b, c and setup input values

*)malloc(size); random_ints(a, N);
*)malloc(size); random_ints(b, N);
*)malloc(size);

Vector Addition on the Device:

main()

// Copy inputs to device

cudaMemcpy(d_a, a, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_b, b, size, cudaMemcpyHostToDevice);

// Launch add() kernel on GPU with N blocks

add<<<N,1>>>(d_a, d_b, d_c);
// Copy result back to host
cudaMemcpy(c, d_c, size, cudaMemcpyDeviceToHost);
// Cleanup
free(a); free(b); free(c);
cudaFree(d_a); cudaFree(d_b); cudaFree(d_c);
return 0;
}

Review (1 of 2)
Difference between host and device
Host CPU
Device GPU

Using __global__ to declare a function as device code

Executes on the device
Called from the host

Passing parameters from host code to a device function

Review (2 of 2)
Basic device memory management
cudaMalloc()
cudaMemcpy()
cudaFree()

Launching parallel kernels

Launch N copies of add() with add<<<N,1>>>();
Use blockIdx.x to access block index

CONCEPTS

Heterogeneous Computing
Blocks
Threads
Indexing

Shared memory
__syncthreads()
Asynchronous operation

INTRODUCING THREADS

Handling errors
Managing devices

CUDA Threads
Terminology: a block can be split into parallel threads
Lets change add() to use parallel threads instead of parallel blocks
__global__ void add(int *a, int *b, int *c) {
c[threadIdx.x] = a[threadIdx.x] + b[threadIdx.x];
}

We use threadIdx.x instead of blockIdx.x

Need to make one change in main()
Open exercises/simple_add_threads/kernel.cu

Vector Addition Using Threads:

#define N 512
int main(void) {
int *a, *b, *c;
int *d_a, *d_b, *d_c;
int size = N * sizeof(int);
// Alloc space for device
cudaMalloc((void **)&d_a,
cudaMalloc((void **)&d_b,
cudaMalloc((void **)&d_c,
// Alloc
a = (int
b = (int
c = (int

main()

// host copies of a, b, c
// device copies of a, b, c

copies of a, b, c
size);
size);
size);

space for host copies of a, b, c and setup input values

*)malloc(size); random_ints(a, N);
*)malloc(size); random_ints(b, N);
*)malloc(size);

Vector Addition Using Threads:

main()

// Copy inputs to device

cudaMemcpy(d_a, a, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_b, b, size, cudaMemcpyHostToDevice);
// Launch add() kernel on GPU with N threads
add<<<1,N>>>(d_a, d_b, d_c);

// Copy result back to host

cudaMemcpy(c, d_c, size, cudaMemcpyDeviceToHost);
// Cleanup
free(a); free(b); free(c);
cudaFree(d_a); cudaFree(d_b); cudaFree(d_c);
return 0;
}

CONCEPTS

Heterogeneous Computing
Blocks
Threads
Indexing

Shared memory
__syncthreads()
Asynchronous operation

COMBINING THREADS
AND BLOCKS

Handling errors
Managing devices

Combining Blocks and Threads

Weve seen parallel vector addition using:
Many blocks with one thread each
One block with many threads

Lets adapt vector addition to use both blocks and threads

Why? Well come to that
First lets discuss data indexing

Indexing Arrays with Blocks and Threads

No longer as simple as using blockIdx.x and threadIdx.x
Consider indexing an array with one element per thread (8 threads/block)
threadIdx.x

threadIdx.x

threadIdx.x

threadIdx.x

0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
blockIdx.x = 0

blockIdx.x = 1

blockIdx.x = 2

blockIdx.x = 3

With M threads/block a unique index for each thread is given by:

int index = threadIdx.x + blockIdx.x * M;

Indexing Arrays: Example

Which thread will operate on the red element?
0

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

threadIdx.x = 5

M = 8
0

blockIdx.x = 2

int index = threadIdx.x + blockIdx.x * M;

=
5
+
2
* 8;
= 21;

Vector Addition with Blocks and Threads

Use the built-in variable blockDim.x for threads per block
int index = threadIdx.x + blockIdx.x * blockDim.x;

Combined version of add() to use parallel threads and parallel

blocks
__global__ void add(int *a, int *b, int *c) {
int index = threadIdx.x + blockIdx.x * blockDim.x;
c[index] = a[index] + b[index];
}

What changes need to be made in main()?

Open exercises/simple_add_blocks_threads/kernel.cu

Addition with Blocks and Threads:

#define N (2048*2048)
#define THREADS_PER_BLOCK 512
int main(void) {
int *a, *b, *c;
int *d_a, *d_b, *d_c;
int size = N * sizeof(int);

// Alloc space for device

cudaMalloc((void **)&d_a,
cudaMalloc((void **)&d_b,
cudaMalloc((void **)&d_c,
// Alloc
a = (int
b = (int
c = (int

main()

// host copies of a, b, c
// device copies of a, b, c

copies of a, b, c
size);
size);
size);

space for host copies of a, b, c and setup input values

*)malloc(size); random_ints(a, N);
*)malloc(size); random_ints(b, N);
*)malloc(size);

Addition with Blocks and Threads:

main()

// Copy inputs to device

cudaMemcpy(d_a, a, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_b, b, size, cudaMemcpyHostToDevice);
// Launch add() kernel on GPU
add<<<N/THREADS_PER_BLOCK,THREADS_PER_BLOCK>>>(d_a, d_b, d_c);
// Copy result back to host
cudaMemcpy(c, d_c, size, cudaMemcpyDeviceToHost);
// Cleanup
free(a); free(b); free(c);
cudaFree(d_a); cudaFree(d_b); cudaFree(d_c);
return 0;
}

Handling Arbitrary Vector Sizes

Typical problems are not friendly multiples of

blockDim.x

Avoid accessing beyond the end of the arrays:

__global__ void add(int *a, int *b, int *c, int n) {
int index = threadIdx.x + blockIdx.x * blockDim.x;
if (index < n)
c[index] = a[index] + b[index];
}

Update the kernel launch:

add<<<(N + M-1) / M,M>>>(d_a, d_b, d_c, N);

Why Bother with Threads?

Threads seem unnecessary
They add a level of complexity
What do we gain?

Unlike parallel blocks, threads have mechanisms to:

Communicate
Synchronize

To look closer, we need a new example

Review
Launching parallel kernels
Launch N copies of add() with add<<<N/M,M>>>();
Use blockIdx.x to access block index
Use threadIdx.x to access thread index within block

Allocate elements to threads:

int index = threadIdx.x + blockIdx.x * blockDim.x;

CONCEPTS

Heterogeneous Computing
Blocks
Threads
Indexing

Shared memory
__syncthreads()
Asynchronous operation

COOPERATING THREADS

Handling errors
Managing devices

1D Stencil
Consider applying a 1D stencil to a 1D array of elements
Each output element is the sum of input elements within a radius

If radius is 3, then each output element is the sum of 7 input

elements:

radius

radius

Implementing Within a Block

Each thread processes one output element
blockDim.x elements per block

Input elements are read several times

With radius 3, each input element is read seven times

Simple Stencil in 1d
Open exercises/simple_stencil/kernel.cu
Finish the kernel and the kernel launch
Each thread calculates one stencil value
Reads 2*RADIUS + 1 values
dim3 type: CUDA 3 dimensional struct used for grid/block sizes

Inserted GPU timers into code to time the execution of the kernel
Try various sizes of N, RADIUS, BLOCK
Time a large (over a million) value of N with a RADIUS of 7

Can we do better?
Input elements are read multiple times
With RADIUS=3, each input element is read seven times!
Neighbouring threads read most of the same elements.
Thread 7 reads elements 4 through 10
Thread 8 reads elements 5 through 11

Can we avoid redundant reading of data?

Sharing Data Between Threads

Terminology: within a block, threads share data via shared memory
Extremely fast on-chip memory, user-managed
Declare using __shared__, allocated per block

Data is not visible to threads in other blocks

Implementing With Shared Memory

Cache data in shared memory (user managed scratch-pad)
Read (blockDim.x + 2 * radius) input elements from global memory to
shared memory
Compute blockDim.x output elements
Write blockDim.x output elements to global memory
Each block needs a halo of radius elements at each boundary

halo on right

halo on left

blockDim.x output elements

Stencil Kernel
__global__ void stencil_1d(int *in, int *out) {
__shared__ int temp[BLOCK_SIZE + 2 * RADIUS];
int gindex = threadIdx.x + blockIdx.x * blockDim.x;
int lindex = threadIdx.x + RADIUS;

// Read input elements into shared memory

temp[lindex] = in[gindex];
if (threadIdx.x < RADIUS) {
temp[lindex - RADIUS] = in[gindex - RADIUS];
temp[lindex + BLOCK_SIZE] =
in[gindex + BLOCK_SIZE];
}

Stencil Kernel
// Apply the stencil
int result = 0;
for (int offset = -RADIUS ; offset <= RADIUS ; offset++)
result += temp[lindex + offset];
// Store the result
out[gindex] = result;
}

Simple Stencil 1d with shared memory

cd exercises/simple_stencil_smem/
Run the code. It will build/run without modification.
If Errors occur, each offending element will be printed to the screen

What is the result with N=10,000 and BLOCK=32?

What is the result with N=10,000 and BLOCK=64?
Why?

Data Race!
The stencil example will not work
Suppose thread 15 reads the halo before thread 0 has fetched it
temp[lindex] = in[gindex];
Store at temp[18]
if (threadIdx.x < RADIUS) {
Skipped, threadIdx > RADIUS
temp[lindex RADIUS = in[gindex RADIUS];
temp[lindex + BLOCK_SIZE] = in[gindex + BLOCK_SIZE];
}
int result = 0;
result += temp[lindex + 1]; Load from temp[19]

__syncthreads()
void __syncthreads();

Synchronizes all threads within a block

Used to prevent RAW / WAR / WAW hazards

All threads must reach the barrier

In conditional code, the condition must be uniform across the block

Insert __syncthreads() into the kernel in the proper location

Compare timing of previous simple stencil with the current shared
memory implementation for same (large N) and BLOCK=512

Stencil Kernel
__global__ void stencil_1d(int *in, int *out) {
__shared__ int temp[BLOCK_SIZE + 2 * RADIUS];
int gindex = threadIdx.x + blockIdx.x * blockDim.x;
int lindex = threadIdx.x + radius;
// Read input elements into shared memory
temp[lindex] = in[gindex];
if (threadIdx.x < RADIUS) {
temp[lindex RADIUS] = in[gindex RADIUS];
temp[lindex + BLOCK_SIZE] = in[gindex + BLOCK_SIZE];
}
// Synchronize (ensure all the data is available)
__syncthreads();

Stencil Kernel
// Apply the stencil
int result = 0;
for (int offset = -RADIUS ; offset <= RADIUS ; offset++)
result += temp[lindex + offset];
// Store the result
out[gindex] = result;

Review (1 of 2)
Launching parallel threads
Launch N blocks with M threads per block with kernel<<<N,M>>>();
Use blockIdx.x to access block index within grid
Use threadIdx.x to access thread index within block

Allocate elements to threads:

int index = threadIdx.x + blockIdx.x * blockDim.x;

Review (2 of 2)
Use __shared__ to declare a variable/array in shared memory
Data is shared between threads in a block
Not visible to threads in other blocks
Using large shared mem size impacts number of blocks that can be
scheduled on an SM (48K total smem size)

Use __syncthreads() as a barrier

Use to prevent data hazards

Thank you!