CUDA Programming with Python: From Basics to Expert Proficiency

CUDA Programming with Python

From Basics to Expert Proficiency

Copyright © 2024 by HiTeX Press

All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law.

Contents

1 Introduction to CUDA Programming

1.1 What is CUDA?

1.2 History and Evolution of CUDA

1.3 Overview of GPU Computing

1.4 Importance of Parallel Processing

1.5 GPU vs CPU: Key Differences

1.6 CUDA Software and SDK

1.7 Basic Terminologies in CUDA

1.8 CUDA Programming Models

1.9 Applications of CUDA: Real-World Examples

1.10 Future of CUDA and GPU Computing

2 Setting Up the Development Environment

2.1 System Requirements for CUDA Development

2.2 Installing CUDA Toolkit

2.3 Setting Up Visual Studio Code for CUDA

2.4 Installing Anaconda and Python

2.5 Setting Up Numba for CUDA Programming

2.6 Verifying the Installation

2.7 Introduction to CUDA Samples

2.8 Managing CUDA Libraries and Dependencies

2.9 Setting Up Jupyter Notebooks for CUDA Development

2.10 Troubleshooting Common Installation Issues

3 Python and Numba Introduction

3.1 Introduction to Python for Scientific Computing

3.2 Installing and Setting Up Python

3.3 NumPy: The Foundation for Data Science in Python

3.4 Understanding JIT Compilation

3.5 Introduction to Numba

3.6 Installing and Setting Up Numba

3.7 Numba Basics: Accelerating Python Functions

3.8 GPU Acceleration with Numba

3.9 Comparing Numba with Other Python Accelerators

3.10 Real-World Applications of Numba

4 CUDA Architecture and Memory Model

4.1 Overview of CUDA Architecture

4.2 Streaming Multiprocessors (SMs)

4.3 CUDA Cores and Their Functionality

4.4 The Memory Hierarchy in CUDA

4.5 Global Memory and Its Characteristics

4.6 Shared Memory: Benefits and Usage

4.7 Constant and Texture Memory

4.8 Registers and Local Memory

4.9 Memory Coalescing and Access Patterns

4.10 Latency and Bandwidth Considerations

4.11 Memory Management and Optimization Strategies

4.12 Understanding the CUDA Execution Model

5 Basic CUDA Programming Concepts

5.1 Introduction to CUDA Programming Basics

5.2 CUDA Program Structure

5.3 Writing and Compiling a Simple CUDA Program

5.4 Understanding Kernels and Thread Hierarchy

5.5 Grid and Block Dimensions

5.6 Memory Allocation and Transfer between Host and Device

5.7 Launching Kernels: Syntax and Parameters

5.8 Synchronizing Threads

5.9 Error Handling in CUDA

5.10 Using CUDA Libraries: An Overview

5.11 Common Pitfalls and Best Practices

6 Parallel Programming Concepts

6.1 Introduction to Parallel Programming

6.2 Types of Parallelism: Data vs Task Parallelism

6.3 Understanding Concurrency and Parallelism

6.4 Amdahl’s Law and Its Implications

6.5 Parallel Programming Models

6.6 Designing Parallel Algorithms

6.7 Synchronization Techniques

6.8 Load Balancing and Partitioning

6.9 Scalability and Performance Metrics

6.10 Case Studies: Parallel Algorithms

7 CUDA with Python: Numba Basics

7.1 Introduction to Numba for CUDA

7.2 Setting Up Numba for CUDA Development

7.3 Writing Your First Numba-CUDA Kernel

7.4 Compiling and Running Numba-CUDA Kernels

7.5 Understanding and Using CUDA Threading Model with Numba

7.6 Memory Management with Numba

7.7 Optimizing Numba-CUDA Code

7.8 Troubleshooting and Common Issues

7.9 Integrating Numba with Other Python Libraries

7.10 Advanced Techniques with Numba-CUDA

8 Advanced CUDA Programming Techniques

8.1 Introduction to Advanced CUDA Programming

8.2 Using Streams for Concurrent Execution

8.3 Asynchronous Memory Transfers

8.4 Dynamic Parallelism in CUDA

8.5 CUDA Graphs and Task Management

8.6 Efficient Memory Management Techniques

8.7 Optimizing Data Transfers

8.8 Advanced CUDA Libraries and Frameworks

8.9 Using Thrust for High-Level Algorithms

8.10 Interoperability with Other GPU APIs

8.11 Advanced Profiling and Analysis Techniques

8.12 Leveraging Peer-to-Peer Memory Access

9 Debugging and Profiling CUDA Applications

9.1 Introduction to Debugging and Profiling CUDA Applications

9.2 Common Debugging Challenges in CUDA

9.3 Using NVIDIA Nsight for Debugging

9.4 Debugging with CUDA-GDB

9.5 Analyzing Memory Errors and Race Conditions

9.6 Introduction to Profiling Tools

9.7 Using NVIDIA Visual Profiler

9.8 Understanding and Interpreting Profiling Reports

9.9 Optimizing Performance Based on Profiling Data

9.10 Debugging and Profiling in Jupyter Notebooks

9.11 Best Practices for Debugging and Profiling

10 Optimization Strategies for CUDA Programs

10.1 Introduction to CUDA Optimization Strategies

10.2 Understanding Performance Metrics

10.3 Code Optimization Techniques

10.4 Memory Optimization Strategies

10.5 Optimizing Kernel Launch Configurations

10.6 Efficient Data Transfer Techniques

10.7 Utilizing Shared Memory Efficiently

10.8 Reducing Divergence in GPU Threads

10.9 Optimizing with CUDA Streams and Events

10.10 Leveraging Advanced CUDA Libraries

10.11 Case Studies in CUDA Optimization

10.12 Best Practices for CUDA Optimization

Introduction

CUDA (Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) model created by NVIDIA. It allows developers to use NVIDIA graphics processing units (GPUs) for general purpose processing—an approach known as GPGPU (General-Purpose computing on Graphics Processing Units). Over the past decade, CUDA has revolutionized industries that require high-performance computing, enabling advancements in scientific research, data analytics, machine learning, and more.

The purpose of this book is to provide a comprehensive and clear guide to CUDA programming using Python, primarily through the Numba library. Numba is an open-source JIT compiler that translates a subset of Python and NumPy code into fast machine code, handling the complexity of GPU programming and allowing developers to leverage powerful GPU resources with minimal hassle.

Understanding CUDA alongside Python is essential for those looking to harness the full potential of their hardware without delving into more complex languages like C++. This book is designed to be accessible to programmers who have a basic understanding of Python and want to expand their knowledge into parallel computing and GPU-accelerated applications. No prior experience with CUDA or GPU programming is required.

We’ll begin by setting up a development environment that ensures compatibility and efficiency, covering installation steps, required tools, and verification processes to avoid common pitfalls. Following this, we will dive into CUDA’s architecture, explaining key concepts such as the execution model, memory hierarchy, and the differentiation between GPU and CPU processing.

Basic concepts of CUDA programming will be explored in detail, including writing simple CUDA programs, managing memory between host and device, understanding kernel functions, and handling errors. These foundational topics are crucial for any developer aiming to write efficient CUDA applications.

Moreover, the book examines parallel programming concepts, offering insights into the design and implementation of parallel algorithms. This includes an understanding of data parallelism and task parallelism, synchronization techniques, and performance metrics critical to optimizing parallel computations.

In the realm of combining CUDA with Python, we delve into Numba’s capabilities for GPU acceleration. The sections will cover setting up Numba, writing CUDA kernels in Python, managing GPU memory, and optimizing code. Advanced techniques and best practices are also discussed for readers aiming to push the performance boundaries of their applications.

Debugging and profiling are essential aspects of CUDA programming, ensuring correctness and achieving peak performance. This book includes sections dedicated to using tools such as NVIDIA Nsight and CUDA-GDB for debugging, and NVIDIA Visual Profiler for performance analysis. Profiling insights guide the optimization processes, providing a methodical approach to enhance program efficiency.

Finally, we explore advanced CUDA programming techniques and optimization strategies. Concurrent execution with streams, efficient memory management, dynamic parallelism, and interoperability with other GPU APIs are topics covered to equip readers with advanced skills necessary for complex and high-performance CUDA applications.

This book aims to serve as a thorough reference for beginners and intermediate programmers, providing the necessary knowledge and tools to develop efficient, high-performance parallel applications with CUDA and Python. Whether you are a researcher, a data scientist, or a software engineer, the principles and practices detailed within will significantly enhance your computational capabilities and performance.

Chapter 1

Introduction to CUDA Programming

CUDA is a parallel computing platform developed by NVIDIA, enabling efficient utilization of graphics processing units (GPUs) for general-purpose computing. This chapter provides an overview of CUDA, tracing its evolution and highlighting the significance of GPU computing in various applications. Readers will be introduced to fundamental concepts such as parallel processing, the distinctions between GPUs and CPUs, essential terminologies, and the basic programming models used in CUDA. Additionally, the chapter explores the practical applications and future prospects of CUDA in advancing computational performance across multiple domains.

1.1

What is CUDA?

CUDA, an acronym for Compute Unified Device Architecture, is a parallel computing platform and application programming interface (API) model created by NVIDIA. It allows software developers to harness the tremendous processing power of NVIDIA GPUs for general-purpose computing, referred to as General-Purpose computing on Graphics Processing Units (GPGPU). Unlike traditional usage of GPUs, which were strictly confined to graphics processing tasks, CUDA transforms these graphics devices into a versatile parallel computing powerhouse.

At its core, CUDA provides a layer of abstraction that enables developers to leverage the massive parallel processing capabilities inherent in GPUs. It extends the C, C++, and Fortran programming languages by providing constructs that express parallelism, allowing developers to write programs where each thread operates independently but simultaneously. CUDA is composed of both the CUDA runtime and the CUDA driver API, facilitating direct interactions with the GPU hardware.

A typical CUDA program consists of host code—executed on the Central Processing Unit (CPU)—and device code, which runs on the GPU. The host is responsible for handling general computation control and data transfer between the host memory and device memory, while the device executes the computationally intensive portions of a program. This segregation of tasks ensures optimal utilization of both the CPU and GPU resources.

A fundamental feature of CUDA is its hierarchical model of parallelism. Threads are organized into blocks, and blocks are grouped into a grid. This arrangement allows for scalability and flexibility in computing resources management. Each thread within a block can share data through shared memory, and multiple blocks can operate independently, making full use of the GPU’s computational units.

To illustrate the introduction of CUDA, consider a simple example of adding two arrays using CUDA. Below is a code snippet demonstrating this in Python with PyCUDA:

import

pycuda

.

driver

as

cuda

import

pycuda

.

autoinit

from

pycuda

.

compiler

import

SourceModule

import

numpy

as

np

#

Kernel

code

in

CUDA

C

kernel_code

=

"

__global__

void

add_arrays

(

float

*

a

,

float

*

b

,

float

*

c

,

int

n

)

{

int

idx

=

threadIdx

.

x

+

blockDim

.

x

*

blockIdx

.

x

;

if

(

idx

<

n

)

{

c

[

idx

]

=

a

[

idx

]

+

b

[

idx

];

}

}

"

#

Compile

the

kernel

code

mod

=

SourceModule

(

kernel_code

)

add_arrays

=

mod

.

get_function

(

"

add_arrays

"

)

#

Define

array

size

N

=

1000

#

Initialize

host

arrays

a

=

np

.

random

.

randn

(

N

)

.

astype

(

np

.

float32

)

b

=

np

.

random

.

randn

(

N

)

.

astype

(

np

.

float32

)

c

=

np

.

empty_like

(

a

)

#

Allocate

device

memory

and

copy

host

arrays

to

device

a_gpu

=

cuda

.

mem_alloc

(

a

.

nbytes

)

b_gpu

=

cuda

.

mem_alloc

(

b

.

nbytes

)

c_gpu

=

cuda

.

mem_alloc

(

c

.

nbytes

)

cuda

.

memcpy_htod

(

a_gpu

,

a

)

cuda

.

memcpy_htod

(

b_gpu

,

b

)

#

Launch

kernel

block_size

=

256

grid_size

=

int

(

np

.

ceil

(

N

/

block_size

)

)

add_arrays

(

a_gpu

,

b_gpu

,

c_gpu

,

np

.

int32

(

N

)

,

block

=(

block_size

,

1,

1)

,

grid

=(

grid_size

,

1)

)

#

Copy

result

back

to

host

cuda

.

memcpy_dtoh

(

c

,

c_gpu

)

print

(

"

Array

addition

result

:

"

,

c

)

This code demonstrates the basic workflow of a CUDA program:

1. Definition of a Kernel: The kernel function, written in CUDA C, is defined to perform element-wise addition of two arrays. 2. Memory Allocation: Host memory is allocated and initialized, followed by allocation on the device (GPU) for the input and output arrays. 3. Data Transfer: Data is transferred from host to device memory. 4. Kernel Launch: The kernel is launched with specified grid and block dimensions. 5. Result Retrieval: The result is copied back from the device to the host memory.

The kernel function add_arrays takes four parameters:

Pointers to the input arrays a and b.

A pointer to the output array c.

An integer n representing the number of elements in the arrays.

The function uses the built-in variables threadIdx, blockDim, and blockIdx to compute the global index idx for each thread. This index is utilized to perform the addition operation on corresponding elements that fall within the array bounds. The resulting values are stored in the output array c.

CUDA’s architecture provides developers with fine-grained control over memory hierarchy, including:

Global Memory: Large memory accessible by all threads but with higher latency.

Shared Memory: Fast, low-latency memory shared among threads within the same block.

Registers: Ultra-fast memory available to each thread.

This control allows for performance optimization by minimizing latency and maximizing throughput.

CUDA supports numerous libraries and tools, such as cuBLAS for linear algebra, cuFFT for Fast Fourier Transforms, and Thrust for parallel algorithms, significantly enhancing productivity and efficiency in application development. Integrating these libraries simplifies complex operations, allowing developers to focus on higher-level design rather than low-level optimizations.

Understanding CUDA’s fundamental concepts and programming model is crucial for effectively leveraging GPU capabilities. Advanced topics such as memory coalescing, warp divergences, and occupancy management provide additional layers of optimization, crucial for attaining peak performance.

The ensuing sections of this chapter delve into the historical context, GPU computing overview, and detailed exploration of parallel processing aspects, setting the stage for deeper insights into CUDA’s capabilities and applications.

1.2

History and Evolution of CUDA

CUDA, or Compute Unified Device Architecture, has its roots in the early developments of parallel computing, which sought to harness the power of multiple processing units working concurrently to solve computational problems more efficiently. Historically, parallel computing relied heavily on intricate programming models and specialized hardware, limiting its broad adoption. The advent of CUDA marked a significant shift by providing a more accessible and versatile framework for parallel computing, specifically leveraging NVIDIA’s Graphics Processing Units (GPUs).

The origins of CUDA can be traced back to NVIDIA’s introduction of the GPU. The concept of a GPU was pioneered to accelerate the rendering of images for computer graphics. Initially, these GPUs were designed with fixed-function pipelines, tailored to specific tasks in rendering graphics. However, as the demand for more complex and realistic graphics grew, so did the need for more programmable and flexible architectures.

In 2000, NVIDIA introduced the GeForce 256, which was termed the world’s first GPU. This marked the beginning of a new era in graphical computation, focusing on programmable shading, which allowed developers to write custom shaders using languages like Cg and HLSL. These advancements laid the groundwork for a more generalized and programmable use of GPUs.

The real breakthrough for general-purpose GPU computing (GPGPU) arrived with the release of CUDA in 2007. CUDA 1.0 was developed in response to the limitations of earlier GPGPU efforts that utilized graphics APIs like OpenGL and Direct3D for non-graphical computations. These efforts were cumbersome and required deep expertise in graphics programming, making them inaccessible to many developers. CUDA provided a more straightforward and cohesive environment by allowing programmers to write scalable and efficient parallel code using a language similar to C.

The initial versions of CUDA were designed to provide essential building blocks for parallel computing, such as thread hierarchies, shared memory, and synchronization primitives. These features made it easier for scientists, engineers, and developers to write parallel code without needing to master the intricacies of traditional graphical APIs.

Subsequent versions of CUDA brought significant improvements and extensions to the initial model. CUDA 2.0, released in 2008, introduced double-precision floating-point support, making it suitable for high-performance computing applications in scientific research. CUDA 3.0, released in 2010, included features like unified addressing, which simplified memory management by consolidating the device and host memory spaces into a single address space.

A notable advancement came with the introduction of CUDA 5.0 in 2012, which provided dynamic parallelism. This allowed a GPU kernel to launch other kernels, enabling more complex and flexible computations directly on the device. This feature significantly enhanced the capability of GPUs to handle more sophisticated algorithms and workflows.

CUDA’s evolution continued with enhancements aimed at improving performance, ease of use, and support for diverse applications. CUDA 6.0 introduced the concept of Unified Memory in 2014, which further simplified memory management by providing a shared memory space accessible by both the CPU and GPU. This advance reduced the need for explicit memory transfers between the host and device, making it easier to develop applications that leverage the GPU’s computational power.

The development trajectory of CUDA has also emphasized backward compatibility, ensuring that existing applications continue functioning with newer versions of the framework. This feature has been instrumental in building a robust ecosystem around CUDA, encouraging long-term investment from academia and industry.

Over the years, CUDA has expanded its ecosystem with an extensive set of libraries and tools designed to accelerate specific types of computations. These include cuBLAS for linear algebra, cuFFT for fast Fourier transforms, and cuDNN for deep neural networks. Such libraries have been optimized to leverage the parallel architecture of GPUs, providing substantial performance improvements over their CPU counterparts.

The timeline of CUDA’s evolution highlights a relentless pursuit of making parallel computing more accessible, potent, and applicable to a wide range of domains, from scientific research to machine learning and real-time data processing. The synergy between continuous hardware advancements and the progressing CUDA platform has cemented NVIDIA GPUs as a pivotal component in the landscape of high-performance computing.

import

pycuda

.

autoinit

import

pycuda

.

driver

as

drv

import

numpy

from

pycuda

.

compiler

import

SourceModule

mod

=

SourceModule

(

"

__global__

void

multiply_them

(

float

*

dest

,

float

*

a

,

float

*

b

)

{

const

int

i

=

threadIdx

.

x

;

dest

[

i

]

=

a

[

i

]

*

b

[

i

];

}

"

)

multiply_them

=

mod

.

get_function

(

"

multiply_them

"

)

a

=

numpy

.

random

.

randn

(400)

.

astype

(

numpy

.

float32

)

b

=

numpy

.

random

.

randn

(400)

.

astype

(

numpy

.

float32

)

dest

=

numpy

.

zeros_like

(

a

)

multiply_them

(

drv

.

Out

(

dest

)

,

drv

.

In

(

a

)

,

drv

.

In

(

b

)

,

block

=(400,1,1)

,

grid

=(1,1)

)

print

(

dest

)

[ 0.15315579 -0.4211322  1.6233644 -0.25260237 0.9508752 -1.9649584

-1.7057542  0.13941771 -0.14287743 -1.0599248 0.17026755 0.67843133

...

0.77307427 0.6395133 ]

1.3

Overview of GPU Computing

Graphics Processing Units (GPUs) were originally designed for the primary purpose of accelerating image rendering tasks. However, due to their highly parallel structure, GPUs have evolved to serve broader computational purposes beyond graphics rendering. GPU computing leverages this parallelism, allowing a significant acceleration in a wide range of computational tasks by offloading portions of the code from the Central Processing Unit (CPU) to the GPU. This section delves into the architecture of GPUs, the fundamental principles of GPU computing, and their implications for modern computing.

GPU Architecture

GPUs differ from CPUs in several key areas related to their architecture. While CPUs are optimized for single-thread performance, focusing on minimizing the latency of individual tasks, GPUs are optimized for parallel throughput, focusing on maximizing the number of simultaneous tasks that can be executed. This is achieved through several specific architectural designs:

Streaming Multiprocessors (SMs): GPUs contain hundreds or thousands of small cores organized into streaming multiprocessors. Each SM can execute many threads concurrently. These threads can share resources like registers and memory within the SM, allowing efficient parallel processing.

Warp Execution: The basic execution unit in a GPU is called a warp, typically consisting of 32 threads. Warps are executed in a Single Instruction, Multiple Threads (SIMT) model, where all threads of a warp execute the same instruction simultaneously but on different data.

Memory Hierarchy: GPUs have a sophisticated memory hierarchy designed to maintain high data throughput. This includes global memory (large but relatively slow), shared memory (fast but limited in size and shared among threads in an SM), and various types of cache (e.g., L1, L2).

High Bandwidth: GPUs are designed with high-bandwidth memory interfaces to handle the massive data requirements of parallel processing. Technologies like High-Bandwidth Memory (HBM) and GDDR6 significantly exceed the data transfer rates of typical CPU memory.

The combination of these architectural features enables GPUs to handle a massive number of operations concurrently, overshadowing CPUs in tasks suited to parallel execution.

Principles of GPU Computing

GPU computing, or GPGPU (General-Purpose computing on Graphics Processing Units), follows several principles to efficiently utilize the massively parallel nature of GPU architecture:

Parallelism: Exploiting parallelism is crucial for making full use of GPU resources. In CUDA programming, this involves designing algorithms that can be decomposed into numerous small tasks that can be executed concurrently.

Data Locality: Efficient use of GPU memory bandwidth and latency considerations necessitate careful management of data locality. Frequently accessed data should be placed in shared or local memory rather than global memory to reduce access times.

Memory Coalescing: Memory access patterns should be optimized so that threads access contiguous blocks of memory, a process known as memory coalescing. This results in fewer, larger memory transactions rather than many small transactions, improving efficiency.

Minimizing Divergence: Minimize thread divergence within warps; since all threads in a warp execute the same instruction sequence, divergence can lead to underutilization of GPU resources. This involves structuring code to reduce conditional statements and branches that adversely affect parallel execution.

Understanding these principles allows developers to write efficient CUDA programs that leverage the full power of GPUs.

Implications for Modern Computing

The adoption of GPU computing has heralded significant advancements across various fields:

Scientific Research: GPUs have accelerated simulations and data processing in disciplines like physics, chemistry, and biology, enabling researchers to tackle larger and more complex problems. An example is the use of molecular dynamics simulations in drug discovery.

Machine Learning and AI: The parallelism of GPUs is well-suited to the demands of training large neural networks. Frameworks like TensorFlow and PyTorch leverage GPUs to significantly reduce the time required for training and inference.

Real-Time Data Processing: Applications that require real-time processing, such as video streaming, gaming, and autonomous driving, benefit from the low-latency and high-throughput characteristics of GPUs.

Financial Computing: High-frequency trading and risk assessment in finance utilize GPUs for the rapid processing of large datasets, allowing for quicker decision-making.

GPU computing represents a paradigm shift in how complex computational tasks are approached. It underscores the importance of parallel processing in achieving superior computational performance and efficiency, laying the groundwork for advancements in numerous fields.

Example: CUDA Program for Vector Addition

To illustrate the practical application of GPU computing, consider the classic example of vector addition using CUDA. The following CUDA program adds two vectors on the GPU.

#

include

<

cuda_runtime

.

h

>

#

include

<

stdio

.

h

>

__global__

void

vectorAdd

(

const

float

*

A

,

const

float

*

B

,

float

*

C

,

int

numElements

)

{

int

i

=

blockDim

.

x

*

blockIdx

.

x

+

threadIdx

.

x

;

if

(

i

<

numElements

)

{

C

[

i

]

=

A

[

i

]

+

B

[

i

];

}

}

int

main

(

void

)

{

int

numElements

=

50000;

size_t

size

=

numElements

*

sizeof

(

float

)

;

float

*

h_A

=

(

float

*)

malloc

(

size

)

;

float

*

h_B

=

(

float

*)

malloc

(

size

)

;

float

*

h_C

=

(

float

*)

malloc

(

size

)

;

for

(

int

i

=

0;

i

<

numElements

;

++

i

)

{

h_A

[

i

]

=

rand

()

/(

float

)

RAND_MAX

;

h_B

[

i

]

=

rand

()

/(

float

)

RAND_MAX

;

}

float

*

d_A

=

NULL

;

float

*

d_B

=

NULL

;

float

*

d_C

=

NULL

;

cudaMalloc

((

void

**)

&

d_A

,

size

)

;

cudaMalloc

((

void

**)

&

d_B

,

size

)

;

cudaMalloc

((

void

**)

&

d_C

,

size

)

;

cudaMemcpy

(

d_A

,

h_A

,

size

,

cudaMemcpyHostToDevice

)

;

cudaMemcpy

(

d_B

,

h_B

,

size

,

cudaMemcpyHostToDevice

)

;

int

threadsPerBlock

=

256;

int

blocksPerGrid

=

(

numElements

+

threadsPerBlock

-

1)

/

threadsPerBlock