Automatic CPU-GPU Communication

Management and Optimization

Thomas B. Jablin Prakash Prabhu James A. Jablin †
Nick P. Johnson Stephen R. Beard David I. August
Princeton University, Princeton, NJ †Brown University, Providence, RI
{tjablin, pprabhu, npjohnso, sbeard, august}@cs.princeton.edu [email protected]

Abstract memories. Unfortunately, not all communication management is

The performance benefits of GPU parallelism can be enormous, efficient; cyclic communication patterns are frequently orders of
but unlocking this performance potential is challenging. The ap- magnitude slower than acyclic patterns [15]. Transforming cyclic
plicability and performance of GPU parallelizations is limited by communication patterns to acyclic patterns is called Optimizing
the complexities of CPU-GPU communication. To address these Communication. Naı̈vely copying data to GPU memory, spawn-
communications problems, this paper presents the first fully auto- ing a GPU function, and copying the results back to CPU memory
matic system for managing and optimizing CPU-GPU communca- yields cyclic communication patterns. Copying data to the GPU
tion. This system, called the CPU-GPU Communication Man- in the preheader, spawning many GPU functions, and copying the
ager (CGCM), consists of a run-time library and a set of com- result back to CPU memory in the loop exit yields an acyclic com-
piler transformations that work together to manage and optimize munication pattern. Incorrect communication optimization causes
CPU-GPU communication without depending on the strength of programs to access stale or inconsistent data.
static compile-time analyses or on programmer-supplied annota- This paper presents CPU-GPU Communication Manager
tions. CGCM eases manual GPU parallelizations and improves the (CGCM), the first fully automatic system for managing and op-
applicability and performance of automatic GPU parallelizations. timizing CPU-GPU communication. Automatically managing and
For 24 programs, CGCM-enabled automatic GPU parallelization optimizing communication increases programmer efficiency and
yields a whole program geomean speedup of 5.36x over the best program correctness. It also improves the applicability and perfor-
sequential CPU-only execution. mance of automatic GPU parallelization.
CGCM manages and optimizes communication using two parts,
Categories and Subject Descriptors D.1.3 [Programming Tech- a run-time library and a set of compiler passes. To manage com-
niques]: Concurrent Programming—Parallel Programming munication, CGCM’s run-time library tracks GPU memory alloca-
tions and transfers data between the CPU memory and GPU mem-
General Terms Algorithms, Experimentation, Performance ory. The compiler uses the run-time library to manage and opti-
Keywords GPU, communication, management, optimization mize CPU-GPU communication without strong analysis. By rely-
ing on the run-time library, the compiler postpones, until run-time,
questions that are difficult or impossible to answer statically. Three
1. Introduction novel compiler passes for communication optimization leverage
Currently, even entry-level PCs are equipped with GPUs capable the CGCM run-time: map promotion, alloca promotion, and glue
of hundreds of GFLOPS. Real applications, parallelized to take ad- kernels. Map promotion transforms cyclic CPU-GPU communica-
vantage of GPUs, regularly achieve speedups between 4x and 100x tion patterns into acyclic communication patterns. Alloca promo-
[8, 10, 20]. Unfortunately, parallelizing code for GPUs is difficult tion and glue kernels improve the applicability of map promotion.
due to the typical CPU-GPU memory architecture. The GPU and The contributions of CGCM over prior work are:
CPU have separate memories, and each processing unit may effi- • The first fully automatic CPU-GPU communication manage-
ciently access only its own memory. When programs running on the ment system.
CPU or GPU need data-structures outside their memory, they must • The first fully automatic CPU-GPU communication optimiza-
explicitly copy data between the divided CPU and GPU memories. tion system.
The process of copying data between these memories for correct
Figure 1 shows a taxonomy of CPU-GPU communication man-
execution is called Managing Communication. Generally, program-
agement techniques. No prior work fully automates CPU-GPU
mers manage CPU-GPU communication with memcpy-style func-
communication, but several semi-automatic techniques can manage
tions. Manually managing CPU-GPU communication is tedious
communication if programmers supply annotations [12, 24, 26].
and error-prone. Aliasing pointers, variable sized arrays, jagged ar-
Some of these communication management techniques are strongly
rays, global pointers, and subversive typecasting make it difficult
coupled with automatic parallelization systems [12, 24]; others are
for programmers to copy the right data between CPU and GPU
not [26]. None of the semi-automatic communication systems op-
timize CPU-GPU communications. Some prior automatic paral-
lelization techniques require manual communication [3, 13, 25].
The earliest GPU parallelization systems feature manual paral-
Permission to make digital or hard copies of all or part of this work for personal or lelization and manual communication [6, 11, 16]. These systems
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
remain the most popular. CGCM enables fully-automatic commu-
on the first page. To copy otherwise, to republish, to post on servers or to redistribute nication management for manual and automatic parallelizations.
to lists, requires prior specific permission and/or a fee. Communication management is also a problem for distributed
PLDI’11, June 4–8, 2011, San Jose, California, USA. memory systems. Inspector-executor techniques automatically
Copyright c 2011 ACM 978-1-4503-0663-8/11/06. . . $10.00 manage communication for distributed memory systems [4, 14, 22]
Communication Management
Listing 1: Manual explicit CPU-GPU memory management
char *h h array[M] = {
Automatic
CGCM “What so proudly we hailed at the twilight’s last gleaming,”,
...
};

 global void kernel(unsigned i, char **d array);

Automatic

Lee [12]
Semi-

JCUDA [26]
PGI [24] void foo(unsigned N) {
/* Copy elements from array to the GPU */
 char *h d array[M];
 for(unsigned i = 0; i < M; ++i) {
CUDA [16] Baskaran [3]  size t size = strlen(h h array[i]) + 1;
Manual

 cudaMalloc(h d array + i, size);

OpenCL [11] Leung [13]  cudaMemcpy(h d array[i], h h array[i], size,
CUDA-lite [25]
 cudaMemcpyHostToDevice);
BrookGPU [6]
 }

Manual Automatic /* Copy array to the GPU */

 char **d d array;
Parallelism Extraction  cudaMalloc(&d d array, sizeof (h d array));
Figure 1. A taxonomy of related work showing automatic and  cudaMemcpy(d d array, h d array, sizeof (h d array),
 cudaMemcpyHostToDevice);
manual communication and parallelization as independent axes.
 for(unsigned i = 0; i < N; ++i)
 kernel<<<30, 128>>>(i, d d array);
but have not been used for CPU-GPU systems. Inspector-executor
techniques can reduce the number of bytes transferred, but the
/* Free the array */
overall communication pattern remains cyclic.
 cudaFree(d d array);
We have coupled CGCM with an automatic parallelizing com-
piler to produce a fully automatic GPU parallelization system. To
compare a strong cyclic communication system against CGCM’s /* Copy the elements back, and free the GPU copies */
 for(unsigned i = 0; i < M; ++i) {
acyclic communication, we adapted inspector-executor to GPUs.
 size t size = strlen(h h array[i]) + 1;
Across 24 programs, CGCM coupled with automatic paralleliza-  cudaMemcpy(h h array[i], h d array[i], size,
tion shows a geomean whole program speedup of 5.36x over se-  cudaMemcpyDeviceToHost);
quential CPU-only execution versus 0.92x for inspector-executor.  cudaFree(h d array[i]);
This paper provides a detailed description of the design and im-  }
plementation of CGCM and presents an evaluation of the system. }
Section 2 describes background information about the challenges
of CPU-GPU communications and limitations of related work. Sec-  Useful work  Communication  Kernel spawn
tion 3 describes the run-time support library. Section 4 explains
how the compiler automatically inserts calls to the CGCM library.
The map promotion, alloca promotion, and glue kernel techniques
for optimizing CPU-GPU communication appear in Section 5. Sec- Listing 2: Automatic implicit CPU-GPU memory management
tion 6 interprets the performance results for 24 programs taken from char *h h array[M] = {
the PARSEC [5], StreamIt [23], Rodinia [7], and PolyBench [17] “What so proudly we hailed at the twilight’s last gleaming,”,
benchmark suites. This section also compares CGCM to an ideal- ...
ized inspector-executor system. Section 7 discusses related work, };
and Section 8 concludes.
 global void kernel(unsigned i, char **d array);
2. Motivation
void foo(unsigned N) {
The divided CPU-GPU memories create communication manage-  for(unsigned i = 0; i < N; ++i) {
ment and optimization difficulties that motivate CGCM. Manag-  kernel<<<30, 128>>>(i, h h array);
ing communication means copying data between CPU and GPU }
memories to ensure each processing unit has the data it needs. }
Manual communication management is time-consuming and error-
prone, and semi-automatic communication management in prior  Useful work  Communication  Kernel spawn
work has limited applicability. Optimizing communication means
transforming communication patterns to remove cyclic dependen-
cies. Optimizing communication is vital for program performance,
but increases the difficulty of communication management. Lack 2.1 Communication Management
of optimized communication limits the performance of prior semi- Communication management presents a major difficulty for man-
automatic communication management frameworks. CGCM is the ual and automatic GPU parallelizations. Current GPU program-
first fully-automatic communication management and optimization ming languages, such as CUDA and OpenCL, require manual com-
system. munication management using primitive memcpy-style functions.
 Applicability
Comm. Requires Acyclic
Framework Opti. Weak Type
Annotations CPU- Aliasing Irregular Pointer Max Communication
GPU Pointers Accesses Systems Arithmetic Indirection

JCUDA [26] × Yes X X X × × ∞ No

Named Regions
× Yes X X × X × 1 No
[12]
Affine [24] × × × × With
Yes X X 1
Annotation
Inspector-Executor
× Yes × × X X × 1 No
[4, 14, 22]
After
CGCM X No X X X X X 2 Optimization

Table 1. Comparison between communication systems

Manually copying complex data-types from CPU memory to GPU Naïve Inspector−Executor Acyclic
memory is tedious and error-prone. Listing 1 shows how a CUDA Host Device Host Device Host Device

programmer might copy an array of strings to and from the GPU,

allocating and freeing memory as necessary. Almost every line of

Time
1 1 1
code in the listing involves communication management and not
2
useful computation. Furthermore, the programmer must manage
3
buffers and manipulate pointers. Buffer management and pointer
2 4
manipulation are well-known sources of bugs. 2
5
Automatic communication management avoids the difficulties
6
of buffer management and pointer manipulation, improving pro-
gram correctness and programmer efficiency. However, automati- 3
3

cally managing communication based on compile-time static anal-

ysis is impossible for general C and C++ programs. In C and C++,
any argument to a GPU function could be cast to a pointer, and 4
4
an opaque pointer could point to the middle of data-structures of
arbitrary size. Prior work restricts its applicability to avoid these Savings from
acyclic communication
difficulties [12, 24, 26]. 5
Table 1 compares the applicability of prior communication 5
management techniques. Prior GPU techniques [12, 24, 26] re-
quire programmer annotations, do not handle the full generality of 6
pointer arithmetic and aliasing, create cyclic CPU-GPU commu- 6
nication patterns by default, and do not optimize communication.
Consequently, prior automatic communication management tech-
Useful work Communication Kernel Spawn Inspector
niques have limited applicability and frequently yield poor perfor-
mance. To gain acceptance in the GPU programming community, Figure 2. Execution schedules for naı̈ve cyclic, inspector-
automatic communication management techniques must overcome executor, and acyclic communication patterns
these limitations.
2.2 Communication Optimization
Figure 2 shows execution schedules for three communication pat- elements exceeds the cost of the inspection. However, in real pro-
terns: a naı̈ve cyclic pattern, an inspector-executor pattern, and an grams the cost of sequential inspection may exceed the benefits
acyclic pattern. In the naı̈ve schedule, data transfers to and from of parallel execution. Generally, inspector-executor is cyclic since
GPU memory create cyclic dependencies, forcing the CPU and it transfers only the needed bytes for a single iteration at a time.
GPU to wait for each other. GPU programmers understand cyclic Acyclic inspector-executor variants require annotations or strong
dependencies dramatically increase execution time. Nevertheless, static analysis [18, 21].
all prior automatic GPU communication systems generate cyclic Figure 2 shows the execution schedule for acyclic CPU-GPU
communication [12, 24, 26]. Cyclic communication patterns pre- communications. Removing cyclic communication avoids the la-
vent these systems from efficiently parallelizing complex programs tency of back-and-forth communication and allows the CPU and
that launch many GPU functions. GPU to work in parallel. The performance benefit of acyclic com-
Inspector-executor systems manage communication in clusters munication is significant.
with distributed memory [4, 14, 22]. The inspector-executor ap-
proach breaks loops into an inspector, a scheduler, and an ex- 2.3 Overview
ecutor. The inspector simulates the loop to determine which ar- CGCM avoids the limitations of prior work by employing a run-
ray offsets the program reads or writes during each iteration. Af- time support library and an optimizing compiler to automatically
ter the inspection, a scheduler assigns loop iterations to cluster manage and optimize CPU-GPU communication, respectively. The
nodes and transfers the appropriate data. Executors on each clus- run-time library determines the size and shape of data-structures
ter node compute loop iterations in parallel. The inspector-executor during execution. The compiler uses the run-time library to man-
schedule in Figure 2 outperforms the naı̈ve communication sched- age memory without strong analysis and then optimizes commu-
ule because the benefit of communicating only the needed array nications to produce acyclic patterns. CGCM has two restrictions:
 units in CPU memory and allocation units in GPU memory allows
Inputs

Parallel Code
the run-time library to translate pointers.

3.1 Tracking Allocation Units

Communication
Unlike inspector-executor systems which manage memory on a
Optimization
Section 5 per-byte or per-word granularity, CGCM manages memory at the
Glue Kernel granularity of allocation units. CGCM determines which bytes to
Identification transfer by finding allocation information for opaque pointers to
Section 5.3
the stack, heap, and globals. In C and C++, an allocation unit is
a contiguous region of memory allocated as a single unit. Blocks
Communication of memory returned from malloc or calloc, local variables, and
Alloca
Promotion Management
Sections 3 and 4
global variables are all examples of allocation units. All bytes in an
Section 5.2
array of structures are considered part of the same allocation unit,
Compile Time

but two structures defined consecutively are different allocation

Map Map GPU Code units. Transferring entire allocation units between CPU and GPU
Promotion Insertion Generation memories ensures that valid pointer arithmetic yields the same re-
Section 5.1 Section 4
sults on the CPU and GPU, because the C99 programming stan-
dard [1] stipulates that pointer arithmetic outside the bounds of a
single allocation unit is undefined.
Copying an allocation unit between CPU and GPU memories
Outputs

CPU Inferred type GPU

Code Information Code requires information about the allocation unit’s base and size. The
run-time library stores the base and size of each allocation unit in
a self-balancing binary tree map indexed by the base address of
each allocation unit. To determine the base and size of a pointer’s
allocation unit, the run-time library finds the greatest key in the
Run-Time allocation map less than or equal to the pointer. Although allocation
Library
Application GPU information for global variables is known at compile-time, stack
Section 3 Functions and heap allocations change dynamically at run-time. The run-time
Instrumentation library uses different techniques to track the allocation information
Execution-Time

Section 3.1 for global, stack, and heap memory.

Allocation Map
• To track global variables, the compiler inserts calls to the run-
Section 3.2 time library’s declareGlobal function before main. Declar-
ing addresses at run-time rather than at compile-time or link-
CPU (Host) GPU (Device)
time avoids the problems caused by position independent code
and address space layout randomization.
Figure 3. High-level overview of CGCM
• To track heap allocations, the run-time library wraps around
malloc, calloc, realloc, and free. These wrappers modify
CGCM does not support pointers with three or more degrees of in- the allocation map to reflect the dynamic state of the heap at
direction, and it does not allow pointers to be stored in GPU func- run-time.
tions. Using CGCM, programmers can write the example code in • To track escaping stack variables, the compiler inserts calls
Listing 1 as Listing 2. Replacing explicit CPU-GPU communica- to declareAlloca. The registration expires when the stack
tion with CGCM yields dramatically shorter, simpler, clearer code variable leaves scope.
and prevents several classes of programmer error.
Figure 3 shows a high-level overview of CGCM’s transforma- 3.2 CPU-GPU Mapping Semantics
tion and run-time system. The run-time library provides mapping
functions which translate CPU pointers to equivalent GPU point- Table 2 lists each function in the run-time library and its arguments.
ers (Section 3). The compiler inserts mapping functions to manage The run-time library contains functions that translate between CPU
CPU-GPU communication (Section 4). Map promotion optimizes and GPU pointers. The three basic functions are map, release,
CPU-GPU communication by transferring data to the GPU early and unmap. Each of these functions operates on opaque pointers to
and keeping it there as long as possible (Section 5). Two enabling CPU memory.
transformations, glue kernels and allocation promotion, improve • Mapping a pointer from CPU to GPU memory copies the cor-
map promotion’s applicability. responding allocation unit to GPU memory, allocating memory
if necessary. The run-time library employs reference counting
3. Run-Time Library to deallocate GPU memory when necessary. Mapping a pointer
from CPU to GPU memory increases the GPU allocation unit’s
The CGCM run-time library enables automatic CPU-GPU commu- reference count.
nication management and optimization for programs with complex
• Unmapping a CPU pointer updates the CPU allocation unit
patterns of memory allocation and unreliable typing. To accomplish
this feat, the run-time library correctly and efficiently determines with the corresponding GPU allocation unit. To avoid redun-
which bytes to transfer. For correctness, the run-time library copies dant communication, the run-time library will not copy data if
data to the GPU at allocation unit granularity. A pointer’s allocation the CPU allocation unit is already up-to-date. Since only a GPU
unit comprises all bytes reachable from a pointer by valid pointer function can modify GPU memory, unmap updates each alloca-
arithmetic. Using the concept of allocation units, the run-time li- tion unit at most once after each GPU function invocation.
brary can support the full semantics of pointer arithmetic without • Releasing a CPU pointer decreases the corresponding GPU
strong static analysis. A one-to-one mapping between allocation allocation unit’s reference count, freeing it if necessary.
 Function prototype Description

Maps from host to device pointer, allocating and copying memory if

map(ptr)
necessary. Increases the allocation unit’s reference count.
Maps to host memory if the allocation unit’s epoch is not current.
unmap(ptr)
Updates the allocation unit’s epoch.
Decreases the reference count of the allocation unit. If the reference
release(ptr)
count is zero, frees resources.
Maps from host to device pointer, allocating and copying memory if
mapArray(ptr)
necessary. Increases the allocation unit’s reference count.
Maps to host memory if the allocation unit’s epoch is not current.
unmapArray(ptr)
Updates the allocation unit’s epoch.
Decreases the reference count of the allocation unit. If the reference
releaseArray(ptr)
count is zero, frees resources.
Allocates memory on the stack and registers it with the run-time
declareAlloca(size)
library.

declareGlobal(name,ptr,size,isReadOnly) Registers a global with the run-time library.

Table 2. CGCM’s run-time library interface

Listing 3: Listing 2 after the compiler inserts run-time functions Algorithm 1: Pseudo-code for map
(unoptimized CGCM). Require: ptr is a CPU pointer
char *h h array[M] = { Ensure: Returns an equivalent GPU pointer
“What so proudly we hailed at the twilight’s last gleaming,”, info ← greatestLTE(allocInfoMap, ptr)
... if info.refCount = 0 then
}; if ¬info.isGlobal then
info.devptr ← cuMemAlloc(info.size)
 global void kernel(unsigned i, char **d array); else
info.devptr ← cuModuleGetGlobal(info.name)
void foo(unsigned N) { cuMemcpyHtoD(info.devptr, info.base, info.size)
 for(unsigned i = 0; i < N; ++i) {
 char **d d array = mapArray(h h array);
info.refCount ← info.refCount + 1
 kernel<<<30, 128>>>(i, d d array); return info.devptr + (ptr − info.base)
 unmapArray(h h array);
 releaseArray(h h array);
}
} Algorithm 2: Pseudo-code for unmap
Require: ptr is a CPU pointer
 Useful work  Communication  Kernel spawn Ensure: Update ptr with GPU memory
info ← greatestLTE(allocInfoMap, ptr)
if info.epoch 6= globalEpoch ∧ ¬info.isReadOnly then
cuMemcpyDtoH(base, info.devptr, info.size)
Each of the primary run-time library functions has an array
variant. The array variants of the run-time library functions have info.epoch ← globalEpoch
the same semantics as their non-array counterparts but operate on
doubly indirect pointers. The array mapping function translates
each CPU memory pointer in the original array into a GPU memory When copying heap or stack allocation units to the GPU, map
pointer in a new array. It then maps the new array to GPU memory. dynamically allocates GPU memory, but global variables must be
Using run-time library calls, Listing 2 can be rewritten as Listing 3. copied into their associated named regions. The map function calls
cuModuleGetGlobal with the global variable’s name to get the
3.3 Implementation variable’s address in GPU memory. After increasing the reference
The map, unmap, and release functions provide the basic func- count, the function returns the equivalent pointer to GPU memory.
tionality of the run-time library. The array variations follow the The map function preserves aliasing relations in GPU memory,
same patterns as the scalar versions. since multiple calls to map for the same allocation unit yield point-
Algorithm 1 is the pseudo-code for the map function. Given ers to a single corresponding GPU allocation unit. Aliases are com-
a pointer to CPU memory, map returns the corresponding pointer mon in C and C++ code and alias analysis is undecidable. By han-
to GPU memory. The allocaInfoMap contains information about dling pointer aliases in the run-time library, the compiler avoids
the pointer’s allocation unit. If the reference count of the allocation static analysis, simplifying implementation and improving applica-
unit is non-zero, then the allocation unit is already on the GPU. bility.
Algorithm 3: Pseudo-code for release Algorithm 4: Pseudo-code for map promotion
Require: ptr is a CPU pointer forall region ∈ Functions ∪ Loops do
Ensure: Release GPU resources when no longer used forall candidate ∈ findCandidates(region) do
info ← greatestLTE(allocInfoMap, ptr) if ¬pointsToChanges(candidate, region) then
info.refCount ← info.refCount − 1 if ¬modOrRef(candidate, region) then
if info.refCount = 0 ∧ ¬info.isGlobal then copy(above(region), candidate.map)
cuMemFree(info.devptr) copy(below(region), candidate.unmap)
copy(below(region), candidate.release)
deleteAll(candidate.unmap)

The pseudo-code for the unmap function is presented in Algo-

rithm 2. Given a pointer to CPU memory, unmap updates CPU
memory with the latest state of GPU memory. If the run-time li-
brary has not updated the allocation unit since the last GPU func-
tion call and the allocation unit is not in read only memory, unmap Listing 4: Listing 3 after map promotion
copies the GPU’s version of the allocation unit to CPU memory. To char *h h array[M] = {
determine if the CPU allocation unit is up-to-date, unmap maintains “What so proudly we hailed at the twilight’s last gleaming,”,
an epoch count which increases every time the program launches a ...
};
GPU function. It is sufficient to update CPU memory from the GPU
just once per epoch, since only GPU functions alter GPU memory.
Algorithm 3 is the pseudo-code for the release function.  global void kernel(unsigned i, char **d array);
Given a pointer to CPU memory, release decrements the GPU
allocation’s reference count and frees the allocation if the reference void foo(unsigned N) {
count reaches zero. The release function does not free global  mapArray(h h array);
variables when their reference count reaches zero. Just as in CPU  for(unsigned i = 0; i < N; ++i) {
codes, it is not legal to free a global variable.  char **d d array = mapArray(h h array);
 kernel<<<30, 128>>>(i, d d array);
 releaseArray(h h array);
4. Communication Management }
 unmapArray(h h array);
CPU-GPU communication is a common source of errors for man-  releaseArray(h h array);
ual parallelization and limits the applicability of automatic par- }
allelization. A CGCM compiler pass uses the run-time library to
automatically manage CPU-GPU communications. For each GPU  Useful work  Communication  Kernel spawn
function spawn, the compiler determines which values to transfer
to the GPU using a liveness analysis. When copying values to the
GPU, the compiler must differentiate between integers and floating
point values, pointers, and indirect pointers. The C and C++ type the CPU. Finally, for each live-in pointer, the compiler inserts a call
systems are fundamentally unreliable, so the compiler uses simple to release or releaseArray to release GPU resources.
type-inference instead.
The communication management compiler pass starts with se- 5. Optimizing CPU-GPU Communication
quential CPU codes calling parallel GPU codes without any CPU-
GPU communication. All global variables share a single common Optimizing CPU-GPU communication has a profound impact on
namespace with no distinction between GPU and CPU memory program performance. The overall optimization goal is to avoid
spaces. For each GPU function, the compiler creates a list of live-in cyclic communication. Cyclic communication causes the CPU to
values. A value is live-in if it is passed to the GPU function directly wait for the GPU to transfer memory and the GPU to wait for the
or is a global variable used by the GPU. CPU to send more work. The map promotion compiler pass manip-
The C and C++ type systems are insufficient to determine which ulates calls to the run-time library to remove cyclic communication
live-in values are pointers or to determine the indirection level of patterns. After map promotion, programs transfer memory to the
a pointer. The compiler ignores these types and instead infers type GPU, then spawn many GPU functions. For most of the program,
based on usage within the GPU function, ignoring usage in CPU Communication flows one way, from CPU to GPU. The results of
code. If a value “flows” to the address operand of a load or store, GPU computations return to CPU memory only when absolutely
potentially through additions, casts, sign extensions, or other opera- necessary. The alloca promotion and glue kernels compiler passes
tions, the compiler labels the value a pointer. Similarly, if the result improve the applicability of map promotion.
of a load operation “flows” to another memory operation, the com-
piler labels the pointer operand of the load a double pointer. Since 5.1 Map Promotion
types flow through pointer arithmetic, the inference algorithm is The overall goal of map promotion is to hoist run-time library calls
field insensitive. Determining a value’s type based on use allows out of loop bodies and up the call graph. Algorithm 4 shows the
the compiler to circumvent the problems of the C and C++ type pseudo-code for the map promotion algorithm.
systems. The compiler correctly determined unambiguous types for First, the compiler scans the region for promotion candidates.
all of the live-in values to GPU functions in the 24 programs mea- A region is either a function or a loop body. Each promotion
sured. candidate captures all calls to the CGCM run-time library featuring
For each live-in pointer to each GPU function, the compiler the same pointer. Map promotion attempts to prove that these
transfers data to the GPU by inserting calls to map or mapArray. pointers point to the same allocation unit throughout the region,
After the GPU function call, the compiler inserts a call for each and that the allocation unit is not referenced or modified in the
live-out pointer to unmap or unmapArray to transfer data back to region. If successful, map promotion hoists the mapping operations
out of the target region. The specific implementation varies slightly PARSEC [5] benchmark suites. The named region technique [12]
depending on whether the region is a loop or a function. and inspector-executor [4, 14, 22] were only applicable to 80. With-
For a loop, map promotion copies map calls before the loop, out communication optimizations, many programs show limited
moves unmap after the loop, and copies release calls after the speedup or even dramatic slowdown with automatic communica-
loop. Map promotion copies the map calls rather than moving them tion management. By optimizing communication, CGCM enables
since these calls provide CPU to GPU pointer translation. Copying a whole program geomean speedup of 5.36x over best sequential
release calls preserves the balance of map and release operations. CPU-only execution.
Inserting map calls before the loop may require copying some code
from the loop body before the loop. 6.1 Experimental Platform
For a function, the compiler finds all the function’s parents in the The performance baseline is an Intel Core 2 Quad clocked at 2.40
call graph and inserts the necessary calls before and after the call GHz with 4MB of L2 cache. The Core 2 Quad is also the host
instructions in the parent functions. Some code from the original CPU for the GPU. All GPU parallelizations were executed on an
function may be copied to its parent in order to calculate the pointer NVIDIA GeForce GTX 480 video card, a CUDA 2.0 device with
earlier. 1,536 MB of global memory and clocked at 1.40 GHz. The GTX
The compiler iterates to convergence on the map promotion op- 480 has 15 streaming multiprocessors with 32 CUDA cores each,
timization. In this way, map operations can gradually climb up the for a total of 480 cores. The CUDA driver version is 3.2 release
call graph. Recursive functions are not eligible for map promotion candidate 2.
in the present implementation. The parallel GPU version is always compared with the original
CGCM optimizes Listing 3 to Listing 4. Promoting the initial single threaded C or C++ implementation running on the CPU,
mapArray call above the loop causes the run-time library to trans- even when multithreaded CPU implementations are available. All
fer h h array’s allocation units to the GPU exactly once. The sub- figures show whole program speedups, not kernel or loop speedups.
sequent calls to mapArray inside the loop do not cause additional All program codes are compiled without any alterations.
communication since the GPU version of the allocation units is al- The sequential baseline compilations are performed by the
ready active. Moving the unmapArray call below the loop allows clang compiler version 2.9 (trunk 118020) at optimization level
the run-time to avoid copying allocation units back to CPU memory three. The clang compiler produced SSE vectorized code for the
each iteration. The optimized code avoids all GPU to CPU commu- sequential CPU-only compilation. The clang compiler at optimiza-
nication inside the loop. Spawning GPU functions from the CPU is tion level three does not use automatic parallelization techniques
the only remaining communication inside the loop. The final result beyond simple vectorization.
is an acyclic communication pattern with information only flowing The nvcc compiler release 3.2, V0.2.1221 compiled all CUDA
from CPU to GPU during the loop. C and C++ programs using optimization level three.
CGCM uses the same performance flags to manage and opti-
5.2 Alloca Promotion mize communication for all programs. The optimizer runs the same
Map promotion cannot hoist a local variable above its parent func- passes with the same parameters in the same order for every pro-
tion. Alloca promotion hoists local allocation up the call graph to gram. A simple DOALL GPU parallelization system coupled with
improve map promotions applicability. Alloca promotion preallo- CGCM and an open source PTX backend [19] performed all auto-
cates local variables in their parents’ stack frames, allowing the matic parallelizations.
map operations to climb higher in the call graph. The alloca promo-
tion pass uses similar logic to map promotion, potentially copying 6.2 Benchmark Suites
code from child to parent to calculate the size of the local variable PolyBench [2, 9] is a suite composed of programs designed to
earlier. Like map promotion, alloca promotion iterates to conver- evaluate implementations of the polyhedral model of DOALL par-
gence. allelism in automatic parallelizing compilers. Prior work on au-
tomatic GPU parallelization reports impressive performance on
5.3 Glue Kernels kernel-type micro-benchmarks without communication optimiza-
Sometimes small CPU code regions between two GPU functions tion. The jacobi-2d-imper, gemm, and seidel programs have
prevent map promotion. The performance of this code is inconse- been popular targets for evaluating automatic GPU parallelization
quential, but transforming it into a single-threaded GPU function systems [3, 12]. The simple DOALL parallelizer found opportuni-
obviates the need to copy the allocation units between GPU and ties in all of the PolyBench programs, and CGCM managed com-
CPU memories and allows the map operations to rise higher in the munication for all GPU functions. Figure 4 shows performance re-
call graph. The glue kernel optimization detects small regions of sults for the entire PolyBench suite.
code that prevent map promotion using alias analysis and lowers The Rodinia suite consists of 12 programs with CPU and GPU
this code to the GPU. implementations. The CPU implementations contain OpenMP
Interrelationships between communication optimization passes pragmas, but CGCM and the DOALL parallelizer ignore them.
imply a specific compilation schedule. Since alloca promotion and The simple DOALL parallelizer found opportunities in six of the
glue kernels improve the applicability of map promotion, the com- 12 Rodinia programs and from one selected program from the
piler schedules these passes before map promotion. The glue ker- PARSEC and StreamIt benchmark suites. CGCM managed com-
nel pass can force some virtual registers into memory, creating new munications for all functions generated by the DOALL parallelizer.
opportunities for alloca promotion. Therefore, the glue kernel op- The StreamIt benchmark suite features pairs of applications writ-
timization runs before alloca promotion, and map promotion runs ten in C and the StreamIt parallel programming language. PARSEC
last. consists of OpenMP parallelized programs for shared memory sys-
tems. The eight applications from Rodinia, StreamIt, and PARSEC
are larger and more realistic than the PolyBench programs.
6. Evaluation
CGCM is applicable to all 101 DOALL loops found by a simple 6.3 Results
automatic DOALL parallelizer across a selection of 24 programs Figure 4 shows whole program speedup over sequential CPU-
drawn from the PolyBench [17], Rodinia [7], StreamIt [23], and only execution for inspector-executor, unoptimized CGCM, opti-
 PolyBench Rodinia Other
Whole Program Speedup (Log2)

256x
128x Inspector-Executor
64x Unoptimized CGCM
32x Optimized CGCM
Manual
16x
8x
4x
2x
1x
0.5x
0.25x
adi

atax

bicg

correlation

convariance

doitgen

gemm

gemver

gesumv

gramschmidt

jacobi-2d-imper

lu

ludcmp

seidel

2mm

3mm

cfd

hotspot

kmeans

lud

nw

srad

fm

blackscholes

geomean

geomean (min 1x)

Figure 4. Whole program speedup of inspector-executor, unoptimized CGCM, optimized CGCM, and manual parallelizations over sequen-
tial CPU-only execution.

Performance Study Applicability Study Manual

Program Suite Limiting
GPU Communication Total Inspector- Named Prior
Factor CGCM
Kernels Executor Regions Work
UnOpti. Opti. UnOpti. Opti.
adi PolyBench GPU 0.02% 100.00% 99.98% 0.00% 7 7 7 7
atax PolyBench Comm. 0.28% 0.28% 98.20 98.44% 3 3 3 3
bicg PolyBench Comm. 4.36% 4.46% 72.38% 74.15% 2 2 2 2
correlation PolyBench GPU 87.49% 87.39% 10.17% 10.12% 5 5 5 5
covariance PolyBench GPU 77.12% 77.28% 18.61% 18.43% 4 4 4 4
doitgen PolyBench GPU 87.48% 87.52% 11.29% 11.20% 3 3 3 3
gemm PolyBench GPU 73.49% 73.76% 19.69% 19.49% 4 4 4 4
gemver PolyBench Comm. 4.06% 4.10% 88.21% 89.36% 5 5 5 5
gesummv PolyBench Comm. 6.17% 6.29% 86.17% 86.74% 2 2 2 2
gramschmidt PolyBench Comm. 1.82% 8.37% 98.18% 90.91% 3 3 3 3
jacobi-2d-imper PolyBench GPU 7.20% 95.97% 92.82% 3.32% 3 3 3 3
seidel PolyBench Other 0.01% 0.01% 0.59% 0.59% 1 1 1 1
lu PolyBench GPU 0.41% 88.05% 99.59% 7.02% 3 3 2 2
ludcmp PolyBench GPU 1.23% 87.38% 98.10% 4.13% 5 5 3 3
2mm PolyBench GPU 75.53% 77.25% 17.96% 18.25% 7 7 7 7
3mm PolyBench GPU 78.75% 79.29% 17.86% 17.85% 10 10 10 10
cfd Rodinia GPU 4.65% 77.96% 85.90% 0.16% 9 9 3 3 [7]
hotspot Rodinia GPU 2.78% 71.57% 92.60% 0.89% 2 2 1 1 [7]
kmeans Rodinia Other 0.65% 0.00% 10.84% 0.05% 2 2 2 2 [7]
lud Rodinia GPU 3.77% 63.57% 91.56% 0.39% 6 6 1 1 [7]
nw Rodinia Other 0.00% 2.44% 100.00% 24.19% 4 4 2 2 [7]
srad Rodinia Other 0.00% 27.08% 100.00% 6.20% 6 6 1 1 [7]
fm StreamIt Other 0.00% 0.00% 0.00% 0.00% 4 4 4 4
blackscholes PARSEC Other 1.74% 3.23% 45.84% 0.96% 1 1 0 0

Table 3. Summary of program characteristics including: program suite, limiting factor for performance, the contributions of GPU and
communication time to total execution time as a percentage, the number of applicable kernels for the CGCM, Inspector-Executor, and
Named Region management techniques, and a citation for prior manual parallelizations.

mized CGCM, and a manual parallelization if one exists. The fig- 0.71x for unoptimized CGCM, and 5.36x for optimized CGCM.
ure’s y-axis starts at 0.25x, although some programs have lower Taking the greater of 1.0x or the performance of each application
speedups. Table 3 shows additional details for each program. The yields geomeans of 1.53x for inspector-executor, 2.81x for unopti-
geomean whole program speedups over sequential CPU only exe- mized CGCM, and 7.18x for optimized CGCM.
cution across all 24 applications are 0.92x for inspector-executor,
 Before optimization, most programs show substantial slow- sequential CPU-only execution. These applications have reached
down. The srad program has a slowdown of 4,437x and nw has the limit of Amdahl’s Law for the current parallelization.
a slowdown of 1,126x. By contrast, ludcmp’s slowdown is only The manual Rodinia parallelizations involved complex algorith-
4.54x. After optimization, most programs show performance im- mic improvements. For example, in hotspot the authors replace
provements and none have worse performance. However, several the original grid-based simulation with a simulation based on the
fail to surpass the CPU-only sequential versions. For comparison, pyramid method. Surprisingly, the simple automatic parallelization
we simulate an idealized inspector-executor system. The inspector- coupled with CGCM is competitive with expert programmers using
executor system has an oracle for scheduling and transfers exactly algorithmic transformations. Table 3 explains why. Programmers
one byte between CPU and GPU for each accessed allocation unit. tend to optimize a program’s hottest loops, but ignore the second
A compiler creates the inspector from the original loop [4]. To mea- and third tier loops which become important once the hottest loops
sure performance ignoring applicability constraints, the inspector- scale to thousands of threads. Automatic GPU parallelizations sub-
executor simulation ignores its applicability guard. CGCM outper- stitutes quantity for quality, profiting from Amdahl’s Law.
forms this idealized inspector-executor system. The disadvantages
of sequential inspection and frequent synchronization were not 7. Related Work
overcome by transferring dramatically fewer bytes.
Figure 4 shows performance results for automatic paralleliza- Although there has been prior work on automatic paralleliza-
tion coupled with automatic communication management. Across tion and semi-automatic communication management for GPUs,
all 24 applications, communication optimizations never reduce per- these implementations have not addressed the problems of fully-
formance. This is a surprising result since the glue kernel optimiza- automatic communication management and optimization.
tion has the potential to lower performance, and CGCM’s imple- CUDA-lite [25] translates low-performance, naı̈ve CUDA func-
mentation lacks a performance guard. Communication optimiza- tions into high performance code by coalescing and exploiting GPU
tion improves the performance for five of the sixteen PolyBench shared memory. However, the programmer must insert transfers to
programs and six of the eight other programs. For many PolyBench the GPU manually.
programs, the outermost loop executes on the GPU, so there are no “C-to-CUDA for Affine Programs” [3] and “A mapping path
loops left on the CPU for map promotion to target. Therefore, op- for GPGPU” [13] automatically transform programs similar to the
timization only improve performance for six of the 16 PolyBench PolyBench programs into high performance CUDA C using the
programs. polyhedral model. Like CUDA-lite, they require the programmer
Table 3 shows the number of GPU kernels created by the to manage memory.
DOALL parallelizer. For each DOALL candidate, CGCM auto- “OpenMP to GPGPU” [12] proposes an automatic compiler
matically managed communication correctly without programmer for the source-to-source translation of OpenMP applications into
intervention. Unlike CGCM, the parallelizer requires static alias CUDA C. Most programs do not have OpenMP annotations. Fur-
analysis. In practice, CGCM is more applicable than the simple thermore, these annotations are time consuming to add and not
DOALL transformation pass. performance portable. Their system automatically transfers named
The table also shows the applicability of named regions [12] and regions between CPU and GPU using two passes. The first pass
inspector-executor management systems [4, 14, 22]. Affine com- copies all named annotated regions to the GPU for each GPU func-
munication management [24] has the same applicability as named tion, and the second cleanup pass removes all the copies that are
regions, but a different implementation. The named region and not live-in. The two passes acting together produce a communica-
inspector-executor techniques require that each of the live-ins is tion pattern equivalent to unoptimized CGCM communication.
a distinct named allocation unit. The named regions technique also JCUDA [26] uses the Java type system to automatically trans-
requires induction-variable based array indexes. The named region fer GPU function arguments between CPU and GPU memories.
and inspector-executor systems are applicable to 66 of 67 kernels in JCUDA requires an annotation indicating whether each parameter
the PolyBench applications. However, they are applicable to only is live-in, live-out, or both. Java implements multidimensional ar-
14 of 34 kernels from the more complex non-PolyBench applica- rays as arrays of references. JCUDA uses type information to flatten
tions. Although inspector-executor and named region based tech- these arrays to Fortran-style multidimensional arrays but does not
niques have different applicability guards, they both fail to transfer support recursive data-types.
memory for the same set of kernels. The PGI Fortran and C compiler [24] features a mode for semi-
Table 3 shows the GPU execution and communication time as automatic parallelization for GPUs. Users target loops manually
a percent of total execution time. The contributions of CPU exe- with a special keyword. The PGI compiler can automatically trans-
cution and IO are not shown. This data indicates the performance fer named regions declared with C99’s restrict keyword to the
limiting factor for each program: either GPU execution, commu- GPU and back by determining the range of affine array indices.
nication, or some other factor (CPU or IO). GPU execution time The restrict keyword marks a pointer as not aliasing with other
dominates total execution time for 13 programs, ten from Poly- pointers. By contrast, CGCM is tolerant of aliasing and does not
bench and three from other applications. The simpler PolyBench require programmer annotations. The PGI compiler cannot paral-
programs are much more likely to be GPU performance bound than lelize loops containing general pointer arithmetic, while CGCM
the other more complex programs. GPU-bound programs would preserves the semantics of pointer arithmetic. Unlike CGCM, the
benefit from more efficient parallelizations, perhaps using the poly- PGI compiler does not automatically optimize communication
hedral model. Communication limits the performance of five pro- across GPU function invocations. However, programmers can use
grams, all from PolyBench. The only application where inspector- an optional annotation to promote communication out of loops.
executor outperforms CGCM, gramschmidt, falls in this category. Incorrectly using this annotation will cause the program to access
Finally, six programs, one from PolyBench and five from else- stale or inconsistent data.
where, are neither communication nor GPU performance bound. Inspector-executor systems [18, 21] create specialized inspec-
Improving the performance of these applications would require par- tors to identify precise dependence information among loop iter-
allelizing more loops. Two of the applications that are neither GPU ations. Some inspector-executor systems achieve acyclic commu-
nor communication-bound, srad and blackscholes, outperform nication when dynamic dependence information is reusable. This
condition is rare in practice. Salz et al. assume a program annota-
tion to prevent unsound reuse [21]. Rauchwerger et al. dynamically
check relevant program state to determine if dependence informa- [6] I. Buck, T. Foley, D. Horn, J. Sugerman, K. Fatahalian, M. Houston,
tion is reusable [18]. The dynamic check requires expensive se- and P. Hanrahan. Brook for GPUs: Stream computing on graphics
quential computation for each outermost loop iteration. If the check hardware. ACM Transactions on Graphics, 23, 2004.
fails, the technique defaults to cyclic communication. [7] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S.-H. Lee,
and K. Skadron. Rodinia: A benchmark suite for heterogeneous
computing. 2009.
8. Conclusion [8] D. M. Dang, C. Christara, and K. Jackson. GPU pricing of exotic
CGCM is the first fully automatic system for managing and opti- cross-currency interest rate derivatives with a foreign exchange volatil-
ity skew model. SSRN eLibrary, 2010.
mizing CPU-GPU communication. CPU-GPU communication is a [9] P. Feautrier. Some efficient solutions to the affine scheduling problem:
crucial problem for manual and automatic parallelizations. Manu- I. one-dimensional time. International Journal of Parallel Program-
ally transferring complex data-types between CPU and GPU mem- ming (IJPP), 1992.
ories is tedious and error-prone. Cyclic communication constrains [10] D. R. Horn, M. Houston, and P. Hanrahan. Clawhmmer: A streaming
the performance of automatic GPU parallelizations. By managing HMMer-Search implementation. Proceedings of the Conference on
and optimizing CPU-GPU communication, CGCM eases manual Supercomputing (SC), 2005.
GPU parallelizations and improves the performance and applica- [11] Khronos Group. The OpenCL Specification, September 2010.
bility of automatic GPU parallelizations. [12] S. Lee, S.-J. Min, and R. Eigenmann. OpenMP to GPGPU: a compiler
framework for automatic translation and optimization. In Proceed-
CGCM has two parts, a run-time library and an optimizing com-
ings of the Fourteenth ACM SIGPLAN Symposium on Principles and
piler. The run-time library’s semantics allow the compiler to man- Practice of Parallel Programming (PPoPP), 2009.
age and optimize CPU-GPU communication without programmer [13] A. Leung, N. Vasilache, B. Meister, M. M. Baskaran, D. Wohlford,
annotations or heroic static analysis. The compiler breaks cyclic C. Bastoul, and R. Lethin. A mapping path for multi-GPGPU accel-
communication patterns by transferring data to the GPU early in erated computers from a portable high level programming abstraction.
the program and retrieving it only when necessary. CGCM out- In Proceedings of the 3rd Workshop on General-Purpose Computation
performs inspector-executor systems on 24 programs and enables on Graphics Processing Units (GPGPU), pages 51–61, 2010.
a whole program geomean speedup of 5.36x over best sequential [14] S.-J. Min and R. Eigenmann. Optimizing irregular shared-memory
CPU-only execution. applications for clusters. In Proceedings of the 22nd Annual Interna-
tional Conference on Supercomputing (SC). ACM, 2008.
[15] NVIDIA Corporation. CUDA C Best Practices Guide 3.2, 2010.
Acknowledgments [16] NVIDIA Corporation. NVIDIA CUDA Programming Guide 3.0,
February 2010.
We thank the entire Liberty Research Group for their support and
[17] L.-N. Pouchet. PolyBench: The Polyhedral Benchmark suite.
feedback during this work. We also thank Helge Rhodin for gen- http://www-roc.inria.fr/ pouchet/software/polybench/download.
erously contributing his PTX backend. Additionally, we thank the [18] L. Rauchwerger, N. M. Amato, and D. A. Padua. A scalable method
anonymous reviewers for their insightful comments. for run-time loop parallelization. International Journal of Parallel
This material is based on work supported by National Science Programming (IJPP), 26:537–576, 1995.
Foundation Grants 0964328 and 1047879, and by United States Air [19] H. Rhodin. LLVM PTX Backend.
Force Contract FA8650-09-C-7918. James A. Jablin is supported http://sourceforge.net/projects/llvmptxbackend.
by a Department of Energy Office of Science Graduate Fellowship [20] S. Ryoo, C. I. Rodrigues, S. S. Baghsorkhi, S. S. Stone, D. B. Kirk, and
(DOE SCGF). W.-m. W. Hwu. Optimization principles and application performance
evaluation of a multithreaded GPU using CUDA. In Proceedings of
the Thirteenth ACM SIGPLAN Symposium on Principles and Practice
References of Parallel Programming (PPoPP), 2008.
[1] ISO/IEC 9899-1999 Programming Languages – C, Second Edition, [21] J. Saltz, R. Mirchandaney, and R. Crowley. Run-time parallelization
1999. and scheduling of loops. IEEE Transactions on Computers, 40, 1991.
[2] C. Ancourt and F. Irigoin. Scanning polyhedra with DO loops. In [22] S. D. Sharma, R. Ponnusamy, B. Moon, Y.-S. Hwang, R. Das, and
Proceedings of the Third ACM SIGPLAN Symposium on Principles J. Saltz. Run-time and compile-time support for adaptive irregular
and Practice of Parallel Programming (PPoPP), 1991. problems. In Proceedings of the Conference on Supercomputing (SC).
[3] M. M. Baskaran, J. Ramanujam, and P. Sadayappan. Automatic C-to- IEEE Computer Society Press, 1994.
CUDA code generation for affine programs. In Compiler Construction [23] StreamIt benchmarks. http://compiler.lcs.mit.edu/streamit.
(CC), 2010. [24] The Portland Group. PGI Fortran & C Accelator Programming Model.
[4] A. Basumallik and R. Eigenmann. Optimizing irregular shared- White Paper, 2010.
memory applications for distributed-memory systems. In Proceedings [25] S.-Z. Ueng, M. Lathara, S. S. Baghsorkhi, and W.-m. W. Hwu. CUDA-
of the Eleventh ACM SIGPLAN Symposium on Principles and Practice Lite: Reducing GPU Programming Complexity. In Proceeding of the
of Parallel Programming (PPoPP), 2006. 21st International Workshop on Languages and Compilers for Parallel
[5] C. Bienia, S. Kumar, J. P. Singh, and K. Li. The PARSEC benchmark Computing (LCPC), 2008.
suite: characterization and architectural implications. In Proceedings [26] Y. Yan, M. Grossman, and V. Sarkar. JCUDA: A programmer-friendly
of the 17th International Conference on Parallel Architectures and interface for accelerating Java programs with CUDA. In Proceedings
Compilation Techniques (PACT), 2008. of the 15th International Euro-Par Conference on Parallel Processing.
Springer-Verlag, 2009.