Graphics Hardware (2007)

Timo Aila and Mark Segal (Editors)

Scan Primitives for GPU Computing

Shubhabrata Sengupta, Mark Harris , Yao Zhang, and John D. Owens

University of California, Davis  NVIDIA Corporation

Abstract

The scan primitives are powerful, general-purpose data-parallel primitives that are building blocks for a broad
range of applications. We describe GPU implementations of these primitives, specifically an efficient formulation
and implementation of segmented scan, on NVIDIA GPUs using the CUDA API. Using the scan primitives, we
show novel GPU implementations of quicksort and sparse matrix-vector multiply, and analyze the performance
of the scan primitives, several sort algorithms that use the scan primitives, and a graphical shallow-water fluid
simulation using the scan framework for a tridiagonal matrix solver.

1. Introduction and Motivation stream programming model in which a single program (a

By the end of 2007, the most advanced processors will sur- kernel) operates in parallel on each input element, pro-
pass one billion transistors on a single chip. This wealth of ducing one output element for each input element. The
computational resources has yielded GPUs that can sustain programmable fragment and vertex parts of the graphics
hundreds of GFLOPS on not just graphics applications but pipeline precisely match this strict stream programming
also a wide variety of computationally demanding general- model. The stream model extends nicely to problems where
purpose problems. each output element is a function not just of a single in-
The primary reason that GPUs deliver such high per- put but of a small, bounded neighborhood of inputs (a
formance is that the GPU is a highly parallel machine. “neighborhood-access iterator”). Most general-purpose ap-
NVIDIA’s latest flagship GPU, for instance, boasts 128 pro- plications that have been mapped efficiently to GPUs fit
cessors. GPUs keep these processors busy by juggling thou- nicely into this more general stream programming model
sands of parallel computational threads. Graphics workloads (for instance, particle systems, image processing, grid-based
are perfectly suited to delivering large amounts of fine- fluid simulations, and dense matrix algebra).
grained parallel work. The programmable parts of the graph-
ics pipeline operate on primitives (vertices and fragments), 1.1. More Complex Operations are Needed
with hundreds of thousands to millions of each type of prim- Consider a fragment program that operates on n fragments.
itive in a typical frame. These primitive programs spawn a With a single-access iterator, each output fragment must ac-
thread for each primitive to keep the parallel processors full. cess a single input fragment; with a neighborhood-access it-
It is instructive to look at the graphics programming erator, each output fragment must access a small bounded
model as the GPU becomes more general purpose. Let us number of input fragments. The total number of accesses
consider fragment processing as a representative example. necessary to compute all fragments is O(n).
In both the OpenGL and DirectX APIs, fragment processing Many interesting problems, however, have more complex
is completely data-parallel. The fragment processors iterate access requirements than we can support with a single- or
over each input fragment, producing a fixed number of out- neighborhood-access iterator. It is these problems that we
puts per fragment that depend only on the incoming frag- address in this paper. A common algorithmic pattern that
ment (Lefohn et al. call this access pattern a “single-access arises in many parallel applications with complex access re-
iterator” [LKS∗ 06]). It is this explicit data parallelism that quirements is the prefix-sum algorithm. The input to prefix-
enables the effective use of so many parallel processors in sum is an array of values. The output is an equally sized array
recent GPUs. in which each element is the sum of all values that preceded
This explicit parallelism, in turn, is well suited for a it in the input array:

Copyright  c 2007 by the Association for Computing Machinery, Inc.

Permission to make digital or hard copies of part or all of this work for personal or class-
room use is granted without fee provided that copies are not made or distributed for com-
mercial advantage and that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than ACM must be honored. Ab-
stracting with credit is permitted. To copy otherwise, to republish, to post on servers, or
to redistribute to lists, requires prior specific permission and/or a fee. Request permissions
from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail [email protected].
GH 2007, San Diego, California, August 04 - 05, 2007
c 2007 ACM 978-1-59593-625-7/07/0008 $ 5.00
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

in: 3 1 7 0 4 1 6 3 Efficient CUDA programs exploit both thread parallelism

out: 0 3 4 11 11 14 16 22 within a thread block and coarser block parallelism across
How do we compute the value of the last element in the thread blocks. Because only threads within the same block
output (22)? That value is a function of every value in the can cooperate via shared memory and thread synchroniza-
input stream. With a serial processor, such a computation is tion, programmers must partition computation into multi-
trivial, but with a parallel processor, it is more difficult. The ple blocks. While this adds complexity to the programming
naive way to compute each output in parallel is for every el- model compared to using a single partition (such as with
ement in the output stream to sum all preceding values from pixel shaders in the graphics API), the potential performance
the input stream. This approach requires O(n2 ) total memory benefits are large. For instance, we previously compared
accesses and addition operations. This cost is too high. our optimized OpenGL unsegmented scan implementation
against our CUDA implementation on the same GPU and
found a 7× speedup for large scans [HSO07].
1.2. Scan: An Efficient Parallel Primitive
We are interested in finding efficient solutions to parallel
problems in which each output requires global knowledge of 2.1. Scan
the inputs. We attack these problems using a family of algo- Scan, first proposed as part of APL [Ive62], was popular-
rithms called the scan primitives. Our scan implementation ized by Blelloch as a fundamental primitive on the Con-
has a serial work complexity of O(n). While the standard nection Machine [Ble90]. On the GPU, scan was first used
scan primitive was introduced to the GPU by Horn [Hor05], by Horn for “non-uniform stream compaction” [Hor05] as
in this paper we introduce the segmented scan primitive to part of a collision-detection application. Hensley et al. im-
the GPU, and present new approaches to implementing sev- proved Horn’s implementation in their summed-area-table
eral classic applications using the scan primitives. work [HSC∗ 05], but the serial work complexity of both Horn
and Hensley’s algorithms was O(n log n). The next year Sen-
2. Primitives gupta et al. and Greß et al. demonstrated the first O(n) GPU-
based scan implementations [SLO06, GGK06].
We chose NVIDIA’s CUDA GPU Computing environ-
ment [NVI07] for our implementation. CUDA provides a The inputs to a scan operation are a vector of data ele-
direct, general-purpose C language interface to the pro- ments and an associative binary function ⊕ with an iden-
grammable processors on NVIDIA’s 8-series GPUs, elim- tity element i† . If the input is [a0 , a1 , a2 , a3 , . . .], an exclu-
inating much of the complexity of writing GPGPU appli- sive scan produces the output [i, a0 , a0 ⊕a1 , a0 ⊕a1 ⊕a2 , . . .],
cations using graphics APIs such as OpenGL. Furthermore, while an inclusive scan produces the output [a0 , a0 ⊕a1 , a0 ⊕
CUDA exposes some important new hardware features that a1 ⊕ a2 , a0 ⊕ a1 ⊕ a2 ⊕ a3 , . . .]. Note that the scan output re-
have large benefits to the performance of data-parallel com- quires global knowledge of all inputs, as we discussed in
putations: Section 1; the output at the last element is a function of all
inputs. In this paper, we implement an exclusive sum-scan
General Load-Store Memory Architecture CUDA al-
as our fundamental primitive and generate inclusive scan by
lows arbitrary gather and scatter memory access from
adding the input vector to the exclusive output vector. We
GPU programs.
also support backward scans by reversing the input elements
On-chip Shared Memory Each multiprocessor on the
at the beginning of the scan (typically when they are read
GPU contains a fast on-chip memory (16 kB on NVIDIA
from GPU main memory).
8-series GPUs). All threads running on a multiprocessor
can load and store data from this memory. We briefly outline the details of the O(n) unsegmented
Thread Synchronization A barrier instruction is provided CUDA scan implementation we use in this paper, which is
to synchronize between all threads active on a GPU mul- more fully described in our previous work [HSO07]. The
tiprocessor. Together with shared memory, this feature al- implementation logically follows the work-efficient formu-
lows threads to cooperatively compute results. lation of Blelloch [Ble90] and the GPU implementation of
Sengupta et al. [SLO06], but is adapted for efficiency on
NVIDIA’s 8-series GPUs feature multiple physical multi-
CUDA. Our work-efficient scan of n elements requires two
processors, each with a shared memory and multiple scalar
passes over the array, called reduce and down-sweep, shown
processors (for example, the NVIDIA GeForce 8800 GTX
in Algorithm 1 and 2, respectively, and depicted in Figure 1.
has 16 multiprocessors with eight processors each). CUDA
Each of these two passes requires log n parallel steps. The
structures GPU programs into parallel thread blocks of up to
amount of work is cut in half at each step, resulting in an
512 SIMD-parallel threads. Programmers specify the num-
overall work complexity of O(n).
ber of thread blocks and threads per block, and the hardware
and drivers map thread blocks to parallel multiprocessors on
the GPU. Within a thread block, threads can communicate
through shared memory and cooperate by combining shared † Typical binary functions are plus [with identity 0], min, max, log-
memory with thread synchronization. ical and, and logical or.


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

threads 1: x[n − 1] ← 0
2: for d = log2 n − 1 down to 0 do
0 1 2 3 4 5 6 7 3: for all k = 0 to n − 1 by 2d+1 in parallel do
4: t ← x[k + 2d − 1]
5: x[k + 2d − 1] ← x[k + 2d+1 − 1]
a 6: x[k + 2d+1 − 1] ← t + x[k + 2d+1 − 1]

reduce
Algorithm 2: The down-sweep phase of a work-efficient
parallel unsegmented scan algorithm.
i j

2.2. Segmented Scan

2nd-level The major contribution of this paper is the introduction of
steps

segmented the segmented scan primitive [Sch80] to the GPU. Seg-

scan mented scan generalizes the scan primitive by allowing par-
allel scans on arbitrary partitions (“segments”) of the in-
put vector. Segments are demarcated by flags, where a set
w x k flag marks the first element of a segment. Our segmented
downsweep

scan implementation has the same O(n) complexity as scan

and is only three times slower than scan. Segmented scan
is useful as a building block for a broad variety of applica-
y z tions [Ble90] that have not previously been efficiently im-
plemented on GPUs.

2.2.1. Work-efficient segmented scan algorithm

Our work-efficient segmented scan implementation follows
the high-level structure of our work-efficient implementation
Figure 1: Multi-block segmented scan communication. Cells of scan [HSO07], shown in Figure 1. Squares that have the
with the same shading belong to the same block. The exam- same color (light gray or white) represent threads that be-
ple above processes 4 threads per block and requires three long to the same block. The arrows show data movement.
steps: 2 blocks running in parallel for the reduce step, 1 Two arrows entering the same square signify a binary op-
block to combine the results from the reduce step and pre- eration, which can be a copy, a binary scan operation, or a
pare for the downsweep, and 2 blocks running in parallel logical OR. In the discussion below, we use sum as the bi-
in the downsweep. Letters refer to the discussion in Sec- nary scan operation, but the algorithm works for any other
tion 2.2.1. binary associative operator.
Our contribution is to extend the reduce and down-sweep
phases of the unsegmented scan algorithm to efficiently im-
plement segmented scan on the GPU. Though Schwartz first
presented the concept of segmented scans [Sch80], he did
Each thread processes two elements; if the number of el- not describe how segmented scan can be implemented using
ements exceeds the maximum number that a single thread scan’s balanced-tree approach. Chatterjee et al.’s implemen-
block can process, the array is divided across multiple thread tation of segmented scan [CBZ90] was closely tied to the
blocks and the partial sum tree results are used as input to a Cray-MP architecture. They used the wide vector registers
second-level recursive scan. Each element of the output of to carve up large input arrays into smaller chunks. Within
this scan is then added to all elements of the corresponding each block the segmented scan ran serially. The flags were
block in the first-level scan. This recursion continues as nec- stored in vector mask registers of the Cray-MP which made
essary for scans of very large arrays. accessing them quite efficient using the merge operation.
We also show how to factor the algorithm into chunks,
1: for d = 0 to log2 n − 1 do which is necessary when the input vector is too long to be
2: for all k = 0 to n − 1 by 2d+1 in parallel do processed in shared memory by a single thread block. The
3: x[k + 2d+1 − 1] ← x[k + 2d − 1] + x[k + 2d+1 − 1] simple parallelization structure of unsegmented scan is not
directly applicable to segmented scan.
Algorithm 1: The reduce (up-sweep) phase of a work-
Just as in scan, the segmented reduce phase traverses a
efficient parallel unsegmented scan algorithm.
binary tree with n leaves and d = log2 n levels with 2d nodes
each. Reduce traverses up the tree from the leaves to the root


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

computing partial-sum results at each internal node of the Head flag representation and API In our implementation,
tree. For example, in Figure 1, a is the sum of the values in segments are denoted by a head flag vector that has the same
threads 2 and 3. Segmented scan must compute intermediate length as the input vector. If an element in the input vector
results for both the data and the head flags. The partial OR is the first element of a segment, the corresponding entry in
flag is simply the logical OR of two input head flags. The the head flag is set to 1. All other entries in the head flag
partial data sums are computed just as in unsegmented scan vector are set to 0. Even though head flags appear on top
unless the right parent’s head flag is set, in which case the of elements in this case, it is conceptually easier to think of
right parent’s data element is taken unmodified. Pseudocode head flags as situated between two elements, the first element
for the reduce phase is shown in Algorithm 3. of the current segment and the last element of the previous
segment (going from left to right). This distinction becomes
1: for d = 1 to log2 n − 1 do important in the case of backward segmented scan where
2: for all k = 0 to n − 1 by 2d+1 in parallel do simply flipping the flags is conceptually different from doing
3: if f [k + 2d+1 − 1] is not set then the segmented scan from right to left. In this case, the flags
4: x[k + 2d+1 − 1] ← x[k + 2d − 1] + x[k + 2d+1 − 1] must be shifted right by one.
5: f [k + 2d+1 − 1] ← f [k + 2d − 1] | f [k + 2d+1 − 1] flag: 0 0 0 1 0 0 0 0
data: 1 2 3 4 5 6 7 8
Algorithm 3: The reduce (up-sweep) phase of a segmented
scan algorithm. x denotes the partial sums and f denotes the flip-flag: 0 0 0 0 1 0 0 0
partial OR flags. flip-data: 8 7 6 5 4 3 2 1
We see that flipping the flag and the data isn’t the same
As in scan, the down-sweep phase traverses back down as walking from right to left since the segment should start
the tree from the root using the partial sums from the reduce from 3 but instead starts from 4. Hence the flags need to be
phase. The key difference from the unsegmented scan algo- shifted one position to the right after flipping. Now we can
rithm is that the right child (z in Figure 1) is not always set use the segmented scan algorithm to do backward segmented
to the sum of its parent’s value (x) and the former value of its scan by flipping the final result at the end.
left sibling (w). The right child (z) of x is set to 0 (the iden-
tity element) if the flag in the position to the right of x is set Efficient storage of flags Since flags are Boolean values, it
in the input flag vector. Otherwise, if the partial OR stored is space-inefficient to store them as 4-byte words. We orig-
in the same position in the tree is set, the right child (z) is set inally intended to pack 32 flags into a single integer. How-
to the original value of its left sibling (w). If neither flag is ever, the read-modify-write semantics of parallel threads in
set, the right child (z) receives the sum of its parent (x) and CUDA preclude this approach (Section 5).
the original value of its left sibling (w). The pseudocode for
the down-sweep phase is shown in Algorithm 4.
f0 | f32 | f64 | f96 f1 | f33 | f65 | f97 ··· f31 | f63 | f95 | f127
        
1: x[n − 1] ← 0 word 0 word 1 word 31
2: for d = log2 n − 1 down to 0 do
Instead, we store one flag per byte as shown in the figure
3: for all k = 0 to n − 1 by 2d+1 in parallel do
above. To reduce shared memory bank conflicts, we allocate
4: t ← x[k + 2d − 1]
flags in chunks of 32 contiguous 4-byte words with four flags
5: x[k + 2d − 1] ← x[k + 2d+1 − 1]
per word. Our choice of 32 reflects the 32-thread SIMD warp
6: if fi [k + 2d ] is set then
size in NVIDIA 8-Series GPUs. We stripe the flags across
7: x[k + 2d+1 − 1] ← 0
the 32 words in a chunk with a spacing of four bytes between
8: else if f [k + 2d − 1] is set then
consecutive flags, wrapping to the beginning of the chunk
9: x[k + 2d+1 − 1] ← t
every 32 flags.
10: else
11: x[k + 2d+1 − 1] ← t + x[k + 2d+1 − 1] Our implementation may benefit from other head flag rep-
12: Unset flag f [k + 2d − 1] resentations. Blelloch proposes two [Ble90]: a vector of seg-
ment lengths and a vector of head pointers (similar to row-
Algorithm 4: The down-sweep phase of a segmented scan Ptr in Section 4.2). Head flags have two advantages over
algorithm. these representations; first, they are associated directly with
each element rather than being stored in a separate data
structure; second, these separate data structures have a dif-
2.2.2. Segmented scan implementation ferent size from the vector of data elements and are thus
Besides the algorithmic difference, our segmented scan im- harder to parallelize across thread blocks. Nonetheless, the
plementation differs from our scan implementation [HSO07] difficulties we had with storing and computing head flags
in two major ways: the representation and storage of flags motivate continued investigation into alternate segmented
and the merging of results for multi-block segmented scans. representations.


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

Multi-block segmented scan In contrast to unsegmented 2.3.1. Enumerate

scan, segmented scan results cannot be propagated across Enumerate’s inputs are a vector and a true/false value per el-
blocks by a uniform add. For segmented scan, the add op- ement in the vector. Enumerate can be used with either seg-
eration operates on only the first segment of each block. To mented or non-segmented inputs. The output for each input
make this work, we implement segmented scan using sepa- element is a count of the number of true elements to the left
rate reduce and down-sweep CUDA kernels. of that element:
At the end of the reduce step, we write the partial sum
enumerate([t f f t f t t]) = [0 1 1 1 2 2 3]
tree and the partial OR tree to global memory. This saves
the state so that the down-sweep step can be started later. At We implement enumerate by setting a 1 for each true el-
this point we do a second-level segmented scan. The inputs ement and a 0 for each false element in a temporary vector,
to the second-level segmented scan are the flags and data then (exclusive) scanning that vector.
from the last element of the partial sum and partial OR tree Enumerate is useful in pack (compact) operations, where
of each block. In addition, the second-level segmented scan we only wish to keep elements marked as true. For each true
takes as a third input the first element of the flag vector of element, the output of enumerate is the address to which the
each block‡ . Thus if we have B blocks, each input vector is output must be scattered [Hor05].
B elements long. This multi-block data movement is shown
in Figure 1 by the arrows leading out of i and j, which are the 2.3.2. Distribute (copy)
last elements of their respective blocks.
Like enumerate, distribute can be used with either segmented
The down-sweep phase of the top-level segmented scan or non-segmented inputs; it can also be performed in either a
is run upon completion of the second-level segmented scan. forward or backward direction. Distribute copies the element
The state that was saved at the end of the reduce phase at the head (or tail) of the segment to all other elements in
is reloaded from global memory. Instead of assigning 0 to that segment:
the last element of each block (as in unsegmented scan), it
is assigned the corresponding element from the output of distribute([a b c][d e]) = [a a a][d d]
the second-level segmented scan. For example, the last el- For segmented inputs, we implement distribute by setting
ement of the third block is set to the third element of the all non-head elements to 0 in a temporary vector and per-
output of the second-level segmented scan. This data move- forming a segmented inclusive scan on that vector. For non-
ment is shown by the arrows leading out of the second-level segmented inputs, it is more efficient for one thread to write
segmented scan stage into x and k, which are the last ele- the head element into shared memory and all other threads
ments of their respective blocks. We now proceed with the to read that shared element.
down-sweep as in the single-block case. The multi-block
segmented scan algorithm is summarized in Algorithm 5. 2.3.3. Split and split-and-segment
Like enumerate, split takes two inputs: a vector of elements
1: Perform reduce on all blocks in parallel and a vector of true/false values. Split divides the input vec-
2: Save partial sum and partial OR trees to global memory tor into two pieces, with all the elements marked false on
3: Do second-level segmented scans with final sums the left side of the output vector and all the elements marked
4: Load partial sum and partial OR trees from global mem- true on the right. Split-and-segment operates on segmented
ory to shared memory inputs and performs a stable independent split on each seg-
5: Set last element of each block to corresponding element ment, additionally dividing each segment into two segments,
in the output of second-level segmented scan one for the falses and one for the trues:
6: Perform down-sweep on all blocks in parallel
split-and-segment([at bf ct][df et ff]) =
Algorithm 5: A multi-block segmented scan algorithm. [bf][at ct][df ff][et]
Blelloch implements split with two enumerates (requiring
two scans), one for the falses going forward, and another for
2.3. Primitives Built Atop Scan
the trues going backward. (Note the first half of the algo-
The scan primitives are used to implement several higher- rithm is essentially just like pack.) To minimize the number
level primitives that are well-suited as building blocks for of scans, we instead reduce that to one enumerate and per-
general-purpose tasks [Ble90]. Below, we outline the prim- form additional computation to derive the true addresses as
itives that we have implemented for use in the applications follows§ .
we describe in Section 4.

‡ Note that the second-level segmented scan takes three inputs in- § Split-and-segment requires an additional scan to copy the number
stead of the usual two. The third input is a partial OR tree repre- of falses in each segment to all elements in that segment. We use a
sented as a vector. For the top-level segmented scan, the flag vector similar technique to reduce Blelloch’s 3-scan split-and-segment to
and partial OR tree are identical so the third input is implicit. our 2-scan split-and-segment.


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

[t f t f f t f] # in [5 3 7 4 6] # initial input
[0 1 0 1 1 0 1] # e = set 1 in false elts. [5 5 5 5 5] # distribute pivot across segment
[0 0 1 1 2 3 3] # f = enumerate with false=1 [f f t f t] # input > pivot?
4 # add two last elts. in e, f [5 3 4][7 6] # split-and-segment
# == total # of falses [5 5 5][7 7] # distribute pivot across segment
# set as shared variable NF [t f f][t f] # input >= pivot?
[0 1 2 3 4 5 6] # each thread knows its id [3 4 5][6 7] # split-and-segment, done!
[4 5 5 6 6 6 7] # t = id - f + NF Figure 2: This quicksort example requires two passes, with
[4 0 5 1 2 6 3] # addr = e ? f : t three segmented scans per pass (one for the pivot distribute,
[f f f f t t t] # out[addr] = in (scatter) two within split-and-segment).

3. Experimental Methodology
For our results, we used an NVIDIA GeForce 8800 GTX 4. Applications
GPU connected via PCI Express 16x to an Intel Xeon
4.1. Quicksort
3.0 GHz CPU. Our applications ran atop Windows XP with
NVIDIA driver version 97.73 and the beta release of the Quicksort is a popular, efficient primitive for sorting on se-
CUDA Toolkit (version dated 14 February 2007). rial machines. Its control complexity and irregular paral-
lelism have prevented previous implementations on GPUs,
Unless otherwise noted, timing information is specified
but the segmented scan primitive leads to an elegant formu-
for GPU computation only and does not include transfer time
lation credited to Blelloch [Ble90].
to or from the GPU. We feel this best reflects the use of the
primitives we describe here, which we expect will be used as Our algorithm runs in parallel over all segments in the in-
components in larger GPU-based computations and would put. All communication between elements (threads) in the
rarely stand alone. To amortize startup costs, we typically algorithm is within a single segment, so the segmented scan
run each computation serially many times and present an av- primitives are an ideal fit. We begin by choosing a pivot el-
erage runtime for each computation. ement in each segment (we choose the first element in the
segment) and then distributing that pivot across the segment.
We then compare the input element to the pivot. On alternat-
3.1. Impact of G80 Hardware ing passes through the algorithm we compare either greater-
NVIDIA 8-Series GPUs deliver new hardware functionality than or greater-than-or-equal. The comparison produces a
that adds new capabilities to the GPGPU toolbox. From the segmented vector of trues and falses, which we use to split-
perspective of this work, the two most important are the ad- and-segment the input; smaller elements move to the head
dition of 16 kB of shared memory per multiprocessor and the of the vector and larger elements to the end. Each segment
support of generalized scatter within the GPU programmable splits into two¶ . We begin with a single segment that spans
units. the entire input and finish when the output is sorted, which
we check with a global reduction after each step. Figure 2
Shared memory The addition of shared memory enables
shows a small example.
more efficient data sharing between threads and improves
overall performance by eliminating memory traffic to
main GPU memory. However, shared memory does not 4.2. Sparse Matrix-Vector Multiply
introduce any new functionality that was not available on Matrix-based numerical computations are a good match for
previous hardware. the GPU for two reasons: first, they are computationally in-
Scatter Scatter, on the other hand, does introduce function- tense, and second, they exhibit substantial parallelism. Rep-
ality not present on older NVIDIA GPUs. While the scan resentative GPGPU efforts focusing on matrix computations
and segmented scan primitives could be implemented on include Larsen and McAllister’s 2001 dense-matrix multipli-
older hardware, scatter enables higher performance and cation study [LM01], Moravánszky’s dense matrix represen-
storage efficiency. Scatter is efficient but not necessary tation [Mor02], or Krüger and Westermann’s general linear
for such operations as pack; when the vector of scanned algebra framework [KW03].
data items is ordered, as in the enumerate operation
Sparse matrices are key components of many important
used by pack, Horn’s gather-search operation [Hor05] em-
numerical computing algorithms, including singular value
ulates scatter at the cost of log n passes. Buck summa-
rizes other methods for implementing scatter on older
GPUs [Buc05].
¶ Instead of alternating comparisons and a 2-way split, Blelloch
Scatter functionality is available in the R520 and R600 always uses the same comparison and uses a 3-way split instead.
families of AMD GPUs [PSG06]. Thus we expect that the We found the additional control complexity of a 3-way split, and
techniques we describe are also directly applicable to AMD the additional number of segmented scans it required, did not justify
GPUs. its fewer number of passes overall.


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

decomposition, conjugate gradient, and multigrid. The ap- ⎛ ⎞ ⎛ ⎞⎛ ⎞

y0 a 0 b x0
peal of a sparse matrix representation is less storage and ⎝y1 ⎠ + = ⎝c d e ⎠ ⎝x1 ⎠
computation than its dense cousin. The ideal sparse ma-
y2 0 0 f x2
trix representation only stores the non-zero elements and re-
quires neither padding (which requires extra storage space)
value = [a, b, c, d, e, f ]
nor sorting (which requires additional computation). How-
ever, the irregularity of sparse matrices makes parallelizing index = [0, 2, 0, 1, 2, 2]
operations on them difficult. rowPtr = [0, 2, 5]
The most notable current work in this area on the GPU
is from Bolz et al. [BFGS03]. Their “jagged diagonal” for-
mulation for sparse matrices [AS89] is simpler to parallelize product = [x0 a, x2 b, x0 c, x1 d, x2 e, x2 f ] (1)
than a truly sparse representation, but requires a sort of its = [[x0 a, x2 b][x0 c, x1 d, x2 e][x2 f ]] (2)
rows to place them in descending order by length. Krüger = [[x0 a + x2 b, x2 b]
and Westermann render a separate point for every four non-
[x0 c + x1 d + x2 e, x1 d + x2 e, x2 e][x2 f ]] (3)
zero entries in a row [KW03]. Brook’s spMatrixVec test
uses a compressed sparse row format but runs in parallel y = y + [[x0 a + x2 b, x0 c + x1 d + x2 e, x2 f ] (4)
on rows, so the runtime on each row is proportional to the
largest number of elements in any row [BH03]. Figure 3: Segmented scan can be used to perform sparse-
matrix vector multiplication. Our goal is to solve the top
We choose the compressed sparse row (CSR) format for equation; we represent the sparse matrix with the three
our sparse matrices. CSR requires neither preprocessing nor vectors value, index, and rowPtr. The four computa-
padding and is one of the representations of choice in the nu- tion steps at bottom, described in Section 4.2, compute the
merical computing community. Our CSR representation and sparse-matrix-vector product. Our formulation is more effi-
sparse-matrix-vector multiplication algorithm both roughly cient than previous methods, which required either sorting
follow that of Blelloch et al. [BHZ93]. value by rows [BFGS03] or wasting work when rows were
We represent an n × n CSR sparse matrix containing e not of uniform length [BH03].
non-zero elements (entries) with the following three data
structures:
4.3. Tridiagonal Matrix Solvers and Fluid Simulation
1. The e-element value vector contains all e non-zero el-
ements (entries) in the matrix, read in scan order (left to The reduce/downsweep structure of the scan primitive can
right, top to bottom). We store this as a float vector. also be applied to other problems, including the solution of
2. The e-element index vector contains an integer identi- tridiagonal systems y = T x, where the tridiagonal matrix T
fying the column for each element in the value vector. has entries only along the main diagonal and the adjacent di-
3. The n-element rowPtr vector contains an integer index agonals. Kass and Miller demonstrated the use of tridiagonal
into value that points to the first element in each row. systems for fluid simulation using a shallow-water assump-
tion [KM90], and Kass et al. later used a similar computa-
The matrix consists of these three data structures. We tion for real-time depth-of-field [KLO06]. The latter work
multiply it by a vector x of length n and add the result to was the first to use the technique of cyclic reduction to solve
a vector y of length n. With these data structures in main tridiagonal systems on the GPU. Here we implement a GPU-
GPU memory, and additional flag and product tempo- based fluid simulation algorithm using the method of Kass
rary data structures with e entries each, matrix multiplication and Miller with a GPU scan-based tridiagonal solver. This
then proceeds in four steps. We show an example in Figure 3. inexpensive, physically accurate method for shallow fluid
simulation is a viable alternative to other PDE solution meth-
1. The first kernel runs over all entries. For each entry, it ods [KW03]. Our implementation leverages our scan frame-
sets the corresponding flag to 0 and performs a mul- work for the related problem of cyclic reduction and demon-
tiplication on each entry: product = x[index] * strates that our CUDA-based code can be easily integrated
value. into a graphics application.
2. The next kernel runs over all rows and sets the head flag
Following the treatment of Kass and Miller, in our wa-
to 1 for each rowPtr in flag through a scatter. This
ter simulation (Plate 1), we describe the water surface as an
creates one segment per row.
n × m 2D array of heights. We use the alternating direction
3. We then perform a backward segmented inclusive sum
method to first solve a tridiagonal system for each of n rows
scan on the e elements in product with head flags in
in parallel, then for each of m columns in parallel. Figure 4
flag.
shows the difference in the reduce step communication pat-
4. To finish, we run our final kernel over all rows, adding the
tern between scan and the tridiagonal solver. Because the
value in y to the gathered value from products[idx].
tridiagonal solver requires more communication, it is more


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

threads
Backward Segmented Scan
0 1 2 3 4 5 6 7 Segmented Scan
10
Scan

Elapsed Time (ms)

Fast Scan

1
steps

0.1

210 211 212 213 214 215 216 217 218 219 220 221 222 223

Number of Elements

Figure 4: The communication pattern between threads in the Figure 5: Performance results for four types of scan. The
reduce step of a tridiagonal solver is similar to scan. Black most meaningful comparison between the types is between
arrows show communication used by both scan and tridi- scan and the two segmented scans; the “fast” scan processes
agonal solve; gray arrows show additional communication eight elements per thread rather than two and is provided for
needed for tridiagonal solve. The down-sweep communica- reference [HSO07].
tion requirements are also more complex for the tridiagonal
solver.

cient hardware support for flags makes their segmented scan

implementations “only marginally more expensive than the
difficult to divide across blocks. Our implementation can
unsegmented versions” [CBZ90].
process grids of up to 512 × 512 elements with each line
of the grid processed in shared memory by a single thread Large segmented scans are about three times slower than
block. For larger simulations, we divide each line of the in- large unsegmented scans for two reasons. First, segmented
put across multiple thread blocks and use global memory scan does more I/O from global memory than scan. The flag
rather than shared memory to communicate between threads. vector and the partial OR tree are read twice from global
memory, once at the beginning of the reduce phase and again
at the beginning of the down-sweep phase. Both are also
5. Results and Analysis
written when the state is saved at the end of the reduce
Scan vs. Segmented Scan For 1M elements, with 128 phase. Segmented scan also does more computation than
threads (256 elements) per thread block and using the same scan. In the reduce phase the flags are logically ORed, and in
formulation for all scans, a forward unsegmented scan takes the down-sweep phase segmented scan has an if-then-elseif
0.79 ms, a backward unsegmented scan takes 0.88 ms, a for- construct. The segmented scan implementation does signif-
ward segmented scan takes 2.61 ms, and a backward seg- icantly more address calculation to pack and unpack flags
mented scan takes 4.29 ms. The runtime cost is linear start- from the striped compact representation.
ing at roughly 4k elements, which corresponds to 16 active
Backward segmented scans are about two times slower
thread blocks (the GeForce 8800 GTX has 16 multiproces-
than forward segmented scans. This is because of uncoa-
sors). Figure 5 shows performance results for four scan vari-
lesced reads and writes of the flag vector and the partial OR
ants up to 8M elements.
tree from global memory. To implement backward scan, we
One of the most difficult aspects of our segment imple- read the data in reverse order from shared memory. How-
mentation is the representation of segment head flags. Allo- ever, this violates coalesced I/O restrictions imposed by the
cating more than a bit per flag is storage-inefficient, but the GPU since threads access memory in decreasing order. In the
lack of atomic read-modify-write shared memory instruc- future we will coalesce the I/O of flags by reading them in
tions in CUDA preclude optimal packing. It could be im- increasing order from global memory and reversing them in
plemented at additional cost with a per-warp parallel reduc- shared memory. This will result in similar performance for
tion, which we plan to try in the future. Hardware support backward and forward segmented scan, which will in turn
for a packed flag representation in general would also solve improve the performance of sparse matrix-vector multiply.
this problem. Another possible hardware solution, albeit one
that is fairly special-purpose, is associating an extra hard- Sparse Matrix-Vector Multiply Our sparse matrix-vector
ware bit with each register (essentially a 33-bit wide data multiply performance approaches, but does not yet meet,
path), which may have applications beyond segmented scan. the throughput of a best-of-breed CPU implementation.
Chatterjee et al.’s results on the Cray Y-MP show that effi- For evaluation we tested the medium-sized “raefsky2” ma-


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

trix [Dav94], a 3242×3242, 294,276-element sparse ma- in the multiprocessors (according to the profiler, the occu-
trix that represents incompressible flow in a pressure- pancy is only 1/6). In the long term, as compute costs be-
driven pipe. Using standard accounting for this operation, come cheaper compared to communication costs, complex
we achieve 215 MFLOPS on matrix-vector multiplication programs with large register usage may become more attrac-
(365 matrix-vector multiplies per second), compared to the tive.
highly-optimized, self-tuning “oski” CPU implementation’s We would like to extend our sort implementations to sup-
December 2006 results of 522 MFLOPS on a Pentium 4 and port key-value sorts for better comparison with previous
294 MFLOPS on an AMD Opteron [Gah06]. Most of our GPU sorts. The bitonic sort in the NVIDIA CUDA SDK is
runtime is spent in the backward segmented scan operation single-block only and was thus not suitable for comparison.
and we are hopeful that continued improvements in our im-
plementation will yield further performance gains. Tridiagonal Solver Our tridiagonal solver has little diffi-
Comparison to previous results is difficult because they culty maintaining real-time performance for shallow water
were implemented on older hardware, but as a comparison, simulation. For the 128×128 simulation pictured in the mid-
Bolz et al.’s 2003 implementation on a 500 MHz NVIDIA dle row of Plate 1, we measure 1207 simulation steps per
GeForce FX (15 GFLOPS, 12.8 GB/s bandwidth) achieved second for compute only. The compute time is consider-
120 matrix-vector multiplies per second (not including the ably less than the overhead of mapping and unmapping the
sort time) on a 37k-entry sparse matrix [BFGS03], a factor of vertex buffer(roughly 4:1 overhead:compute). We also com-
24 slower than our implementation on newer (345 GFLOPS, pared our 512×512 solver against a CPU cyclic reduction
86 GB/s) hardware. solver (which is not the serially optimal CPU solver). Run-
ning from GPU main memory, the CUDA implementation
Sort We compared 5 sort implementations: split-based was 3 times faster than the CPU, and using shared memory,
radix sort per block, followed by a parallel merge sort of 12 times faster (2.3 ms vs. 27.6 ms).
blocks [HSO07]; quicksort per block, followed by the paral-
lel merge; a (global) split-based radix sort across all inputs 6. Conclusion
(no merge necessary); and 2 CPU-based sorts using STL’s One of the difficulties with GPGPU programming has been
sort and C’s quicksort routines. For a sort of 4M 32-bit inte- the vertical nature of GPGPU program development. With
gers, runtimes follow. few exceptions, most applications are developed by a single
Test Runtime (ms) team from the API interface to the hardware up to the appli-
GPU global radix 165.0 ms cation itself, with little to no code reuse from other projects.
GPU radix/merge 317.8 ms The field would benefit from a more horizontal model of pro-
CPU STL sort 571.7 ms gram development with libraries of GPGPU primitives avail-
CPU quicksort 908.8 ms able for use by GPGPU applications allowing code reuse and
GPU quicksort/merge 2050.3 ms factorization by GPGPU developers.
These results parallel existing literature; Govindaraju et What should those primitives be? This is a broad and im-
al. concluded that GPU sorts were appreciably but not mas- portant question facing the GPGPU community. For a highly
sively faster than CPU-based sorts [GGKM06], and Blelloch parallel machine such as the GPU, we believe it is impor-
finds “the most practical parallel sorting algorithm” (bitonic tant not just to consider traditional scalar primitives but also
sort) and split-based radix sort had similar performance on primitives that were designed for parallel programming en-
the Connection Machine [Ble90]. Our interest in this paper vironments and machines. The scan primitives are not par-
is more oriented to demonstrating a new GPU-based sort- ticularly well suited for scalar machines, but they are an ex-
ing algorithm, namely quicksort. Quicksort has an excellent cellent match for a broad set of problems on parallel hard-
expected complexity of O(n log n). On scalar CPUs, quick- ware generally and, we believe, specifically the GPU. We
sort is in theory and in practice superior to bitonic sort’s expect that continued investigation of a wide range of po-
O(n log2 n) and in practice nearly always better than radix tential parallel primitives, and optimized implementations of
sort’s O(b) passes. On the GPU, however, quicksort is a poor these primitives, will provide insight for future hardware and
performer for several reasons. software for GPGPU and provide more productive GPGPU
programming environments for future developers.
High-performance sorts are typically bound by band-
width, not compute, but our quicksort implementation is
very definitely bound by compute. In particular, the book- Acknowledgements
keeping instructions to manage multiple active regions in Many thanks to Jim Ahrens, Guy Blelloch, Jeff Inman, and
shared memory and the staging of regions to and from the Pat McCormick for thoughtful discussions about our scan
bank-conflict-free representations used by the segmented implementation and its applications. Richard Vuduc helped
scan together result in both lengthy programs and a large with oski results, David Luebke and Ian Buck provided ex-
number of active registers. Long programs are slow, and us- cellent support of the CUDA tools, and Jeff Bolz helped
ing many registers reduces the occupancy of the program compare our sparse-matrix results to his previous work. Eric


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

Lengyel generated the images for the shallow water simula- [Hor05] H ORN D.: Stream reduction operations for
tion using his C4 game engine. GPGPU applications. In GPU Gems 2, Pharr M., (Ed.).
This work was supported by the Department of En- Addison Wesley, Mar. 2005, ch. 36, pp. 573–589.
ergy (Early Career Principal Investigator Award DE-FG02- [HSC∗ 05] H ENSLEY J., S CHEUERMANN T., C OOMBE
04ER25609, the SciDAC Institute for Ultrascale Visualiza- G., S INGH M., L ASTRA A.: Fast summed-area table
tion, and Los Alamos National Laboratory) and by the Na- generation and its applications. Computer Graphics Fo-
tional Science Foundation (grant 0541448), as well as gen- rum 24, 3 (Sept. 2005), 547–555.
erous hardware donations from NVIDIA.
[HSO07] H ARRIS M., S ENGUPTA S., OWENS J. D.: Par-
allel prefix sum (scan) with CUDA. In GPU Gems 3,
References Nguyen H., (Ed.). Addison Wesley, Aug. 2007, ch. 31.
[AS89] A NDERSON E., S AAD Y.: Solving sparse triangu-
[Ive62] I VERSON K. E.: A Programming Language. Wi-
lar systems on parallel computers. International Journal
ley, New York, 1962.
of High Speed Computing 1, 1 (May 1989), 73–95.
[KLO06] K ASS M., L EFOHN A., OWENS J.: Inter-
[BFGS03] B OLZ J., FARMER I., G RINSPUN E.,
active Depth of Field Using Simulated Diffusion on
S CHRÖDER P.: Sparse matrix solvers on the GPU:
a GPU. Tech. Rep. #06-01, Pixar Animation Stu-
Conjugate gradients and multigrid. ACM Transactions on
dios, Jan. 2006. http://graphics.pixar.com/
Graphics (Proceedings of ACM SIGGRAPH 2003) 22, 3
DepthOfField/.
(July 2003), 917–924.
[KM90] K ASS M., M ILLER G.: Rapid, stable fluid dy-
[BH03] B UCK I., H ANRAHAN P.: Data Parallel Com-
namics for computer graphics. In Computer Graphics
putation on Graphics Hardware. Tech. Rep. 2003-03,
(Proceedings of SIGGRAPH 90) (Aug. 1990), pp. 49–57.
Stanford University Computer Science Department, Dec.
2003. [KW03] K RÜGER J., W ESTERMANN R.: Linear alge-
bra operators for GPU implementation of numerical algo-
[BHZ93] B LELLOCH G. E., H EROUX M. A., Z AGHA
rithms. ACM Transactions on Graphics 22, 3 (July 2003),
M.: Segmented Operations for Sparse Matrix Computa-
908–916.
tion on Vector Multiprocessors. Tech. Rep. CMU-CS-93-
173, School of Computer Science, Carnegie Mellon Uni- [LKS∗ 06] L EFOHN A. E., K NISS J., S TRZODKA R.,
versity, Aug. 1993. S ENGUPTA S., OWENS J. D.: Glift: Generic, efficient,
random-access GPU data structures. ACM Transactions
[Ble90] B LELLOCH G.: Vector Models for Data-Parallel
on Graphics 26, 1 (Jan. 2006), 60–99.
Computing. MIT Press, 1990.
[Buc05] B UCK I.: Taking the plunge into GPU computing. [LM01] L ARSEN E. S., M C A LLISTER D.: Fast matrix
In GPU Gems 2, Pharr M., (Ed.). Addison Wesley, Mar. multiplies using graphics hardware. In Proceedings of the
2005, ch. 32, pp. 509–519. 2001 ACM/IEEE Conference on Supercomputing (Nov.
2001), p. 55.
[CBZ90] C HATTERJEE S., B LELLOCH G. E., Z AGHA
M.: Scan primitives for vector computers. In Super- [Mor02] M ORAVÁNSZKY A.: Dense matrix algebra on
computing ’90: Proceedings of the 1990 Conference on the GPU. In ShaderX2: Shader Programming Tips and
Supercomputing (1990), pp. 666–675. Tricks with DirectX 9.0, Engel W. F., (Ed.). Wordware
Publishing, 2002, pp. 352–380.
[Dav94] DAVIS T. A.: The University of Florida
sparse matrix collection. NA Digest 92, 42 (16 Oct. [NVI07] NVIDIA C ORPORATION: NVIDIA CUDA
1994). http://www.cise.ufl.edu/research/ compute unified device architecture programming guide.
sparse/matrices. http://developer.nvidia.com/cuda, Jan.
2007.
[Gah06] G AHVARI H. B.: Benchmarking Sparse Matrix-
Vector Multiply. Master’s thesis, University of California, [PSG06] P EERCY M., S EGAL M., G ERSTMANN D.: A
Berkeley, Dec. 2006. performance-oriented data parallel virtual machine for
GPUs. In ACM SIGGRAPH 2006 Conference Abstracts
[GGK06] G RESS A., G UTHE M., K LEIN R.: GPU-based and Applications (Aug. 2006).
collision detection for deformable parameterized surfaces.
Computer Graphics Forum 25, 3 (Sept. 2006), 497–506. [Sch80] S CHWARTZ J. T.: Ultracomputers. ACM Transac-
tions on Programming Languages and Systems 2, 4 (Oct.
[GGKM06] G OVINDARAJU N. K., G RAY J., K UMAR
1980), 484–521.
R., M ANOCHA D.: GPUTeraSort: High performance
graphics coprocessor sorting for large database manage- [SLO06] S ENGUPTA S., L EFOHN A. E., OWENS J. D.:
ment. In Proceedings of the 2006 ACM SIGMOD Inter- A work-efficient step-efficient prefix sum algorithm. In
national Conference on Management of Data (June 2006), Proceedings of the Workshop on Edge Computing Using
pp. 325–336. New Commodity Architectures (May 2006), pp. D–26–27.


c Association for Computing Machinery, Inc. 2007.
 Sengupta, Harris, Zhang, and Owens / Scan Primitives for GPU Computing

Plate 1: The pictures above are frames from rendering our shallow-water simulation, which uses our GPU-based tridiagonal
matrix solver. Each row displays several frames from a different simulation, with the starting point of the simulation at the left.
The top row simulates a 256×256 grid, the middle row a 128×128 grid, and the bottom row a 64×64 grid. Thanks to Eric
Lengyel for generating these images using his C4 game engine.


c Association for Computing Machinery, Inc. 2007.