Large Bandwidth-Efficient FFTs on

Multicore and Multi-Socket Systems

Doru Thom Popovici, Tze Meng Low, Franz Franchetti
Department of Electrical and Computer Engineering
Carnegie Mellon University
Email: {dpopovic, lowt, franzf}@cmu.edu

Abstract—Current microprocessor trends show a steady in-

crease in the number of cores and/or threads present on the same
CPU die. While this increase improves performance for compute-
bound applications, the benefits for memory-bound applications
are limited. The discrete Fourier transform (DFT) is an example
of such a memory-bound application, where increasing the
number of cores does not yield a corresponding increase in
performance. In this paper, we present an alternate solution
for using the increased number of cores/threads available on
a typical multicore system. We propose to repurpose some of
the cores/threads as soft Direct Memory Access (DMA) engines
so that data is moved on and off chip while computation
is performed. Overlapping memory accesses with computation
permits us to preload and reshape data so that computation is
more efficient. We show that despite using fewer cores/threads
for computation, our approach improves performance relative to Fig. 1: The plot shows the percentage from achievable
MKL and FFTW by 1.2x to 3x for large multi-dimensional DFTs peak performance at full bandwidth speed obtained with the
of up to 20483 on one and two-socket Intel and AMD systems. STREAM benchmark on the Intel Kaby Lake 7700K. MKL
and FFTW achieve at most 47% of achievable peak, while our
I. I NTRODUCTION approach achieves 80% to 90% of peak. The labels on each
The number of cores on modern shared-memory architec- of the bars represent the unnormalized performance in Gflop/s
tures has been steadily increasing since the introduction of for the three parallel implementations.
multicore/multi-socket systems around the turn of the cen-
tury. While the increased number of computational units has efficient when data is located in the upper levels of the memory
benefited compute-bound application such as matrix-matrix hierarchy. We separate data movement and computation to
multiplication, the performance gain of memory-bound appli- be able to copy data from memory to cache at bandwidth
cations such as the Fourier transform (FFT) is still limited. speed in parallel with the computation. In addition, we reshape
Large Fourier transforms do not use the cache hierarchy and (pack) data into an appropriate layout to improve performance
bandwidth to main memory efficiently due to the non-unit- of subsequent stages. The astute reader may recognize that
stride access patterns inherent to the algorithm. This makes the this approach is similar to the double-buffering approach used
task of hiding the latency of strided memory access patterns in the Out-of-Core (OoC) algorithms such as [2], [3], [4].
when accessing main memory difficult. The difference is that OoC algorithms read data from disk
The inefficient use of the memory hierarchy can be seen into main memory, while we buffer the data in the cache
in Figure 1, where we compare the results of three parallel hierarchy. Since there are no explicit functions to copy data
implementations against the achievable performance on the in the cache similar to the functions used to move data from
Intel Kaby Lake 7700K. We compute the achievable peak disk to memory, implementing the double-buffering approach
performance in Gflop/s assuming data is streamed in and out requires less obvious means that are tackled in this paper.
of the cache hierarchy at full bandwidth speed measured using Contributions. The main contributions of this paper are:
STREAM [1]. It can be seen that state-of-the-art 3D Fourier • Systematic way of applying software pipelining. We show
transforms offered by MKL 2017.0 and FFTW 3.3.6 achieve a systematic way of separating and overlapping data
at most 47% of the achievable peak when using all eight movement from memory and computation on cached data
available threads. In comparison, using various techniques to for the multi-dimensional Fourier transforms.
be discussed in this paper, our parallel implementation yields • Double-buffering on cache hierarchies. We implement a
80% to 90% of the achievable peak performance. double-buffering mechanism on multicore CPUs. Caches
In this paper, we improve the overall performance of large are set-associative and have replacement policies. Full
multidimensional FFTs that do not fit on on-chip cache. Our control of data through the cache hierarchy is needed for
approach is based on the observation that the FFT is more an efficient double-buffering.
 • Improve performance of multidimensional FFTs. We
show that our approach improves bandwidth utilization
for large Fourier transforms (up to 128 GB of input
data) and achieves 1.2x to 3x performance improvements
relative to MKL and FFTW on a range of machines from
quad-core Haswell/AMD to dual-socket Xeon/Opteron.
II. BACKGROUND
A. Multicore Architectures
Modern systems ranging from off-the-shelf desktops to
high-end servers provide a multitude of features such as large
number of threads/cores, large shared cache levels, specialized Fig. 2: Two architectures targeted in this paper. Figure A
data movement instructions. All these features are meant is typical for Intel architectures, where each core has two
to improve performance. However, combining the features hardware threads that share the L1 and L2 caches. All cores
to improve memory-bound applications such as the Discrete share L3. Figure B is typical for AMD architectures, where
Fourier transform becomes cumbersome. each core has one thread per core with its own L1 data cache.
Threads and cores. Most modern CPUs offer an increased Each two cores share an L2, while L3 is shared by all cores.
number of threads and cores to improve parallel execution.
Depending on the vendor, threads/cores share different com- The dense matrix DF Tn is the 1D linear transform of size n.
ponents such as the instruction pipelines and/or the cache Similarly, the multidimensional DFT (MDFT) can be viewed
hierarchy. This suggests possible locations where threads may as a matrix-vector multiplication, i.e. y = DF Tn0 ×n1 ×...×nk x,
contend for resources. Threads executing the same instruction where the matrix DF Tn0 ×n1 ×...×nk represents the multi-
mix will issue instructions to the same execution pipeline. This dimensional FFT.
will cause threads to stall and not make forward progress. Computing the DFT or MDFT as a matrix-vector multi-
Threads with different memory access patterns may conflict plication incurs a O(n2 ) arithmetic complexity. Fast Fourier
within the cache hierarchy and evict each others data. Evicting transform algorithms, such as the Cooley-Tukey algorithm,
data to the lower levels of the memory will increase memory reduce complexity to O(n log(n)). This reduces computation
access latency if the data is required for future computation. complexity, however the non-unit-stride access patterns in-
Cache hierarchy. Caches are data storage that are smaller herent to the algorithm persist. Over the past years, there
and faster than main memory located on the same-die as the have been numerous works on improving performance for
compute units. Figure 2 shows two typical cache hierarchies multidimensional FFTs through optimizing data movement.
for Intel and AMD CPUs. Threads share the different levels of These works can be classified into two main categories.
the cache hiearchy. Each cache level is characterized by size Reducing memory round-trips. Access to main memory is
and set-associativity. The set-associativy is used to improve costly due to the long latency. Reducing the number of round-
cache utilization. Since caches have finite storage, replacement trips to main memory reduces the overall time incurred by
policies are enforced to automatically evict data when other data movement. P3DFFT [5] reorganizes the computation of
data points are required for computation. This hides explicit the 3D FFT by fusing the first and second stages of the 3D FFT
data movement from the developers at the cost of reduced when the number of processors is small. This decomposition
control of where data resides. is known as the slab-pencil decomposition and reduces the
Non-temporal load and store operations. Loading and number of round trips to main memory from three to two.
storing data from and to main memory is done with temporal Johnson et al. [6] proposed the dimensionless FFT where
and non-temporal load/store instructions. Prefetches may be the middle stages are decomposed using the Cooley-Tukey
used, however they are only hints. Of interest to this work algorithm and are fused with the neighboring compute stages.
are the non-temporal loads and stores. While their temporal Similarly, this method reduces the number of round-trips to
counterparts store data in all the cache levels, non-temporal main memory from three to two. Takahashi et al. [7] proposed
instructions move data “directly” to registers, with the goal a 1.5D decomposition of the parallel 3D FFT implementation.
of reducing cache pollution. The downside of non-temporal These implementations are orthogonal to our approach since
operations appears when data is frequently reused. Since data they focus on merging stages, while we are focusing on the
is not stored in the upper levels of the cache hierarchy, it must row-column algorithm to achieve better streaming behaviour.
always be retrieved from memory, incurring higher latencies. Improving data access through layout transforms. Another
B. Discrete Fourier Transform class of optimizations for multidimensional FFTs focuses on
The discrete-time Fourier transform of n input samples improving memory accesses and data movement through the
x0 , x1 , . . . , xn−1 is described as a matrix-vector multiplication cache hierarchy by data layout reorganization. Recall that all
y = DF Tn x, where stages of the MDFT require non-unit-stride memory accesses.
2π Akin et al. [8] have proposed FFT implementations for FPGAs
DF Tn = [ωnkl ]0≤k,l<n , where ωn = e−j n . that perform blocked data layout transformations to improve
 Matrix formula Matlab pseudo code In addition to the Kronecker product, SPL offers constructs
y = (An Bn )x
t[0:1:n-1] = B(x[0:1:n-1]); such as diagonal and permutation matrices. Diagonal matrices,
y[0:1:n-1] = A(t[0:1:n-1]); denoted by Dnmn , are used to specify scaling operations.
for (i=0; i<m; i++) Permutations and transpositions are used to reshape data. The
y = (Im ⊗ Bn )x
y[i*n:1:i*n+n-1] = B*x[i*n:1:i*n+n-1];
basic permutation is represented by the L matrix, defined
for (i=0; i<n; i++)
y = (Am ⊗ In )x
y[i:n:i+m*n-n] = A*x[i:n:i+m*n-n]; Lmn
n : in + j → jm + i, 0 ≤ i < m, 0 ≤ j < n.
mn x for (i=0; i<n; i++)
y= Dn
y[i] = Dn[i, i]*x[i]; If the input data is viewed as a matrix of size m × n, then
for (i=0; i<m; i++) applying the construct Lmn n on the input will produce the
y = Lmn
m x for (j=0; j<n; j++)
transpose matrix of size n × m. Combining the L matrix with
y[i+m*j] = x[n*i+j];
for (i=0; i<m; i++)
the identity matrix I through the Kronecker product allows one
y = (Lmn
m ⊗ Ik )x for (j=0; j<n; j++) to specify block permutations/transpositions, where the block
y[k*(i+m*j):1:k*(i+m*j)+k-1] = size is given by the size of the identity matrix.
x[k*(n*i+j):1:k*(n*i+j)+k-1];
SPL introduces generalizations for the identity matrix In
 
TABLE I: From matrix formulas to code, in Matlab notation. In
Im×n = , m ≥ n,
Om−n×n
memory accesses for multidimensional FFTs. Inda et al. [9] Im×n =
 
Im Om×n−m , m < n.
have proposed a simple implementation of the FFT using the
BSP model where data is reshaped to improve communication Om×n is the m×n all-zero matrix. In×n is the identity matrix
between threads. Yzelman et al. [10] proposed a C library In . Moreover, there are various identities such as
that separates data movement from computation, however it Am ⊗ Bn = Lmn

mn
and Lmn mn
m Bn ⊗ Am Ln m Ln = Imn .
does overlapping tasks. Frigo et al. [11] proposed buffering
data by copying it to the upper levels of the cache hierarchy that connect the various constructs. All of the constructs can be
before the FFT computation. In all the above papers data is translated into code following Table I. More on the identities
copied before each compute stage. Tang et al. [12] proposed and implementation details can be found in [15], [16].
utilizing multiple threads on the Xeon Phi to copy data into
the L1 cache and then apply smaller 1D FFTs on it. Each
thread executes data movement and computation, however all
four threads overlap operations. Song et al. [13] apply data
and computation overlapping in the context of large scale 3D
FFTs implemented with asynchronous MPI calls. Our work is
similar however we target multicore/multi-socket systems with
large amounts of main memory, so called fat memory nodes.
C. Kronecker Product Formalism Fig. 3: The implementation of the 2D FFT algorithm. Data is
In this paper we use the matrix-vector representation of the viewed as 2D matrix of size n × m, with the x-dimension
discrete Fourier transform (DFT). We briefly present the Signal corresponding to size m laid out in the fast dimension in
Processing Language (SPL) described in [14]. The language memory. The first stage applies 1D FFTs in rows, while the
is used to capture the implementation of fast algorithms for second stage applies the pencils in columns.
DFTs as matrix factorizations.
Matrix formalism and SPL. The implementation of fast D. Fast Implementations for DFT and MDFT
algorithms for the DFT and MDFT is obtained by factorizing 1D DFT. Fast Fourier transform (FFT) algorithms for
the dense matrix into a product of structured sparse matrices. both DFT and MDFT can be expressed using the SPL
The decomposition of the DFT and MDFT is captured by the language [17]. For example, the well-known Cooley-Tukey
SPL language, which is a mathematical description based on algorithm that recursively decomposes a 1D DFT of size
matrix factorization. The language captures the data-flow of N = mn into smaller sub-problems of size m and n can
the algorithm at a higher level of abstraction. As the backbone be expressed as follows
of this language, the Kronecker product or tensor product is
DF Tmn = DF Tm ⊗ In Dnmn Im ⊗ DF Tn Lmn



m .
defined as follows:
Decomposing the algorithm using the SPL language allows
A ⊗ B = [ak,l B], for A = [ak,l ].
one to visualize the algorithm’s data-flow. The algorithm first
The decomposition of the DFT and MDFT revolves around transposes the data using the L matrix, applies the DF Tn
two constructs, namely Im ⊗ Bn and Am ⊗ In . The first on contiguous blocks of data, scales the result by the twiddle
construct applies the matrix Bn on m contiguous blocks of factors and finally applies the DF Tm on strided data points.
size n, while the latter applies the matrix Am on m data points Since the focus of this paper is on fast implementations of
located at a stride distance of n elements. the 2D and 3D DFTs, we recommend the reader the following
 while computation is performed on data previously stored
in cache. This requires a re-evaluation of the method the
multidimensional FFTs are computed and most importantly
how data access can be separated from computation.
A. 2D and 3D FFT Revisited
In this paper, we focus on optimizing data movement for
the 2D and 3D FFT. Recall that in each stage, 1D FFTs are
Fig. 4: The implementation of the 3D FFT algorithm. Data is applied in each dimension. Some dimensions require the 1D
viewed as a 3D cube of size k × n × m, with the x-dimension FFTs to be applied at large strides. To reduce those strides and
corresponding to size m laid out in the fast dimension in thus have better memory access patterns, data can be reshaped
memory. The 1D FFTs are applied in all three dimensions. after each compute stage. For the 2D FFT, this translates into a
papers [18], [19] for more details on efficient implementations transposition along the main diagonal using an L matrix after
of the 1D FFT. The main take-away of this section is that each stage of computation expressed as
FFT algorithms are strided algorithms that suffer from low
DF Tn×m = Lmn Im ⊗ DF Tn Lmn



bandwidth utilization and cache conflicts. n m In ⊗ DF Tm .
| {z }| {z }
2D and 3D DFT. Similar to the 1D FFT, the multidimen- Stage 2 Stage 1
sional FFTs are captured by the SPL notation. We start with the We change the transposition from an element-wise transpo-
2D FFT and expand to the 3D FFT. The 2D FFT is expressed sition to a blocked transposition as
as a dense matrix DF Tn×m that can be decomposed using
DF Tn×m = Lmn/µ



the Kronecker as n ⊗ Iµ Im/µ ⊗ DF Tn ⊗ Iµ Stage 2
mn/µ


DF Tn×m = DF Tn ⊗ DF Tm . Lm/µ ⊗ Iµ In ⊗ DF Tm Stage 1.

Using the general property that multidimensional DFTs are The ⊗Iµ produces memory access in cacheline size µ pack-
separable operations, the 2D DFT can be further decomposed ets [20]. This modification offers benefits when implementing


data movement with SIMD instructions. Moreover, it reduces
DF Tn×m = DF Tn ⊗ Im In ⊗ DF Tm . the false sharing when threads are applied for parallelism.
| {z }| {z }
Stage 2 Stage 1 The data reshape technique can be extended to the 3D FFT
implementation. However, the transposition in three dimen-
The overall operation described mathematically above is
sions becomes a rotation along the data cube’s main diagonal.
depicted in Figure 3. The data is viewed as a n × m matrix
Similar to the 2D FFT, the rotation is meant to assist memory
stored in row-major order. The first stage of the 2D FFT
accesses in subsequent stages. The definition of the rotation
applies 1D FFTs of size m in the rows, whereas the second k,n
matrix Km is
stage applies 1D FFTs of size n in the columns direction. The
k,n
= Lmk mn



above decomposition of the 2D FFT is called the pencil-pencil Km m ⊗ In Ik ⊗ Lm .
decomposition, where each pencil refers to a 1D FFT applied
in each dimension. Recall that the 1D FFTs require strided It can be seen that the K matrix has two parts. The first
access, therefore for large 2D FFTs each of the pencils will part transposes the front xy-plane. The second part transposes
require data to be accessed strided from memory. the side xz-plane. The rotation is depicted in Figure 5. This
The same approach can be applied to decompose the 3D implementation is an element-wise rotation. Therefore, for the
FFT, where DF Tk×n×m represents the dense matrix. same reasons discussed for the 2D case, we block the rotation


to move entire cachelines. The adopted 3D FFT decomposition
DF Tn×m = DF Tk ⊗ Imn Ik ⊗ DF Tn ⊗ Im is represented as
| {z }| {z }
Stage 3 Stage 2 n,m/µ



DF Tk×n×m = Kkµ ⊗ Iµ Inm/µ ⊗ DF Tk ⊗ Iµ Stage 3
Ikn ⊗ DF Tm . m/µ,k



| {z } Knµ ⊗ Iµ Imk/µ ⊗ DF Tn ⊗ Iµ Stage 2
Stage 1 k,n


Km/µ ⊗ Iµ Ikn ⊗ DF Tm Stage 1.
Data is viewed as a 3D cube of size k × n × m, stored in
row major order with the x-dimension corresponding to the For the remainder of this section, we discuss overlapping
size m laid out in the fastest memory dimension. The 3D FFT computation and communication for the first stage. Same steps
applies 1D FFTs in each of the three dimensions. The problem are applied to the other stages.
of accessing data at strides remains.
B. Data Movement From Memory vs. Cache
III. OVERLAPPING DATA M OVEMENT WITH Recall that we want to overlap data movement from memory
C OMPUTATION with computation on cached data. This means that it is
The goal of our approach is to overlap data movement necessary to separate main memory data movement from cache
with computation so that data can be streamed from memory data movement. Since all stages of a multidimensional FFT are
 cally/horizontally stacking all-zero matrices Ou×v of various
sizes and the identity matrix I of size b.
 
Oib×b
Sn,b,i =  Ib  where Sn,b,i ∈ Rn×b
O(n/b−i−1)b×b
The Gn,b,i matrix is the transposed version of Sn,b,i . To better
understand the meaning behind the two constructs, they can
Fig. 5: 3D rotation applied on the data cube after each compute be viewed as sliding windows that read/write blocks of size
stage. The original data of size k × n × m is rotated to a cube b elements from the input/output. If we consider the In as a
of size m × k × n. The data in the xy-plane is moved to the copy operation of n elements, then the two matrices are slices
data points in the yx-plane in the rotated data cube. through the columns and rows of the identity matrix,
 
Gn,b,0
similar, we will illustrate the separation of the two types of
   Gn,b,1 

data movements using the first stage of the 3D FFT. 
In = Sn,b,0 Sn,b,1 . . . Sn,b,n−1 =  .. .
Working on cached data. The first step towards applying  . 
our approach is tiling the computation such that the data Gn,b,n−1
required for computation fits in the cache. This means that
Combining the two constructs with the K matrix
we can define
instead of applying all 1D FFTs, we apply a smaller batch k,n
the data movement matrices Wb,i = Km/µ ⊗Iµ Sknm,b,i and
k,n

Rb,i = Gknm,b,i Iknm .
Km/µ ⊗ Iµ Ikn ⊗ DF Tm =
| {z } We separate computation from data movement to/from off-
block by b chip memory. This creates three dependent tasks as seen in
k,n



Figure 6. The Load task moves b contiguous elements from the
Km/µ ⊗ Iµ Iknm/b ⊗ Ib/m ⊗ DF Tm .
| {z } input to the cached buffer. The Compute task applies the b/m
Compute
1D FFTs of size m inplace. The Store task copies data back
The batch size is determined by the size of the shared buffer. If to main memory from the cached buffer once computation
b represents the size of the buffer, then b/m represents the total finished. The tasks are parallelized across the available threads.
number of 1D pencils that can be applied, where m represents
C. Task Parallelization and Scheduling
the size of the 1D FFT.
Each knm/b iteration applies a batch of 1D FFTs on the The SPL notation also allows us to identify which compo-
buffer of size b. We denote the Ib/m ⊗ DF Tm construct as nents need to be parallelized. Recall that computation must be
the compute kernel. We further decorate the compute kernel performed on data stored in the cached buffer. This implies
with (·) to specify that the computation must be done inplace, that the parallelism must be applied on the Ib/m ⊗ DF Tm
i.e. the input is overwritten with the computed result. The construct which will modify the first child as follows
computation kernel is parallelized across the threads assigned


for computation or compute-threads. Iknm/b ⊗ Wb,i Ib/m ⊗ DF Tm Rb,i =
| {z }
Identifying main memory access. Data needs to be read parallel on pd , pc
from memory and stored into the local buffer before com-




Iknm/b ⊗ Wb,i Ipc ⊗ Ib/(mpc ) ⊗ DF Tm Rb,i .
putation can start. Once computation has completed data |{z} | {z } |{z}
pd pc pd
needs to be stored to main memory from the local buffer.
The computation kernel determines the data movement. The Given pc threads for computation, each thread will apply its
SPL notation helps us capture the read/write operations. By Ib/pc m ⊗ DF Tm on its own disjoint data points.
reformulating the SPL formula, we obtain the following: Data movement is parallelized across pd threads. The matrix
Rb,i copies contiguous blocks of size b from main memory to
k,n
Km/µ




⊗ Iµ Iknm/b ⊗ Ib/m ⊗ DF Tm = the cached buffer. Since data is contiguous, data is streamed in.


The matrix Wb,i writes blocks equal to the size of the cacheline
Iknm/b ⊗ Wb,i Ib/m ⊗ DF Tm Rb,i . from the cached buffer back to main memory. Since data is
|{z} | } |{z}
rotated and thus written at strides, bandwidth utilization may
{z
Store Compute Load
drop. The write matrix is also parallelized across pd threads.
Let Wb,i and Rb,i denote the data movement matrices that read We finally apply software pipelining [21] to the outermost
and write blocks of size b from and to main memory every i construct represented by Iknm/b . Software pipelining allows
iteration. The variable i takes values in 0, knm/b − 1. the skewing of the Load, Compute and Store tasks and permits
We define two additional constructs Sn,b,i and Gn,b,i to the tasks to be executed in parallel. We group the Load and
help with the construction of the read and write matrices. Store tasks into one task, with the observation that the store
The two constructs are rectangular matrices obtained by verti- operation must precede the load operation. Table II shows
 Iteration Store and Load with pd threads Compute with pc threads
i=0 t[i mod 2] = Rb,i x
Prologue
i=1 t[i mod 2] = Rb,i x t[(i + 1) mod 2] = Ib/m ⊗ DFTm t[(i + 1) mod 2]


i=2 y = Wb,i−2 t[i mod 2] t[i mod 2] = Rb,i x t[(i + 1) mod 2] = Ib/m ⊗ DFTm t[(i + 1) mod 2]
... ... ...
Steady
i = knm/b−1 y = Wb,i−2 t[i mod 2] t[i mod 2] = Rb,i x t[(i + 1) mod 2] = Ib/m ⊗ DFTm t[(i + 1) mod 2] State


i = knm/b y = Wb,i−2 t[i mod 2] t[(i + 1) mod 2] = Ib/m ⊗ DFTm t[(i + 1) mod 2] Epilogue
i = knm/b+1 y = Wb,i−2 t[i mod 2]



TABLE II: Applying software pipelining to the outer loop of the construct Iknm/b ⊗ Wb,i Ib/m ⊗ DF Tm Rb,i .
ID and the socket ID, since each thread must do its own task.
We copy the generated code within the template framework in
order to get the full 2D and 3D FFT implementation.
IV. M ITIGATING I NTERFERENCE
Threads/cores share resources such as execution pipeline,
cache hierarchy, main memory and links between multiple
sockets. Threads/cores contend for these resources. Usually,
interference on the shared resources causes slower execution.
Fig. 6: Separating data movement from computation for the In this section, we focus on the main causes of interference
first stage of the 3D FFT. Data is streamed into the local and provide solutions to reduce the effects of possible conflicts
buffer. Computation applies batches of 1D FFTs inplace. Data and thus increase overall performance.
is rotated/transposed back to main memory.
A. Single Socket Execution
the prologue, steady state and epilogue of the SPL construct. Both Intel and AMD offer multiple threads that share the
The prologue loads data into the shared buffer and signals the same floating point functional units and have private and/or
threads to start computation. The epilogue stores data back to shared caches. Pinning the threads to specific cores influences
main memory once computation has been completed. In steady the overall performance of the application. We discuss the
state data movement and computation is executed in parallel. interference at the functional units and cache level and show
While the compute threads apply computation on one half of solutions to reduce contention.
the shared buffer, the data threads store and load data to and Interference in the execution pipeline. Irrespective of
from main memory into the other half of the buffer. Given p vendor, threads share the floating point functional units.
the total number of threads, this implies that p = pc + pd , Therefore, in our approach we group one data-thread and
where pc and pd represent the compute and data threads. one compute-thread. The threads are pinned together to the
same core to share the functional units. Data-threads only
D. Parallel Framework and Code Generation
load and store data. Compute-threads execute some load and
The above mathematical descriptions are translated into store operations, however they predominantly execute com-
code. We first construct a general C code template for putation such additions/multiplications. On most architectures
doing the double-buffering mechanism. We parallelize the load/store instructions and arithmetic instructions are issued to
framework using OpenMP [22], more precisely we use different pipelines. Choosing threads with the same instruction
#pragma omp parallel region, since it gives better mix is not recommended since the threads will conflict for the
control over the threads. Based on the thread ID returned same execution pipeline. We use NOP instructions interleaved
by omp_get_thread_num, we determine which threads do within the data-threads task to allow the compute-threads to
data movement and which do computation. Half the threads issue their loads and make progress. Data-threads issue only
are used for data movement and half the threads are used for load/store operations, thus the threads may fully occupy the
computation. We use kmp_affinity on the Intel architec- load/store pipeline. Even though compute-threads have fewer
tures and sched_setaffinity on the AMD architectures load/store operations, they still require some for computation.
to pin the threads to the specific cores and we explicitly use Interference at the cache hierarchy. Pinning a data-thread
them within the C code. In addition, we use #pragma omp and a compute-thread so that they share the functional units,
barrier to synchronize the threads. makes the threads share some of the cache levels. For example,
The SPL notation described in this work used to capture the on the Intel architecture, the two threads/hyperthreads on the
computation and data movement is implemented within the cores share both the L1 and L2 cache. In addition to the
SPIRAL system [15], which is a framework for automatically different mixes of instructions, the two type of threads also
generating code for linear transforms. We use SPIRAL to auto- have different memory access patterns. Data-threads stream
matically generate the computation and data movement using through while reading and rotate the data on the write back.
either AVX or SSE, depending on the underlying architecture However, the FFT accesses data at strides. The different access
we target. We parameterize the generated code by the thread patterns causes cache evictions.
 B. Dual Socket Execution
We extend our approach to two socket systems, where each
socket belongs to a Non-Uniform Memory Access (NUMA)
domain. Each NUMA domain has private main memory. The
sockets can access their memory through fast bandwidth buses
and they can access neighbor’s main memory through data-
links, such as Intel’s QuickPath Interconnect (QPI) or AMD’s
HyperTransport (HT). Figure 7 shows the topology of a typical
Fig. 7: Two socket system with two NUMA domains. Each
two socket system. It is worthwhile noticing that bandwidth to
NUMA domain consists of one CPU and local main memory.
main memory within a NUMA domain is higher compared to
The two processors are connected via QPI(Intel) or HT(AMD).
the bandwidth over the data links. The difference in bandwidth
suggests computation to be done on data stored locally and
Non-temporal loads and stores. Recall that non-temporal transfers over the interconnects to be kept to a minimum.
loads and stores bypass the cache hierarchy to reduce cache We only extend the 3D FFT to two sockets. The expression
pollution. However, not all loads and stores within the code


must be non-temporal. In our approach, only the Wb,i and Iknm/b ⊗ Wb,i Ib/m ⊗ DF Tm Rb,i =
Rb,i matrices must utilize the non-temporal operations, since
| {z }
parallel on sk, pc , pd
those are the only operations that move data to and from



Isk ⊗ Iknm/bsk ⊗ Wb,i,sk Ib/m ⊗ DF Tm Rb,i,sk
memory. The matrix Rb,i must read data non-temporally, | {z }
however it must store the data temporally in the shared buffer. parallel on pc , pd

The compute-threads must apply the FFT on the data in the specifies the order in which we parallelize the first stage of
subsequent iteration. However, the matrix Wb,i can read and the 3D FFT, the same steps are taken for the other two stages.
write non-temporally since the computed data is not required We parallelize the construct first over the number of sockets
until the next FFT stage. The write matrix non-temporally sk and then over the number of data-threads pd and compute-
stores cacheline blocks at large strides in main memory. threads pc . The parallelism over the sockets modifies the read
Cache aware FFT. Computation is cacheline aware, since and write matrices, since all data points are required for the
the FFT is computed at cacheline granularity. Recall that a 1D computation of the 3D FFT. Thus, data needs to be exchanged
FFT accesses data at non-unit strides. Accessing data at large across the data links.
strides may cause data to be placed in the same set and thus Since communication over the QPI/HT links is costly, we
evict other cachelines before the remaining data within the apply a slab-pencil decomposition to the 3D FFT, where the
cacheline is fully consumed. We use SIMD instruction such first two stages communicate only within the NUMA domain.
as SSE and AVX to implement the computation. We follow We split the data-set in the z-dimension, where
each socket
the details from [18], where the FFT performs a data-format receives a contiguous block of size k/sockets ×n×m. Each
change of the complex data storage between compute stages. node can compute a 2D FFT locally and transpose locally,
The format change swaps from complex interleaved, where the without crossing the interconnect. In order to compute the 1D
real and imaginary components are interleaved in memory, to a FFT in the z-dimension data needs to be exchanged across
block interleaved format, where blocks of the real components the data-links. Another communication over the interconnect
and blocks of the imaginary components are consecutive in is needed once the 1D pencil is fully computed so that data is
memory. This format change is meant to make computation put in the correct order. Figure 8 depicts the data movement in
more efficient. Separating the real and imaginary components the three compute stages of the 3D FFT. Table III presents the
and using AVX instructions allows computation to be done at generalized versions of write Wb,i matrices. All three matrices
cacheline granularity. Since the 2D and 3D FFTs have multiple are parameterized by the number of sockets k. By setting
stages, the format change is applied once in the first stage, the the number of sockets equal to sk = 1, the implementation
rest of the computation is done in block interleaved and in the defaults to the single-socket implementation. Reading data is
last stage data is changed to complex interleaved. This change done by each socket from the local memory and has the same
of format is different from the one presented in [9] which is representation as the single-socket implementation.
meant to improve communication between threads.
V. E XPERIMENTAL R ESULTS
Cache aware buffer allocation. The cached buffer t shared
by the data-threads and the compute-threads must reside within Experimental setup. We now evaluate the performance
the cache hierarchy. More precisely, the buffer must be located of our approach. We run experiments on Intel and AMD
in the last level cache (LLC) since it is shared between all the systems with one or two sockets. All systems provide multiple
threads. We set the size of the buffer to be equal with half cores/threads that share a large last level cache. All Intel
of the LLC, b = sizeLLC /2. The buffer cannot fully occupy architectures have hyperthreads enabled. AMD does not offer
the LLC since computation also requires extra temporaries for hyperthread support. For the dual-socket systems we configure
storing partial results and constants such as the twiddle factors. the QPI/HT protocol to use Home Snoop. The Home Snoop
Fig. 8: The three different data cube shapes after each 3D FFT stage. Data is distributed across the k-dimension to each socket.
The first stage reads and writes the data locally, while the other two stages read data locally but write data across the sockets.
Matrix SPL Representation bound we consider infinite compute resources and do not
1
Wi,b,sk =
n,k/sk

Isk ⊗ Km/µ ⊗ Iµ Sknm,b,i take computation time into account. We use the STREAM
2
Wi,b,sk =
sknm/µ

Lnm/µ ⊗ Ikµ/sk Isk ⊗ Kn
k/sk,m/µ

⊗ Iµ Sknm,b,i
benchmark [1] to determine the achievable bandwidth (GB/s)
3 m/µ,n/sk for each targeted architecture. We determine the performance
Lskk



Wi,b,sk = k ⊗ Imn/sk Isk ⊗ Kk ⊗ Iµ Sknm,b,i
when streaming the total amount of data as
TABLE III: The SPL representations for the three write 5 · N · log(N ) · bandwidthSTREAM
matrices (rotation matrices) applied after each compute stage. Pio = ,
2 · N · nrstages · sizeof(double)
protocol improves bandwidth over the interconnect since it where nrstages represents the number of compute stages in the
reduces cache traffic used for cache coherence. FFT, N represents the size of the FFT and sizeof(double)
1) One socket systems: Intel Haswell 4770K, Intel Kaby represents the size of a double precision floating point in
Lake 7700K and AMD FX-8350 - 8 threads, 8 MB L3 GB. The current implementation offers support for complex
cache, 32/64/64 GB DRAM, bandwidth 20/40/12 GB/s numbers therefore the total size is multiplied by two.
2) Two socket systems: Intel Haswell 2667v3, AMD 6276 2D FFT. We first present results for the 2D FFT implemen-
Interlagos (Bluewaters): 16 threads, 20/16 MB L3 cache, tation. Figure 9 shows the results of the three implementations
256/64 GB DRAM, bandwidth 85/20 GB/s compared to the achievable peak when streaming data at 40
We compare our implementation to MKL 2017.0 and GB/s. Our double-buffering approach achieves on average
FFTW 3.3.6 on the Intel architectures. On the AMD systems 74% of the achievable peak. However, MKL and FFTW
we compare our implementation only against FFTW 3.3.6. implementation achieve on average 50% of peak. Two aspects
All libraries are compiled with OpenMP, AVX and SSE are worth noticing. First, for small sizes bandwidth utilization
enabled. On the Intel architectures, we use MKL_DYNAMIC, is less than 80% since the number of iterations iter = mn/b
MKL_NUM_THREADS and KMP_AFFINITY to control the in each compute stage is small. For example, iter = 4 when
number of threads and the placement of the threads within b = 131, 072, m = 512 and n = 1, 024. Second, as the size of
the MKL library. On the AMD architectures, we use the 2D FFT increases bandwidth utilization drops. Recall that
OMP_NUM_THREADS and GOMP_AFFINITY to control the after each compute stage data needs to be transposed similarly
threads within the FFTW library. We compile all code with to the rotation described in Figure 6. The transposition is
the ICC compiler version 2017.0 and the GCC compiler 4.8.5- applied on a panel of size b/m×m, where m is the size of the
20150623. All code is compiled with the -O3 flag. We do 1D FFT and b is the size of the shared buffer. As m increases,
not compare the current implementation against the SPIRAL b/m decreases, therefore TLB misses cannot be amortized.
generated parallel code presented in [20]. The previous parallel We leave as future work other methods of separating data
implementation targeted medium size 1D FFTs and did not movement from computation for cases where the size of the
offer support for compute/communication overlap. 1D FFT is equal or greater than the size of the shared buffer.
Performance metric. We report performance for our ap- 3D FFT. We show results for the 3D FFT on multiple
proach, MKL and FFTW as billion floating point operations architectures and compare them against the achievable peak.
divided by runtime in seconds. We use 5N log(N ) as estimate We evaluate large 3D FFTs that do not fit on the cache.
for the flop count which over-estimates the number of opera- Figure 1 shows the results on the Intel Kaby Lake 7700K.
tions. The resulting Pseudo-Gflop/s is proportional to inverse Our implementation achieves 80% to 90% of practical peak,
runtime and is an accepted performance metric for the FFT. whereas MKL and FFTW achieve at most 47%. Our approach
We use the rdtsc time-stamp counter and the CPU frequency uses the bandwidth and the cache hierarchy more efficiently
to compute the overall execution time. and therefore outperforms MKL and FFTW by almost 3x. It
We also compare performance of all three parallel im- is important to state that the 3D FFT does not experience
plementations against the performance when streaming data the problems displayed by the 2D FFT. First, the number
in and out of cache at bandwidth speed. For the upper of iterations in each compute stage is higher. For example,
 We further present scaling results keeping the problem
size fixed and increasing the number of sockets from one
to two. The bottom two plots in Figure 11 show the results
on the two-socket Intel and AMD architecture. For the Intel
architecture, given the data movement in Figure 8 our approach
improves performance on average by 1.7x when increasing
the number of sockets. Communication over the QPI link and
conflicts between the two types of threads limit the overall
improvement. On the AMD system the HT link runs at a
similar bandwidth speed as the bus to main memory, therefore
the slowdown caused by the interconnect is smaller. We do
not report comparison results against FFTW for the AMD
Fig. 9: The plot shows performance of the 2D FFT on the two-socket system since the FFTW library misbehaves on the
Intel Kaby Lake 7700K. We compare against the achievable Bluewaters system and provides buggy results.
performance when data is streamed at bandwidth speed. Our
approach achieves on average 75% ofl peak. The labels on the
bars show unnormalized performance.

iter = 1, 024, when b = 131, 072, m = 512, n = 512,

k = 512. Given the problem sizes tackled in this paper, the
size of the 1D FFTs will not influence the data movement.
Opening new memory pages can be easily be amortized. The
3D FFT will most likely run out of main memory.
Similar results are obtained on the Intel Haswell 4770K (top
left plot in Figure 11), where our implementation achieves on
average 30 Gflop/s and almost 2x faster execution compared to
MKL and FFTW. Our approach improves bandwidth utiliza-
tion and the implementation runs at 92% of peak. Due to space
constraints we do not present these bandwidth results. The top
right plot in Figure 11 shows the results on the AMD FX-8350,
where our double-buffering approach improves bandwidth Fig. 10: The plot shows performance for the 3D FFT on the
utilization and performance. It is important to notice that the two-socket Intel Haswell 2667. We report Gflop/s and show
speedup over FFTW on AMD is only 1.6. The reason is that that our implementation outperforms MKL and FFTW on two
FFTW uses the slab-pencil decomposition for the 3D FFT and socket systems given the data movement in Figure 8.
this decomposition is suitable for AMD’s larger caches. VI. C ONCLUSION
Extending to two sockets. We finally present results on In this paper, we make the observation that large FFTs
platforms with two sockets. For all implementation, we use that do not fit in cache under-utilize the cache hierarchy
the NUMA library to allocate and partition data on each and the bandwidth to main memory. By repurposing some
NUMA node. Figure 10 shows the performance results on of the compute-threads as soft DMA engines to load and
the two-socket Intel Haswell 2667. It can be seen that our im- reshape data in parallel to computation our approach streams
plementation outperforms those offered by MKL and FFTW. data from memory at near peak bandwidth speed. We show
However, the 3D FFT implementation only achives 1.2x to that despite using less threads for computation our approach
1.6x performance improvements. The performance drop is improves overall bandwidth utilization to within 10% of peak
caused by the communication over the QPI link. Recall that bandwidth on single socket systems and within 30% on dual-
the double-buffering 3D FFT requires two write operations socket systems. Our implementation improves performance
over the QPI link as described in Figure 8. Writing data relative to MKL and FFTW by 1.2x to 3x on single and dual-
over the interconnect is expensive compared to reading data, socket systems for problem sizes that have a memory footprint
therefore our implementation is penalized. Assuming that of up to 128 GB and do not fit on modern GPUs.
data is streamed at bandwidth speed and that the QPI link
does not influence the transfer rate, than our implementation ACKNOWLEDGMENT
is within 20% to 30% of achievable peak. However, given This work was sponsored partly by the DARPA PERFECT
the data movement presented in Figure 8, where two data and BRASS programs under agreements HR0011-13-2-0007
movements over the QPI are required, our double buffering and FA8750-16-2-003, and NSF through award ACI 1550486.
3D FFT implementation performs within 7% to 15% from the The content, views and conclusions presented in this document
performance given the cummulative bandwidth speed of the are those of the author(s) and do not necessarily reflect the
main memory and QPI links. position or the policy of the sponsoring agencies.
Fig. 11: The top plots present the performance in Gflop/s for the 3D FFT implementation on the Haswell and AMD architecture.
The bottom plots show speedup for large 3D FFTs when increasing the number of sockets for fixed problem sizes.

R EFERENCES [12] J. Park, G. Bikshandi, K. Vaidyanathan, P. T. P. Tang, P. Dubey, and

D. Kim, “Tera-scale 1D FFT with low-communication algorithm and
[1] J. D. McCalpin, “Memory bandwidth and machine balance in current the Intel Xeon Phi coprocessors,” in Proceedings of the International
high performance computers,” IEEE Computer Society Technical Com- Conference on High Performance Computing, Networking, Storage and
mittee on Computer Architecture (TCCA) Newsletter, pp. 19–25, 1995. Analysis, SC ’13, pp. 34:1–34:12, ACM, 2013.
[2] D. H. Bailey, “FFTs in external of hierarchical memory,” in Proceedings [13] S. Song and J. K. Hollingsworth, “Designing and auto-tuning parallel 3-
of the 1989 ACM/IEEE Conference on Supercomputing, Supercomputing D FFT for computation-communication overlap,” in Proceedings of the
’89, pp. 234–242, ACM, 1989. 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel
[3] W. M. Gentleman and G. Sande, “Fast Fourier Transforms: For fun and Programming, PPoPP ’14, pp. 181–192, ACM, 2014.
Profit,” in Proceedings of the November 7-10, 1966, Fall Joint Computer [14] J. Xiong, J. Johnson, R. W. Johnson, and D. Padua, “SPL: A language
Conference, AFIPS ’66 (Fall), pp. 563–578, ACM, 1966. and compiler for DSP algorithms,” in Programming Languages Design
[4] T. H. Cormen, J. Wegmann, and D. M. Nicol, “Multiprocessor out- and Implementation (PLDI), pp. 298–308, 2001.
of-core FFTs with distributed memory and parallel disks (extended [15] M. Püschel, J. M. F. Moura, J. Johnson, D. Padua, M. Veloso, B. Singer,
abstract),” IOPADS ’97, (New York, NY, USA), pp. 68–78, ACM, 1997. J. Xiong, F. Franchetti, A. Gacic, Y. Voronenko, K. Chen, R. W.
[5] D. Pekurovsky, “P3DFFT: A Framework for Parallel Computations of Johnson, and N. Rizzolo, “SPIRAL: Code generation for DSP trans-
Fourier Transforms in Three Dimensions,” SIAM Journal on Scientific forms,” Proceedings of the IEEE, special issue on “Program Generation,
Computing, vol. 34, no. 4, pp. C192–C209, 2012. Optimization, and Adaptation”, vol. 93, no. 2, pp. 232– 275, 2005.
[6] J. Johnson and X. Xu, “A recursive implementation of the dimensionless [16] F. Franchetti, Y. Voronenko, and M. Püschel, “Formal loop merging for
FFT,” in International Conference on Acoustics, Speech, and Signal signal transforms,” in Programming Languages Design and Implemen-
Processing (ICASSP), 2003. tation (PLDI), pp. 315–326, 2005.
[7] D. Takahashi, A. Yee, T. Hoefler, C. Coti, J. Kim, and F. Cappello, [17] C. Van Loan, Computational Frameworks for the Fast Fourier Trans-
“An implementation of parallel 3-D FFT with 1.5-d decomposition,” in form. Philadelphia, PA, USA: SIAM, 1992.
The seventh workshop of the INRIA-Illinois-ANL Joint Laboratory on [18] D. T. Popovici, F. Franchetti, and T. M. Low, “Mixed data layout kernels
Petascale Computing, 2012. for vectorized complex arithmetic,” in 2017 IEEE High Performance
[8] B. Akin, F. Franchetti, and J. C. Hoe, “FFTs with near-optimal memory Extreme Computing Conference (HPEC), 2017.
access through block data layouts: Algorithm, architecture and design [19] M. Frigo, “A fast Fourier transform compiler,” in Proc. 1999 ACM
automation,” Journal of Signal Processing Systems, 2015. SIGPLAN Conf. on Programming Language Design and Implementation,
[9] M. A. Inda and R. H. Bisseling, “A simple and efficient parallel FFT vol. 34, pp. 169–180, ACM, May 1999.
algorithm using the BSP model,” Parallel Computing, vol. 27, no. 14, [20] F. Franchetti, Y. Voronenko, and M. Püschel, “FFT program generation
pp. 1847–1878, 2001. for shared memory: SMP and multicore,” in Supercomputing (SC), 2006.
[10] A. Yzelman, R. H. Bisseling, D. Roose, and K. Meerbergen, “Multi- [21] M. Lam, “Software pipelining: An effective scheduling technique for
corebsp for c: a high-performance library for shared-memory parallel VLIW machines,” pp. 318–328, 1988.
programming,” International Journal of Parallel Programming, vol. 42, [22] OpenMP Architecture Review Board, “OpenMP application program
no. 4, pp. 619–642, 2014. interface version 4.5,” May 2015.
[11] S. G. Johnson and M. Frigo, “Implementing FFTs in practice,” in Fast
Fourier Transforms (C. S. Burrus, ed.), ch. 11, Rice University, Houston
TX: Connexions, September 2008.