Ariane + NVDLA: Seamless Third-Party IP Integration with ESP

Davide Giri, Kuan-Lin Chiu, Guy Eichler, Paolo Mantovani, Nandhini Chandramoorthy‡
and Luca P. Carloni
{davide_giri,chiu,guyeichler,paolo,luca}@cs.columbia.edu,[email protected]
Department of Computer Science, Columbia University in the City of New York, New York, NY 10027
‡ IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598

ABSTRACT ESP accelerator third-party IP

hardware
monolithic
The growth of the RISC-V movement and the demand for specialized hardware interface
proxies proxies bus adapters
have fueled a proliferation of open-source processors and accelerators. Lever-
NoC interface NoC interface
aging the large amount of available IP components requires the capabilities

software
user app w/ ESP API user app (as is!)
for integrating them effectively in a system-on-chip. We address this goal by
ESP library third-party library (as is!)
augmenting ESP, an open-source platform for agile design of heterogeneous ESP drivers third-party driver (as is!)
SoC architectures. Since its release, ESP has provided a set of different flows
Fig. 1: Native and third-party ESP accelerator integration.
to design new accelerators with different specification languages and auto-
mate their SoC integration. We add support for the seamless integration of Each of these specifications can be synthesized into an implemen-
third-party accelerators by developing a new type of interface that retains the tation and evaluated in an FPGA-based prototype of the SoC.
benefits of the ESP platform services. We demonstrate these capabilities by With this work, we add support for the integration of third-
showing how to integrate the Ariane RISC-V core with multiple instances of party accelerators into ESP by developing a new type of socket
the NVIDIA Deep Learning Accelerator into an SoC architecture that leverages interface that retains the benefits of the ESP platform services [11].
a scalable memory hierarchy and network-on-chip. The new contributions, The new third-party flow (TPF) allows any user of ESP to integrate an
which are already available in the ESP release, allow designers to rapidly existing accelerator by simply choosing among a set of bus-standard
prototype complex SoC architectures on FPGAs with a push-button design flow. adapters that interface it with the NoC connecting the tiles in the
architecture. The new TPF socket enables the seamless integration
of a loosely coupled accelerator such that designers can execute
1 INTRODUCTION the original software application as is. The only requirement is that
Hardware accelerators are pervasive across a multitude of SoC the application and the device drivers can be compiled for RISC-V.
architectures that integrate many heterogeneous components [25]. Fig. 1 captures the differences between integrating an ESP accel-
As the complexity of the design effort keeps growing with each SoC erator and a third-party IP with the new TPF. The two top quadrants
generation, the addition of new capabilities is increasingly limited show the main features of the two accelerator sockets. Notably, the
by the engineering effort and team sizes [30]. network interface is the same, granting both accelerators access to
Thanks to the open-source hardware (OSH) movement, which the ESP platform services. However, some of the building blocks of
largely stemmed from the RISC-V project [4, 23], a growing commu- the ESP socket, which are proxies for these services, are replaced
nity of researchers and engineers is contributing to the proliferation by simpler adapters between the NoC and a standard bus interface.
of OSH processor cores and accelerators [24]. Made available in Generic accelerator datapaths, in fact, are not typically decoupled
the public realm, these intellectual property (IP) blocks allow de- from their system interface, which is often part of a monolithic IP
signers to exploit the aggregate expertise of the entire community that behaves according to a bus protocol. The two bottom quad-
when realizing a new SoC. However, leveraging a large amount rants, instead, show the corresponding software stacks. Thanks to
of IP components and combining them into a solution for a target the new TPF, an ESP instance can drive a third-party accelerator
domain remains a challenging task that requires the capabilities for by simply running its original software and device driver.
integrating them effectively into an SoC architecture. We demonstrate these new capabilities by integrating the Ariane
To tackle this challenge we augment ESP, an open-source plat- 64-bit RISC-V core [1, 45] and multiple instances of the NVIDIA
form for the agile design of heterogeneous SoCs [14]. ESP combines Deep Learning Accelerator (NVDLA) [37] into an SoC instance,
an architecture and a methodology. The scalable tile-based archi- which we implement on an FPGA board by means of the ESP push-
tecture simplifies the integration of heterogeneous components. button flow for rapid prototyping. This SoC instance can use up to
The flexible methodology embraces the use of a variety of design four DDR channels to external memory, reconfigure at run-time
flows for component development. In particular, these flows sim- the cache hierarchy, offload multiple concurrent workloads to the
plify the design of new loosely coupled accelerators [15] and their NVDLA instances, and monitor the effect on the NoC traffic and
integration into the architecture. Users can choose to specify a new the memory hierarchy in real time.
accelerator at different abstraction levels, including cycle-accurate The contributions described in this paper are already available
RTL descriptions with SystemVerilog or Chisel, loosely-timed or un- as part of the ESP release [14]. Designers can use the new TPF to
timed behavioral descriptions with SystemC or C/C++, and domain- rapidly prototype RISC-V-based SoCs that combine a mix of their
specific languages such as Keras TensorFlow, PyTorch and ONNX own accelerators with other OSH components in order to run more
for the application domain of embedded machine learning [19]. efficiently their software for the target application domain.
 Giri, et al.

DRAM
ESP multi-port
memory memory
accelerator
ESP accelerator multi-bank
tile HLS (C, SystemC, Tensorflow, private local
controller tile Pytorch), Chisel, Verilog, ... memory
bus
read/write config done
LLC &

flush
directory
private DMA cfg IRQ
cache TLB ctrl regs
1 2 3 4 5 6
NoC coherence DMA IO/IRQ
1 2 3 4 5 6
planes planes plane
coherence DMA IO/IRQ NoC
planes planes plane
processor
(RISC-V Ariane, ...) processor third-party
interrupt level

L1 cache tile accelerator third-party accelerator

tile

multi-plane NoC
(NVDLA, ...)
AXI4 bus read/write port config port done
APB bus AXI4 bus APB bus
L2 cache IRQ
flush I/O memory
tile tile
1 2 3 6 4 5 6
NoC coherence IO/IRQ DMA IO/IRQ NoC
planes plane planes plane

Fig. 2: Example of ESP architecture with 16 heterogeneous tiles interconnected by a multi-plane NoC. The NoC is the core of
the ESP architecture, providing inter-tile communication and supporting the system-level platform services.
2 THE ESP ARCHITECTURE instance of the modular socket for the processor tile augments them
The ESP architecture is structured as a tile grid [11]. For a given with a private unified L2 cache of configurable size. The processor
application domain, the architect decides the structure of the SoC integration into the distributed ESP system is transparent, i.e. no
by determining the number and mix of tiles with the help of the ESP-specific software patches are needed to boot Linux. A MESI
ESP graphical user interface. For example, Fig. 2 shows an SoC directory-based protocol provides support for system-level coher-
instance with 16 tiles organized in a 4 × 4 grid. There are four ence on top of three dedicated planes in the NoC [21]. Another
main types of tiles: processor tile, accelerator tile, memory tile plane supports I/O and interrupt-request channels that are used for
for the communication with main memory, and auxiliary tile for various purposes, including accelerator management.
peripherals (e.g. UART or Ethernet) or system utilities (e.g. the Memory Tile. Each memory tile contains a DDR channel to
interrupt controller). In the first version of ESP, each processor tile external DRAM. The number of memory tiles can be configured
contained a 32-bit LEON3 SPARC core [13] and each accelerator at design time and typically varies from one to four depending on
tile contained a loosely-coupled accelerator designed with the ESP the size and type of the given SoC. All necessary hardware logic
methodology. Thanks to the contributions described in this paper, to support the partitioning of the addressable memory space is
now ESP allows designers to choose between this LEON3 core and automatically generated. This logic is also completely transparent
the 64-bit Ariane RISC-V core [45] for a processor tile as well as to to software. Hence, each memory tile contains a partition of con-
instance a third-party IP block within any accelerator tile. figurable size of the last-level cache (LLC) and the corresponding
Each tile is encapsulated into a modular socket (aka shell) that directory. The directory implements a NoC-based MESI protocol
interfaces it to a network-on-chip (NoC). In addition, the socket extended with support for coherent-DMA transfers for the accel-
implements a set of platform services that provide pre-validated so- erators [21]. The MESI protocol messages are routed through the
lutions for many important SoC operations, including: accelerators three coherence planes. The DMA packets are routed on two addi-
configuration, memory management, sharing of SoC resources, and tional dedicated planes. The memory tile routes non-coherent DMA
dynamic voltage and frequency scaling (DVFS). For example, the requests directly to main memory, bypassing the cache hierarchy,
coherence model of each accelerator can be reconfigured dynami- while the coherent DMA requests are sent to the directory.
cally based on the particular workload [20]. Dynamic voltage and Accelerator Tile. Each accelerator tile contains the specialized
frequency scaling can be applied at the granularity of each tile [34]. hardware for a loosely-coupled accelerator that executes a coarse-
This fine granularity applies also for such services as performance grained task [15]. Once an application running on a processor
counters and operation monitors. The platform services is one of core has acquired and configured a given accelerator, its execution
the keys to rapid prototyping of full SoCs. At design time, it is pos- happens independently from the core while exchanging (typically
sible to choose the combination of services for each tile. At runtime, very large) data sets with the memory hierarchy [33].
many of these services offer reconfigurability options. At design time, the modular socket decouples the design of the
The NoC interacts only with the socket to provide on-chip com- accelerator from the design of the NoC and the rest of the SoC,
munication and support the platform services. In the current ver- and provides pre-designed implementations for a set of accelerator-
sion of ESP, the NoC has a packet-switched 2D-mesh topology with independent platform services. In particular, the services relieve
multiple physical planes. These allow the NoC not only to prevent the accelerator designer from the burden of “reinventing the wheel”
various types of protocol deadlock, but also to distribute messages with respect to implementing various mechanisms such as: accel-
to maximize the performance of processors and accelerators. erator configuration through memory-mapped registers, virtual
Processor Tile. Both the LEON3 and the Ariane RISC-V pro- memory and DMA services for data transfer with the accelerator’s
cessor can run Linux and come with their private L1 caches. The private local memory, and interrupt requests for interactions with
Ariane + NVDLA: Seamless Third-Party IP Integration with ESP

/*
the processor cores. The socket of the accelerator tile supports * Example of existing C application
* with ESP accelerators that replace
* software kernels 2, 3 and 5
also multiple cache-coherence models and it enables accelerator-to- {
*/

int *buffer = esp_alloc(size);

accelerator communication, such that DMA transactions are either for (...) {

routed to memory (shared memory communication), or directly to Application kernel_1(buffer,...); /* existing software */

mode
user
esp_run(cfg_k2); /* run accelerator(s) */
other accelerators (accelerator-to-accelerator communication). ESP Library esp_run(cfg_k3);
kernel_4(buffer,...); /* existing software */
The concept of socket plays a key role in supporting the flexibility ESP accelerator driver }
esp_run(cfg_k5);

kernel
mode
of the ESP methodology in the sense that it accommodates accel- ESP core ESP alloc validate(buffer); /* existing checks */
esp_cleanup(); /* memory free */
erators designed with many different design flows, as mentioned Linux
}

in the introduction. Furthermore, sockets have been instrumental

to adding support for the integration of third-party accelerators Fig. 3: ESP software stack for accelerator programming.
like NVDLA. Fig. 2 contrasts the socket for a newly-designed accel-
• Data bit-width: the default size of the data word for the accel-
erator (top right corner) with the one for third-party accelerators
erator. This can be one byte (8), half word (16), word (32), or double
(bottom right corner). Section 3 discusses in detail the difference of
word (64). This information is used to generate examples of data
integrating these two types of accelerators in ESP.
transaction requests in the ESP accelerator skeleton.
Accelerator Programming. ESP comes with a software stack
• Input data size: the size of the input data set. This entry can
and an application programming interface (API). The ESP accel-
be specified as function of the configuration registers.
erator API library simplifies the invocation of accelerators from a
• Output data size: the size of the output data set. This can be
user application, by exposing only three functions to the program-
specified as a function of the configuration registers as well.
mer [19]. Underneath, the API invokes the accelerators through
• Chunk factor: integer divisor of the input and output data
the Linux device drivers, which are automatically generated for
sets. This parameter determines the size of the input and output
ESP accelerators. The lightweight API can be easily targeted from
data tokens, i.e. the portion of input data to be loaded before starting
existing applications or by a compiler.
computation and the corresponding portion of output results stored
Fig. 3 illustrates the ESP software stack through the (simplified)
back to memory by the accelerator. At each invocation, an ESP
case of an application with five kernels, two executed in software
accelerator typically loads several input data tokens from DRAM,
and three implemented with an accelerator. For a given application,
it processes them, and stores several output data tokens back into
the software execution of a computationally intensive kernel can
DRAM. The three phases of load, compute and store are pipelined,
be replaced with a hardware accelerator by means of a single func-
thanks to the accelerator’s private-local memory (PLM) [40]. The
tion call (esp_run()). Thanks to the esp_alloc() and esp_cleanup()
capacity and the micro-architecture of the PLM depend on the
functions, data can be shared between accelerators and processors
values entered during this initial configuration phase.
in such a way that no data copies are necessary. Data are allocated
• Batch factor: integer multiplier of the input and output data
in an efficient way to improve the accelerator’s access to memory
set. This parameter can be a function of the configuration registers
without compromising the software’s performance [33].
and determines the number of complete input data sets that get
processed in a single invocation of the accelerator. While the chunk
3 ACCELERATOR INTEGRATION factor sets the size of the PLM (in combination with input and
output data size), the batch factor has no impact on local storage.
In order to explain how third-party accelerators can be integrated
• In place: flag specifying whether the store phase should com-
in ESP, we first describe the native ESP accelerator flow. Then, we
pute an offset for write transactions, or assume that output results
show how the ESP accelerator socket can collapse into a third-
can overwrite input data in main memory.
party accelerator socket that offers access to the system’s shared
resources through standard bus interfaces. The ESP accelerator initialization script generates an HLS-ready
skeleton of the accelerator specification together with the unit test-
bench, HLS and simulation scripts, bare-metal driver, Linux driver,
3.1 ESP Accelerator Flow and a sample testing application. When the user selects either the
The ESP accelerator flow consists of a sequence of automated steps. Stratus HLS or Vivado HLS flow, this skeleton is generated as HLS-
The process begins by launching an interactive script that asks synthesizable SystemC or C++ code, respectively. The designer
the user to enter the following application-specific information must complete the specification by adding the accelerator-specific
required to generate a skeleton of the accelerator specification: code for the computation phase to the skeleton and, possibly, by
• Accelerator name and ID: a unique pair of name and device modifying the generated code to account for any irregular mem-
ID for the new accelerator. ory transactions. The resulting specification is the HLS-ready code,
• Design flow: the desired high-level synthesis (HLS) tool. The which is the entry point of the design automation process for both
current choice is among Cadence Stratus HLS [41], Xilinx Vivado the Stratus HLS and Vivado HLS flows. For the HLS4ML flow, in-
HLS [44] and the open-source project HLS4ML [17, 42]. stead, the entry point is a trained machine-learning model created
• Accelerator registers: the list of user-defined configuration with Keras [29], ONNX [6] or Pytorch [38], while the HLS-ready
registers and value ranges. Each entry translates into the generation code is automatically generated as a complete synthesizable C++
of a 32-bit configuration register in the accelerator socket; the specification of the accelerator.
register is exposed to software through the addition of a field into Besides the accelerator specification, ESP users are responsible
the descriptor data structure. for adding application-specific code to the generated unit testbench.
 Giri, et al.

This code should read (or generate) input data and compute the shared system resources. Any accelerator, regardless of its particu-
corresponding golden output data for verification. The code added lar microarchitecture, needs access to the same resources: a minimal
to the C++ or SystemC testbench must be copied into the bare metal list includes memory, configuration-status registers (CSR) exposed
and Linux applications as well. These applications replace the role to software, and interrupt delivery. Without using an automated
of the testbench for system-level simulation and FPGA prototyping. flow, like the one offered by ESP, designers are typically responsible
for implementing the logic to interface with the SoC and access
such resources. Even when leveraging corporate CAD tools that
3.2 ESP SoC Flow facilitate the integration of accelerators into their proprietary sys-
After completing the HLS-ready code, all types of ESP accelerator tems, this logic is generated as part of the new IP block. For instance,
flow follow the same automated steps, regardless of the particular this applies to the well-known Xilinx Vivado tool that combines
language chosen for the specification. The main steps are: HLS with an IP integrator flow specific for ZYNQ SoCs [44].
1. High-level synthesis. Run the chosen HLS tool to gener- In general, the obvious choice for designers, is to pick one or
ate one or multiple RTL implementations of the accelerator. The more standards to interface with a hypothetical SoC. Among the
generated Verilog code is added to the ESP library of IP blocks for available options, the open standards AXI, AHB and APB from ARM
integration. During this step, ESP users can perform a design-space are the most widely adopted [2]. Other popular alternatives include
exploration for the new accelerator by tuning the HLS directives TileLink from UC Berkeley [9], Avalon from Intel [27], CoreConnect
and obtain several functionally-equivalent RTL implementations, from IBM [26] and Wishbone from the OpenCores community [43].
each with a different cost vs. performance trade-off point [32, 39]. For the new TPF, we implemented adapters for ARM-based IP
Thanks to the ESP socket, any of these RTL implementations can be blocks. Specifically, AXI, AXI-Lite and APB adapters are now part
integrated in the SoC, regardless of its throughput and latency [35]. of the open-source release of ESP. We anticipate implementing
This approach to digital design that combines latency-insensitive adapters for other standards in the near future, thus expanding the
interfaces [10, 12] with HLS is raising interests in the silicon in- set of third-party IP blocks that ESP can seamlessly integrate.
dustry, where companies seek paths to more agile design flows for The idea behind the TPF for ESP is simple: we modify the ESP
SoCs [30, 46]. socket by relying on its modularity so that it “collapses” into a set of
2. Accelerator verification. The unit testbench stimulates the bus-standard interfaces between a generic accelerator and the ESP
accelerator implementations and monitors their outputs. Since the NoC, as illustrated by the block diagram of two accelerator tiles
ESP accelerator socket is modeled by the testbench, the completion in Fig. 2. First, we replace the DMA engine with an AXI master port
of this step indicates that the integration of the accelerator is correct that converts AXI transactions into NoC packets. From the acceler-
and the accelerator is ready to be deployed in the SoC. ator designer viewpoint, this new proxy takes the role of the AXI
3. Software build. For each accelerator, ESP automates building crossbar that is commonly used for integration on a traditional bus-
both bare-metal binary and Linux image for testing. The bare-metal based SoC. Since NoC packets can be routed to both non-coherent
driver consists of a loadable file for the external memory available and coherent DMA planes, third-party accelerators can still benefit
on the target FPGA development board. In addition, a standard srec from run-time selection of the coherence model. We remove the
text file is dumped for the memory model used in simulation. optional private L2 cache and the translation-lookaside buffer. The
4. SoC configuration. A simple graphical user interface (GUI) former, when is needed, is usually part of the third-party IP, while
helps designers create an ESP configuration file by selecting where the latter is bypassed by the AXI master interface, which can issue
the new accelerators should be located in the tiles grid, together requests to physical memory based on the internal behavior of the
with processor cores and memory tiles. accelerator.
5. System-level simulation. For each target FPGA board, ESP Similarly, we replace the CSR logic with an existing APB proxy
provides a testbench to simulate the complete execution of a bare- interface. While registers are part of the native ESP accelerator
metal program, including boot loader and interaction with periph- socket, CSRs of a third-party accelerator are integrated into the
erals. Users can specify the target program to be the bare-metal accelerator design. Our APB proxy is used in ESP to connect the
driver for the new accelerator, thereby testing how the accelerator slave port of memory-mapped registers and devices with the NoC.
behaves when driven by an ESP processor tile. This component can be reused as is to integrate a third-party accel-
6. FPGA prototyping. When targeting one of the supported erator. For flexibility, we add an optional adapter to convert APB
FPGA boards, ESP users can prototype their SoC without prior transactions to AXI-Lite or full AXI slave packets.
FPGA experience. The generation of the bitstream file, the program- Finally, we modify the proxy responsible for interrupt delivery so
ming script and the deployment of software are fully automated. that it accepts an interrupt clear (CLR) message from the interrupt
The ESP instance is controlled through an Ethernet interface that controller. This change is necessary to integrate accelerators, like
allows quick loading of programs into main memory, updating the NVDLA, that implement two common behaviors: (a) level-sensitive
boot loader into an on-chip RAM and resetting the processors. interrupt requests (IRQ) with no return to zero required; (b) over-
lapped execution of one task and configuration of the next one. In
this scenario, when software occasionally slows down, due to a vari-
3.3 Third Party Socket able workload or to the non-deterministic behavior of the operating
The ESP flows described above rely on a set of modular components system, distributing interrupts over the NoC can lead to a deadlock
integrated with latency-insensitive interfaces. These are cache con- condition. Consider the following steps: (1) the accelerator asserts
trollers, adapters, arbiters and decoders that act as proxies for the and holds an IRQ; (2) the interrupt controller receives the message
Ariane + NVDLA: Seamless Third-Party IP Integration with ESP

from the interrupt proxy and another proxy on this tile sends a in fact, the main distinguishing factors that make ESP particularly
message to the processor tile; (3) the processor acknowledges the suitable for heterogeneous SoC design. Nevertheless, since proces-
IRQ, saves the state of the interrupt controller, masks further in- sors retain a fundamental role in controlling the execution of any
terrupts and enters the interrupt handler routine; (4) the interrupt SoC, we made sure to extend the flexibility of ESP also to the choice
handler issues an accelerator-specific clear to the accelerator; (5a) of different processor cores. To do so, we designed the processor
upon receiving the clear, the accelerator may be ready to raise a tile by leveraging a combination of the same proxy components
second IRQ and can do so without first bringing the interrupt level and adapters that we use for the accelerator sockets. The resulting
to zero; (5b) alternatively, even if the interrupt level returns to zero, structure is illustrated in Fig. 2. The 64-bits RISC-V Ariane proces-
the new IRQ may reach the interrupt controller when interrupts sor core from ETH Zurich [1] is transparently integrated in ESP
are still masked; (6) the processor restores the state of the interrupt through AXI master ports. These are part of the AXI proxy used in
controller as it exits the interrupt handler routine. If either step the third-party accelerator socket. ESP offers an AHB adapter as
(5a) or (5b) occurs, then the second IRQ message gets lost and the well to interface with the 32-bits SPARC-V8 Leon3 processor core
accelerator gets stuck. We solve this problem in a general way that from Cobham Gaisler [13]. Similarly, non-cacheable read and write
applies not only to NVDLA. We make the proxy on the interrupt operations are forwarded to the NoC with the APB adapter that is
controller tile send a clear message to the accelerator tile when used across all ESP tiles to expose memory-mapped registers and
the processor restores the interrupts state. In this way, while the devices to software. The L2 cache is just an instance of the optional
interrupt handler sends an accelerator-specific clear message, the private cache in the ESP accelerator socket. The write-back L2
interrupt controller proxy sends a generic clear message, informing cache and the companion LLC and directory splits, located in each
the accelerator tile that a new interrupt can be correctly received. memory tile, implement an extended MESI protocol that supports
run-time reconfiguration of the accelerator coherence models [22].
3.4 Third Party Accelerator Flow The interrupt-level proxy, represented in Fig. 2 as a simple queue, is
the only implementation-specific component in the ESP processor
After implementing the third-party socket, we augment the ESP
tile. This proxy is one half of a network adapter that allows pro-
infrastructure to implement a new flow that users can leverage to
cessor cores to interact with the platform interrupt controller and
integrate their existing accelerator IP blocks in a few simple steps:
the system timer. Both are located in the ESP miscellaneous I/O
1. Accelerator Definition. Fill in a short XML file with some
tile, which hosts all peripherals shared in the system (except from
key information. This includes a unique accelerator name and ID,
memory), i.e.: the Ethernet NIC, UART, a digital video interface and
the name of reset and clock signals, an optional prefix for the AXI
a debug link to control ESP prototypes on FPGA.
interface signals, and the user-defined width of AXI control signals.
The interrupt-level proxy delivers single-flit packets over the
2. Source RTL. Create a list file for each type of source RTL,
NoC. In the case of RISC-V, these packets can only originate from
including Verilog, SystemVerilog, VHDL and VHDL packages.
the interrupt controller or the timer in the I/O tile and terminate at
3. Make. Create a Makefile with all targets that apply among RTL
one processor tile. Interrupt claim, acknowledge or clear occur via
generation (vmod), linux device driver (kmd), user-space application
memory-mapped register accesses. Conversely, when generating
(umd), bare-metal driver (bmd).
an ESP instance with the Leon3 core, interrupt-level request and
4. Software Objects. Create a list file for driver modules, soft-
acknowledge use a custom protocol that requires the proxy to send
ware executable, libraries, and any other binary required by the
single-flit packets from processor tiles to the I/O tile as well. The
user application.
payload in these packets depends on the implementation-specific
5. RTL Wrapper. Write a Verilog top-level wrapper to assign
protocol.
any non-standard input port of the third-party accelerator (e.g. dis-
Thanks to the modularity of the ESP tiles, our team integrated
able testmode, if present) and expose the AXI and APB interfaces
Ariane in the span of a few weeks, while keeping the option of
for the new ESP socket. This last step mainly consists in attach-
Leon3 as an alternative core. This is possible because we rely on ESP
ing wires without implementing any logic. ESP users may look
standard bus adapters and proxies to decouple all platform services
at the NVDLA integration example to evaluate the simplicity of
from the particular third-party IP block to be integrated. The sole
implementing the RTL wrapper for the TPF.
exception is the simple interrupt-level proxy, which requires a
After applying these steps, users can configure an instance of ESP,
different implementation for each processor.
run simulations, and prototype their system on FPGA by following
The resulting system can execute any RISC-V program as is, with
the SoC flow described in Section 3.2, starting from Step 3 (software
no ESP-specific patches. This includes Linux, the bootloader of
build). The only notable difference is that the third-party software,
Ariane and third-party device drivers, such as the NVDLA runtime.
including device drivers and applications, is compiled by leveraging
Given the presence of implementation-dependent key compo-
the user-specific Makefile targets described above. For each IP, a
nents, such as the interrupt controller, integrating different third-
new folder is generated in the Linux file system image. This folder is
party processor cores cannot be automated. Nevertheless, the inte-
available inside root home directory after booting the ESP instance.
gration of Ariane proves that ESP still greatly facilitates the task.
We envision to support more processor cores in the near future,
4 PROCESSOR INTEGRATION e.g. by extending the set of proxy and adapters to TileLink and
We developed ESP with a holistic system view, rather than focus- Wishbone. To name a popular example, supporting TileLink would
ing on processor cores like other open-source SoC generators. The allow ESP to seamlessly integrate instances of the RISC-V processor
native ESP accelerator flow and the new TPF for accelerators are, that is part of the Rocket Chip Generator from UC Berkeley [3].
 Giri, et al.

Table 1: Neural Networks for NVDLA Evaluation

4 3.9

frames / sec (normalized)

Model Dataset Layers Input Model Size frames/sec @50MHz MNIST LeNet

LeNet MNIST 9 1x28x28 1.7 MB 3.8 3.1

Convnet CIFAR-10 13 3x32x32 572 KB 4.5 3
SimpleNet MNIST 44 1x28x28 21 MB 1.3
ResNet-50 ILSVRC2012 229 3x224x224 98 MB 0.4
2.1
2
5 EXPERIMENTAL EVALUATION
Although the NVDLA is a highly configurable accelerator, the 1 1

NVDLA Compiler supports only two configurations: NVDLA full 1 NVDLA 2 NVDLA 3 NVDLA 4 NVDLA
1 mem ctrl 2 mem ctrl 3 mem ctrl 4 mem ctrl
and NVDLA small. To test and evaluate the integration of NVDLA in
ESP, we use the NVDLA small, which has an 8-bit integer precision, Fig. 4: Scaling NVDLA instances and DDR channels.
64 multiply-and-accumulate units, and a 64-bit AXI channel.
The task parallelization delivers an approximately linear increase
Table 1 reports the four neural networks for image classification
in performance. For instance, four NVDLA instances with four
that we use in these experiments together with their characteristics.
memory channels bring a 4× speedup for LeNet.
For each network, starting from the Caffe model and topology
specified in prototxt format, we generate a calibration table needed
for adjusting the network model (trained in full precision) to work
6 RELATED WORK
at the 8-bit precision of NVDLA small. We then leverage the NVDLA As the OSH movement has greatly benefited from the success
software stack and feed model, topology and calibration table to the of the RISC-V project [4, 23], the majority of agile flows avail-
NVDLA compiler, which produces an NVDLA Loadable containing able to the community revolve around processor-centric systems.
the layer-by-layer information to configure NVDLA. At runtime, For instance, OpenPiton, the first SMP Linux booting RISC-V sys-
the NVDLA User Mode Driver loads Loadable and input images, tem, supports many Ariane cores with a coherence protocol that
and it submits inference jobs to the NVDLA Kernel Mode Driver. extends across multiple chips [7] and can support multi-ISA re-
SoC Configuration. Once integrated in ESP, an accelerator search [8]. The UC Berkeley team that created RISC-V has released
can be selected with the GUI and instantiated in multiple tiles several projects based on their innovative functional RTL language
during the SoC configuration step. We demonstrate the flexibility Chisel [5]. These open-source systems originate from the Rocket
and the integration capabilities of ESP by generating various SoC Chip Generator, which connects multiple RISC-V cores through a
architectures that include one processor tile with the Ariane core, coherent TileLink bus [31]. Celerity is a many-core RISC-V SoC that
and different numbers of memory tiles and third-party accelerator combines open-source RISC-V processors and a single HLS-based
tiles containing NVDLA. The User Mode Driver and the Kernel neural-network accelerator [16], by leveraging the Rocket Chip and
Mode Driver run on Ariane, which offloads the inference jobs to its custom co-processor interface RoCC. Rocket Chip is also at the
the NVDLA instances as needed. When selecting multiple memory heart of FireSim [28], an FPGA-accelerated RTL simulator that was
tiles, ESP automatically partitions the memory hierarchy to leverage used to simulate the integration of NVDLA in Rocket Chip [18].
the increased off-chip communication bandwidth. Each memory All of these open-source frameworks focus mostly on processor
tile contains a DDR-4 interface to a partition of main memory. cores; reported case studies on the integration of accelerators show
FPGA Prototyping. We deployed each SoC on a proFPGA quad that they are either tight to the cores as co-processors [16, 31],
Virtex Ultrascale Prototyping System, which mounts Xilinx XCVU440 or connected with external bus adapters outside of the SoC back-
FPGAs. On this FPGA, the target ESP runs at 50MHz. First, we ran bone interconnect [18]. The ESP architecture, instead, implements
inference jobs on a single NVDLA instance for the networks in a distributed system which is inherently scalable, modular and het-
Table 1, which reports the average number of frames per second erogeneous. Processors and loosely-coupled accelerators [15] are
(fps) processed by an SoC configuration with one NVDLA and one given the same importance in the SoC. This system-centric view,
memory tile. The performance depends on the network size, vary- as opposed to a processor-centric view, distinguishes ESP from
ing between 0.4 fps for ResNet50 and 4.5 fps for Convnet. As a other OSH platforms. Furthermore, the ESP methodology supports
reference, a performance of 7.3 fps is reported [36] for the ResNet50 different design flows, without imposing any particular tie on the
with an ASIC implementation of NVDLA Small running at a clock choice of the accelerator specification language and synthesis tool.
frequency of 1GHz, which is twenty time faster than ours.
Performance can be improved by parallelizing the execution of 7 CONCLUSIONS
large batches of images across multiple instances of the NVDLA. We augmented Open ESP with support for the integration of third-
With ESP, it is easy to explore the design space of possible SoC party IP blocks, by developing new HW/SW socket interfaces to the
configurations by tuning the number of NVDLA instances and ESP architecture and a new design flow to the ESP methodology.
memory channels utilized in parallel. Since the User Mode and While these contributions have a general nature, we demonstrated
Kernel Mode Drivers provided for NVDLA currently work with a them by realizing FPGA prototypes of SoCs that feature two major
single NVDLA device, we patched them to enable the simultaneous OSH resources: the Ariane processor and the NVDLA accelerator.
invocation of multiple NVDLA instances from the Ariane core. Acknowledgments. This research was supported in part by DARPA (C#:
Fig. 4 show the results for four SoC configurations, each with an HR001118C0122). The views, opinions and/or other findings expressed are
increasing number of NVDLA instances and memory channels uti- those of the authors and should not be interpreted as representing the official
lized in parallel, processing the MNIST dataset with LeNet network. views or policies of the Department of Defense or the U.S. Government.
Ariane + NVDLA: Seamless Third-Party IP Integration with ESP

REFERENCES [25] Mark Horowitz. Computing’s energy problem (and what we can do about it).
[1] Ariane. www.github.com/pulp-platform/ariane. In Digest of Technical Papers of the International Solid-State Circuits Conference
[2] ARM. AMBA AXI and ACE Protocol Specification. http://infocenter.arm.com/help/ (ISSCC), pages 10–14, February 2014.
index.jsp?topic=/com.arm.doc.ihi0022e/index.html. [26] IBM. The CoreConnect Bus Architecture. https://www.ibm.comintel.com/content/
[3] Krste Asanovic, Rimas Avizienis, Jonathan Bachrach, Scott Beamer, David Bian- dam/www/programmable/us/en/pdfs/literature/manual/mnl_avalon_spec.pdf .
colin, Christopher Celio, Henry Cook, Daniel Dabbelt, John Hauser, Adam Izraele- [27] Intel. Avalon Interface Specifications. https://www.intel.com/content/dam/www/
vitz, Sagar Karandikar, Ben Keller, Donggyu Kim, John Koenig, Yunsup Lee, Eric programmable/us/en/pdfs/literature/manual/mnl_avalon_spec.pdf .
Love, Martin Maas, Albert Magyar, Howard Mao, Miquel Moreto, Albert Ou, [28] Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid,
David A. Patterson, Brian Richards, Colin Schmidt, Stephen Twigg, Huy Vo, and Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra,
Andrew Waterman. The Rocket Chip generator. Technical Report UCB/EECS- Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach,
2016-17, UC Berkeley, April 2016. and Krste Asanović. FireSim: FPGA-accelerated cycle-exact scale-out system
[4] Krste Asanovic and David Patterson. The case for open instruction sets. Micro- simulation in the public cloud. In Proceedings of the International Symposium on
processor Report, August 2014. Computer Architecture (ISCA), pages 29–42, 2018.
[5] Jonathan Bachrach, Huy Vo, Brian Richards, Yunsup Lee, Andrew Waterman, [29] Keras. https://github.com/fchollet/keras, 2017.
Rimas Avižienis, John Wawrzynek, and Krste Asanović. Chisel: Constructing [30] Brucek Khailany, Evgeni Khmer, Rangharajan Venkatesan, Jason Clemons, Joel S.
hardware in a Scala embedded language. In Proceedings of the ACM/IEEE Design Emer, Matthew Fojtik, Alicia Klinefelter, Michael Pellauer, Nathaniel Pinckney,
Automation Conference (DAC), pages 1216–1225, 2012. Yakun Sophia Shao, Shreesha Srinath, Christopher Torng, Sam (Likun) Xi, Yanqing
[6] J. Bai et al. Onnx: Open neural network exchange. https://github.com/onnx/onnx, Zhang, and Brian Zimmer. A modular digital VLSI flow for high-productivity
2018. SoC design. In Proceedings of the ACM/IEEE Design Automation Conference (DAC),
[7] Jonathan Balkind, Katie Lim, Fei Gao, Jinzheng Tu, David Wentzlaff, Michael 2018.
Schaffner, Florian Zaruba, and Luca Benini. OpenPiton+Ariane: the first SMP [31] Yunsup Lee, Andrew Waterman, Henry Cook, Brian Zimmer, Ben Keller, Alberto
Linux-booting RISC-V system scaling from one to many cores. In Workshop on Puggelli, Jaehwa Kwak, Ruzica Jevtic, Stevo Bailey, Milovan Blagojevic, Pi-Feng
Computer Architecture Research with RISC-V (CARRV), 2019. Chiu, Rimas Avizienis, Brian Richards, Jonathan Bachrach, David Patterson, Elad
[8] Jonathan Balkind, Katie Lim, Michael Schaffner, Fei Gao, Grigory Chirkov, Ang Alon, Bora Nikolic, and Krste Asanovic. An agile approach to building RISC-V
Li, Alexey Lavrov, Tri M. Nguyen, Yaosheng Fu, Florian Zaruba, Kunal Gulati, microprocessors. IEEE Micro, 36(2):8–20, Mar.-Apr. 2016.
Luca Benini, and David Wentzlaff. BYOC: a "bring your own core" framework for [32] Hung-Yi Liu, Michele Petracca, and Luca P. Carloni. Compositional system-level
heterogeneous-ISA research. In Proceedings of the ACM International Conference design exploration with planning of high-level synthesis. In Proceedings of the
on Architectural Support for Programming Languages and Operating Systems IEEE Conference on Design, Automation, and Test in Europe (DATE), pages 641–646,
(ASPLOS), pages 699–714, March 2020. March 2012.
[9] UC Berkeley. TileLink 0.3.3 Specifications. https://docs.google.com/document/d/ [33] Paolo Mantovani, Emilio G. Cota, Christian Pilato, Giuseppe Di Guglielmo, and
1Iczcjigc-LUi8QmDPwnAu1kH4Rrt6Kqi1_EUaCrfrk8/pub. Luca P. Carloni. Handling large data sets for high-performance embedded appli-
[10] Luca P. Carloni. From latency-insensitive design to communication-based system- cations in heterogeneous systems-on-chip. In Proceedings of the International Con-
level design. Proceedings of the IEEE, 103(11):2133–2151, November 2015. ference on Compilers, Architectures, and Synthesis of Embedded Systems (CASES),
[11] Luca P. Carloni. The case for Embedded Scalable Platforms. In Proceedings of the pages 3:1–3:10, October 2016.
ACM/IEEE Design Automation Conference (DAC), pages 17:1–17:6, June 2016. [34] Paolo Mantovani, Emilio G. Cota, Kevin Tien, Christian Pilato, Giuseppe
[12] Luca P. Carloni, Kenneth L. McMillan, and Alberto L. Sangiovanni-Vincentelli. Di Guglielmo, Ken Shepard, and Luca P. Carloni. An FPGA-based infrastructure
Theory of latency-insensitive design. IEEE Transactions on CAD of Integrated for fine-grained DVFS analysis in high-performance embedded systems. In Pro-
Circuits and Systems, 20(9):1059–1076, September 2001. ceedings of the ACM/IEEE Design Automation Conference (DAC), pages 157:1–157:6,
[13] Cobham Gaisler. Leon3 processor. www.gaisler.com/index.php/products/processors/ 2016.
leon3. [35] Paolo Mantovani, Giuseppe Di Guglielmo, and L. P. Carloni. High-level synthesis
[14] Columbia SLD Group. ESP Release. www.esp.cs.columbia.edu, 2019. of accelerators in embedded scalable platforms. In Proceedings of the Asia and
[15] Emilio G. Cota, Paolo Mantovani, Giuseppe Di Guglielmo, and Luca P. Carloni. South Pacific Design Automation Conference (ASPDAC), pages 204–211, January
An analysis of accelerator coupling in heterogeneous architectures. In Proceedings 2016.
of the ACM/IEEE Design Automation Conference (DAC), 2015. [36] NVIDIA. NVDLA Primer. www.nvdla.org/primer.html, 2018.
[16] Scott Davidson, Shaolin Xie, Christopher Torng, Khalid Al-Hawai, Austin Rovin- [37] NVIDIA. NVIDIA Deep Learning Accelerator. www.nvdla.org, 2018.
ski, Tutu Ajayi, Luis Vega, Chun Zhao, Ritchie Zhao, Steve Dai, Aporva Amarnath, [38] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang,
Bandhav Veluri, Paul Gao, Anuj Rao, Gai Liu, Rajesh K. Gupta, Zhiru Zhang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.
Ronald Dreslinski, Christopher Batten, and Michael B. Taylor. The Celerity Automatic differentiation in PyTorch. 2017.
open-source 511-Core RISC-V tiered accelerator fabric: Fast architectures and [39] Luca Piccolboni, Paolo Mantovani, Giuseppe Di Guglielmo, and Luca P. Carloni.
design methodologies for fast chips. IEEE Micro, 38(2):30–41, February 2018. COSMOS: Coordination of high-level synthesis and memory optimization for
[17] Javier Duarte, Song Han, Philip Harris, Sergo Jindariani, Edward Kreinar, Ben- hardware accelerators. ACM Transactions on Embedded Computing Systems,
jamin Kreis, Jennifer Ngadiuba, Maurizio Pierini, Nga Tran, and Zhenbin Wu. 16(5s):150:1–150:22, September 2017.
Fast inference of deep neural networks in FPGAs for particle physics. Journal of [40] Christian Pilato, Paolo Mantovani, Giuseppe Di Guglielmo, and Luca P. Car-
Instrumentation, 13(07):P07027–P07027, July 2018. loni. System-level optimization of accelerator local memory for heterogeneous
[18] Farzad Farshchi, Qijing Huang, and Heechul Yun. Integrating NVIDIA deep systems-on-chip. IEEE Transactions on CAD of Integrated Circuits and Systems,
learning accelerator (NVDLA) with RISC-V SoC on FireSim. CoRR, abs/1903.06495, 36(3):435–448, March 2017.
2019. [41] David Pursley and Tung-Hua Yeh. High-level low-power system design opti-
[19] Davide Giri, Kuan-Lin Chiu, Giuseppe Di Guglielmo, Paolo Mantovani, and Luca P. mization. In VLSI-DAT, pages 1–4, April 2017.
Carloni. ESP4ML: platform-based design of systems-on-chip for embedded [42] hls4ml. https://fastmachinelearning.org/hls4ml.
machine learning. In Proceedings of the IEEE Conference on Design, Automation, [43] Mohandeep Sharma and Dilip Kumar. Wishbone bus architecture - a survey and
and Test in Europe (DATE), March 2020. comparison. International Journal of VLSI Design and Communication Systems,
[20] Davide Giri, Paolo Mantovani, and Luca P. Carloni. Accelerators & coherence: 3(2):107–124, April 2012.
An SoC perspective. IEEE Micro, 38(6):36–45, November 2018. [44] Xilinx. The Xilinx Vivado design suite. https://www.xilinx.com/products/design-
[21] Davide Giri, Paolo Mantovani, and Luca P. Carloni. NoC-based support of tools/vivado.html.
heterogeneous cache-coherence models for accelerators. In Proceedings of the [45] Florian Zaruba and Luca Benini. The cost of application-class processing: Energy
International Symposium on Networks-on-Chip (NOCS), pages 1:1–1:8, October and performance analysis of a Linux-ready 1.7-GHz 64-Bit RISC-V core in 22-nm
2018. FDSOI technology. IEEE Transactions on Very Large Scale Integration Systems,
[22] Davide Giri, Paolo Mantovani, and Luca P. Carloni. Runtime reconfigurable 27(11):2629–2640, November 2019.
memory hierarchy in embedded scalable platforms. In Proceedings of the Asia and [46] Brian Zimmer, Rangharajan Venkatesan, Yakun Sophia Shao, Jason Clemons,
South Pacific Design Automation Conference (ASPDAC), pages 719–726, January Matthew Fojtik, Nan Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney,
2019. Priyanka Raina, Stephen G. Tell, Yanqing Zhang, William J. Dally, Joel S. Emer,
[23] Samuel Greengard. Will RISC-V revolutionize computing? Communication of C. Thomas Gray, Stephen W. Keckler, and Brucek Khailany. A 0.32-128 TOPS,
ACM, 63(5):30–32, April 2020. scalable multi-chip-module-based deep neural network inference accelerator with
[24] Gagan Gupta, Tony Nowatzki, Vinay Gangadhar, and Karthikeyan Sankaralingam. ground-referenced signaling in 16 nm. IEEE J. of Solid-State Circuits, 55(4):920–932,
Kickstarting semiconductor innovation with open source hardware. IEEE Com- April 2020.
puter, 50(6):50–59, June 2017.