A Hardware Evaluation Framework for Large Language Model Inference

Abstract

The past year has witnessed the increasing popularity of Large Language Models (LLMs). Their unprecedented scale and associated high hardware cost have impeded their broader adoption, calling for efficient hardware designs. With the large hardware needed to simply run LLM inference, evaluating different hardware designs becomes a new bottleneck.

This work introduces LLMCompass, a hardware evaluation framework for LLM inference workloads. LLMCompass is fast, accurate, versatile, and able to describe and evaluate different hardware designs. LLMCompass includes a mapper to automatically find performance-optimal mapping and scheduling. It also incorporates an area-based cost model to help architects reason about their design choices. Compared to real-world hardware, LLMCompass’ estimated latency achieves an average 10.4% error rate across various operators with various input sizes and an average 4.1% error rate for LLM inference. With LLMCompass, simulating a 4-NVIDIA A100 GPU node running GPT-3 175B inference can be done within 16 minutes on commodity hardware, including 26,400 rounds of the mapper’s parameter search.

With the aid of LLMCompass, this work draws architectural implications and explores new cost-effective hardware designs. By reducing the compute capability or replacing High Bandwidth Memory (HBM) with traditional DRAM, these new designs can achieve as much as 3.41x improvement in performance/cost compared to an NVIDIA A100, making them promising choices for democratizing LLMs.

LLMCompass is planned to be fully open-source.

I Introduction

Large Language Models (LLMs), the technology behind OpenAI ChatGPT [47], Github Copilot [20], and Google Bard [22], are gaining widespread attention from the whole society. The capability of LLMs is related to their model size [29, 27], and larger models [7, 10] show impressive abilities [71] compared to smaller counterparts [53, 15], with future models expected to exceed trillions of parameters [16].

This unprecedented scale of LLMs poses challenges to deployment. Serving a GPT-3 (175B parameters) inference requires a minimum of five NVIDIA A100s solely to accommodate the model parameters (in half precision). This substantial hardware cost impedes the broader adoption of LLMs and motivates computer architects to design more cost-effective hardware. We identify three challenges that exist in designing hardware for LLM inference:

Lack of tools to evaluate hardware designs. Before diving into writing the RTL code, hardware designers may want to first sketch and compare different design choices. There are many properties we want for such a hardware evaluation tool before writing RTL. ① Fast and accurate. Due to the intense compute and memory hardware demand required for LLM inference, this tool needs to be as fast as possible without sacrificing accuracy. ② Architecturally descriptive. This tool should be general enough to describe different design choices: If it only applies to a specific architecture, the design space for computer architects will be limited. ③ Performance-optimal. The hardware performance is also affected by how the software is programmed (e.g., how to map the workload to the hardware). The evaluation tool should optimize this software domain to fully demonstrate the hardware capability of each design. ④ Cost-aware. We also want to know how different hardware design choices affect the hardware cost to reason about cost-performance trade-offs.

Existing tools fail to meet these requirements. Roofline model analysis is fast but not accurate, and cycle-level simulators are accurate but slow. FPGA emulation is accurate and provides area statistics but requires significant engineering effort. To evaluate large-scale hardware designs in the era of LLMs, a new hardware evaluation tool is needed.

Figure 1: An Overview of LLMCompass. LLMCompass can aid the hardware design process as a versatile evaluation tool.

Lack of knowledge on how different hardware design choices affect LLM inference performance. As an emerging application, the hardware characteristics of LLMs remain to be understood. Besides the large volume of compute and memory requirements, LLMs are also unique in their auto-regressive way of generating tokens. We are interested in exploring whether these properties of LLMs will change common architecture wisdom.

Lack of cost-effective hardware designs to democratize LLMs. LLMs are powerful and capable, but are cost-prohibitive to deploy. To serve GPT-3, a DGX A100 compute node can cost over $100,000 USD [44], with each NVIDIA A100 featuring 54B transistors and 80 GB of High Bandwidth Memory (HBM). This high hardware cost hinders democratizing LLMs.

In this paper, we tackle these challenges and make three main contributions.

(1) We introduce LLMCompass, a hardware evaluation framework for LLM inference workloads (Sec. III). LLMCompass leverages the fact that mainstream ML hardware platforms share many architectural commonalities, allowing us to develop a general hardware description template for them. We also observe LLMs’ computational graphs are composed of dense operators: matrix multiplication, softmax, layer normalization, etc., all of which have a structural and hence predictable compute and memory access pattern. This allows LLMCompass to perform faster, higher-level tile-by-tile (block-by-block) simulations without losing accuracy compared to cycle-accurate simulators. The framework implements a mapper to manually manage the memory hierarchy and find the performance-optimal mapping and schedule scheme for dense workloads. LLMCompass also features a cost and area model based on public parameters to help designers reason about different design choices.

LLMCompass is validated on three commercial hardware designs: NVIDIA A100 [46], AMD MI210 [2], and Google TPUv3 [43, 28]. Compared to real-world hardware, LLMCompass’ estimated latency achieves 10.4% error rate across various operators with various input sizes and 4.1% error rate for LLM inference. Implemented in Python, LLMCompass is still fast. It takes only 15-16 minutes to simulate a 4-A100 GPU node running GPT-3 175B inference, including 26,400 rounds of the mapper’s parameter search (Figure 4(i), tested on one core of Intel Xeon Gold 6242R CPU @ 3.10GHz).

(2) We leverage LLMCompass to draw architectural implications and explore how hardware design choices affect LLM inference (Sec. IV). We find that prefill and decoding pose different hardware requirements. Prefill can significantly benefit from more compute capability and buffers, while decoding barely gains from these and is more sensitive to memory bandwidth. These insights inspire us to think about new hardware design paradigms.

(3) We propose two cost-effective hardware designs different from conventional wisdom (Sec. V). We find that today’s hardware design paradigms tend to fit massive compute capability and SRAMs in a huge die connected to high-end HBMs. We analyze the LLM inference characteristics and show how current hardware designs are inefficient. ① As LLM inference is mostly IO-bound, HBMs can be used to achieve low latency. However, HBM memory capacity limits the batch size, making it hard to fully utilize the massive compute capability. Based on this observation, we find that 95.3% of the original performance can still be achieved even if we prune the compute capability and buffer size by half. ② Larger batch size can significantly improve throughput as the model parameters are only read once for the whole batch. As memory capacity limits the batch size therefore limiting throughput, we propose to replace HBMs with traditional DRAM. We find that a larger batch size can compensate for the loss in memory bandwidth and can bring a 1.42x improvement in throughput and a 3.41x improvement in performance/cost.

II Background

II-A Large Language Models and Transformers

Large Language Models are variations of Transformer models [67] with a considerable amount of parameters that have been pre-trained on large corpora of data [38]. Today’s LLMs can have as much as one trillion parameters [16]. Compared to smaller models, larger models (e.g. GPT-3 175B [7]) showcase a remarkable set of capabilities such as emergent abilities [71] and few-shot learning [7]. This increase in model size and the consequent memory and compute requirements have posed unique challenges for hardware.

Figure 2: A Decoder-Only Transformer Layer with Tensor Parallelism. GPT-3 175B [7] consists of a stack of 96 such layers.

Figure 3: LLMCompass’ Hardware Description Template. In this example, each device has 2 cores and each core has 2 lanes.

We focus on Decoder-only Transformer models [51], which is the architecture adopted by most of the LLMs today: LLaMA [64], GPTs [7, 53], Bloom [73], PaLM [10], etc. The basic building blocks of these models are Transformer layers. As illustrated in Figure 2, each layer comprises a Multi-Head Attention block followed by an MLP block. These layers are then stacked together, forming the bulk of an LLM’s memory and compute requirement. Transformers also use learned Vocabulary and Position embeddings, but for large models like GPT-3, these do not contribute significantly to either the memory or compute requirement ($<2$%). Without losing generality, we focus on Multi-Head Attention Transformers (GPT-style). There are other variations such as Multi-Query Attention [10], Mixture-of-Experts [16], and parallel Attention and MLP [10]. LLMCompass seamlessly supports all these possible variations as they share a common set of operators.

II-B LLM Inference

Given an input prompt and the required number of output tokens, LLM inference can be divided into two stages [52]. ① Prefill: Processing the input prompt and computing the KV cache. The Key Value (KV) cache refers to the stored Key and Value tensors of the Attention block in each layer [52]. ② Decoding: Generating output tokens one after another in an auto-regressive manner: The Key and Value of the newly generated token will be concatenated to the KV cache and used for generating the next token. The latency of prefill and decoding is mostly determined by the input and output sequence lengths, respectively. In prefill, as the entire input sequence needs to be multiplied by all the parameters, it is usually bounded by compute. In decoding, each new token needs to be multiplied by all the parameters and concatenated to the KV cache, so decoding is usually bounded by reading parameters and KV cache.

Latency and throughput are the key metrics to evaluate LLM inference systems. For interactive use cases such as chatbots [47], it is imperative to optimize latency. For background data processing use cases such as data wrangling [40] or form processing [8], throughput is more important. The tradeoff between latency and throughput is determined by batch size: larger batch increases throughput at the cost of higher latency.

II-C Parallelizing LLM Inference

Due to the large volume of compute and memory operations, it is beneficial to parallelize LLM inference across multiple devices. This leads to much better performance and can be necessary if the model’s parameters along with the KV cache do not fit in a single device’s memory. For LLM inference, there are two model parallelization schemes: pipeline parallelism and tensor parallelism. In pipeline parallelism, different layers of the model are grouped into sequential partitions and assigned to different devices like a hardware pipeline. This scheme has the effect of considerably increasing throughput at the expense of increased latency. On the other hand, tensor parallelism, as proposed by Megatron-LM [59], partitions each layer of the model across the available devices, thereby decreasing latency at the cost of frequent device-device communication and synchronization. As shown in Figure 2, this scheme requires two all-reduce for each Transformer layer, one after the Attention block and another after the MLP block.

III LLMCompass

An overview of LLMCompass (Large Language Model Computation Performance and Area Synthesis) is shown in Figure 1. To evaluate the performance (e.g., throughput and latency) of running a Transformer-based large language model on a hardware system, two inputs are needed: the computational graph of the LLM and a hardware description (Section III-A). Given the input, the performance model (Section III-B) generates a performance report. The mapper conducts a parameter search along with the architecture simulator to find the best mapping and scheduling scheme. At the same time, the area model (Section III-D) generates the area and cost report.

III-A Hardware Description Template

The hardware description template of LLMCompass is introduced below, as shown in Figure 3:

• A system (e.g., a DGX node) is composed of multiple devices connected through a device-device interconnect (e.g., NVLink or Infinity Link).

• Each device (e.g., a GPU) is composed of multiple cores, a shared global buffer, and an off-chip main memory. The global buffer (e.g., L2 cache in NVIDIA GPUs) is connected to the main memory, device-device interconnect, and all the cores.

• Each core (e.g., a Stream Multiprocessor in NVIDIA GPUs) can have multiple lanes sharing a local buffer (e.g., L1 cache in NVIDIA GPUs). The local buffer is connected to the global buffer through the on-chip interconnect.

• Each lane is independent from each other and has its own vector unit, systolic array, registers and control logic.

TABLE I: Examples of LLMCompass’s Hardware Description

Key Specifications	NVIDIA	AMD	Google
	A100 [46]	MI210 [2]	TPUv3 [43]
Frequency (MHz)	1410	1700	940
Core count	108	104	2
Lane count	4	4	1
Vector width	32	16	$4\times 128$
Systolic array	$16\times 16$	$16\times 16$	$128\times 128$
Local buffer (KB)	192	80	8192
Global buffer (MB)	40	8	16384
Global buffer (bytes/clk)	5120	4096	490
Memory bandwidth (TB/s)	2	1.6	-
Memory capacity (GB)	80	64	-
Device-device bandwidth (GB/s)	600	300	162.5

Figure 4: Visualization of a Matrix Multiplication in LLMCompass as in Section III-B1.

In existing devices, the local and global buffers are usually on-chip SRAM: cache, scratchpad, or a combination of both. LLMCompass doesn’t distinguish between cache and scratchpad because the memory is explicitly managed by the mapper. We believe this assumption does not lose generality as a highly optimized library will also carefully manage the memory. The main memory is usually off-chip DRAM: HBM, DDR memory, CXL memory, etc, all of which can be described by our parameterized hardware description template.

We find this hardware description is general enough to describe the mainstream machine learning platforms of today: NVIDIA GPUs, AMD GPUs, and Google TPUs, as shown in Table I with a sample of key specifications listed. It is also flexible enough to explore future architectures.

III-B Performance Model

The computational graph of a Transformer is composed of a stack of Transformer layers. Each layer is composed of a series of operators, including matrix multiplication (Matmul), Softmax, layer normalization (LayerNorm), and activation functions (e.g., GELU [26] as in GPTs [7, 53]). In a multi-device setup, communication primitives such as all-reduce operators are also needed to perform tensor parallelism. The key challenge is how to simulate the performance of different operators and communication primitives on a given hardware system - this requires knowledge about the hardware and how to map and schedule operators on a multi-level compute system with a multi-level memory hierarchy.

To solve this, LLMCompass introduces a mapper and an architecture simulator to build a performance model. Conceptually, we simulate running an operator on the chosen hardware in a recursive manner: we first partition the problem into smaller sub-problems that can fit in the global buffer. The sub-problem is then divided into smaller sub-sub-problems that can fit in each core’s local buffer. The partitioning, mapping, and scheduling are generated by the mapper and a parameter search is conducted to find the optimal mapping and scheduling. LLMCompass always tries to find the performance-optimal mapping to fully demonstrate the hardware capability.

III-B1 Matrix Multiplication

The process of simulating a matrix multiplication is visualized in Figure 4. $\mathbf{A}$ is a $M\times K$ matrix with $M$ rows and $K$ columns. Similarly, $\mathbf{B}$ and $\mathbf{C}$ are $K\times N$ and $M\times N$ matrices respectively. A generalized matrix multiplication is defined as $\mathbf{C}=\mathbf{AB}+\mathbf{C}$.

From main memory to global buffer: To maximize data reuse, matrix multiplication is usually calculated in a tile-by-tile manner [32]. As shown on the left of Figure 4, matrix $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$ are divided into tiles small enough to fit into the global buffer. In each step, one $A\_{tile}_{m,k}$, $B\_{tile}_{k,n}$, and $C\_tile_{m,n}$ are read into the global buffer, the cores then perform the computation, and the results are written back.

From global buffer to local buffer: With tiles inside the global buffer, we now need to parallelize the computation of $C\_tile_{m,n}=A\_{tile}_{m,k}B\_{tile}_{k,n}+C\_tile_{m,n}$ on multiple cores. As shown in the middle of Figure 4, these tiles are further divided into smaller sub-tiles to fit in each core’s local buffer. It then becomes a scheduling problem to map sub-tiles onto cores.

The right of Figure 4 shows two possible schedule schemes:

• Schedule Scheme 1: Different cores working on different $C\_subtile$s in the same column. At wave 0, as core 0 and core 1 both need to read the same $B\_subtile$, their memory access to the global buffer should be merged. In our simulator, this memory access merging is automatically identified and taken care of. As the same core keeps updating the same $C\_subtile$, there is no need to first write the partial result and then read it from the global buffer. This Read-After-Write dependency is also automatically taken care of by the simulator.

• Schedule Scheme 2: Different cores working on the same $C\_subtile$. Core 0 and core 1 first read the data and calculate the partial results, then perform a reduction and write back the final results.

In reality, with more cores and more tiles, the schedule space can be more complicated than the example shown in Figure 4.

From local buffer to lanes: Similarly, within each core, the sub-tiles are further partitioned into sub-sub-tiles to be mapped to lanes sharing a local buffer. After that, the sub-sub-tiles are finally passed to the systolic arrays. LLMCompass leverages SCALE-Sim [57, 56], a cycle-level systolic array simulator, to mimic the behavior of a systolic array and get the cycle count. LLCompass caches the results of SCALE-Sim into a look-up table to avoid duplicated simulation. A reduction will be performed by the vector unit if needed.

Mapper: A parameter search is performed by the mapper to determine the best tiling scheme and schedule scheme. To overlap computation with memory accesses, we also add software pipelines (double buffering) at each level of the memory hierarchy as scheduling options. The downside of enabling software pipeline is that it requires extra buffer space so the maximal tile size will be reduced, causing potentially lower utilization of systolic arrays. However, we find software pipeline to be beneficial in most cases.

III-B2 Communication Primitives

We use the link model as in AHEAD [1] and LogGP [3]. Suppose $L$ is the link latency, $O$ is the additional overhead associated with the data transfer, and $B$ is the link bandwidth. The latency $T$ to transfer $n$ bytes of data through a link is expressed in Equation 1 and 2:

$$T=L+O+\frac{\hat{n}}{B}$$ (1)

$$\hat{n}=\left\lceil\frac{n}{{MaxPayload}}\right\rceil*{Flit\_size}+n$$ (2)

On top of this, we implement ring all-reduce [49], which is a bandwidth-optimal all-reduce algorithm. We use a 16-byte $Flit\_size$ and a 256-byte $MaxPayload$ based on NVLinks [17]. We don’t model more communication primitives as LLM inference only requires all-reduce for tensor parallelism and peer-to-peer for pipeline parallelism.

III-B3 Other Operators

We also model Softmax, LayerNorm, and GELU following a similar methodology as in SectionIII-B1. The only differences are: ① They have fewer dimensions and are therefore simpler: Softmax and LayerNorm operate on two-dimensional data, and GELU operates on one-dimensional data, while Matmul operates on three-dimensional data. As each dimension requires tiling and scheduling, the mapper search space is much smaller. ② They do not use systolic arrays. Softmax is implemented with the online algorithm [37]. GELU is approximated with $tanh$ [26].

(a) Matmul with $M=8192$.

(b) Matmul with $N=K=12288$.

(c) Softmax with $N=4096$.                    

(d) LayerNorm with $N=4096$.

(e) LayerNorm with $M=4096$.

(f) Softmax with $M=4096$.                    

(g) GELU.

(h) All-reduce.

(i) GPU Prefill Latency.

(j) TPU Prefill Latency.

(k) GPU Decoding Latency.

(l) TPU Decoding Latency.               

Figure 5: Performance Model Validations. Matmul takes a $M\times K$ ($M$ rows and $K$ columns) and a $K\times N$ matrix as input. 12288 is the model dimension of GPT-3 [7]. Softmax and LayerNorm take a $M\times N$ matrix and perform normalization on the $N$ dimension. Prefill latency is measured by running one layer of GPT-3 with batch size 8 and sequence length 2048. Decoding latency is per GPT-3 layer per output token and is measured by the latency of generating the 1024th output token with batch size 8 and input sequence length 2048. For (a)-(g), a single GPU/TPU device is used. For (h)-(l), the 4-A100 GPU node and 8-TPUv3-Core TPU node are used with tensor parallelism.

III-C Performance Model Validation

In this section, we validate our framework against three real hardware platforms: (1) a datacenter GPU node with 4 NVIDIA A100 SXM4 GPUs (80 GB) fully connected by NVLinks; (2) a Google Cloud TPU node with 8 TPUv3 cores connected in a 2D torus topology; (3) an AMD MI210 GPU. The results are shown in Figure 5. For NVIDIA GPUs, CUDA 11.7 and PyTorch 2.0 are used to benchmark operators in half precision (FP16) with torch.compile enabled for LayerNorm and GELU to maximize performance. Communication primitive all-reduce is benchmarked with nccl-tests [41], a communication primitive performance benchmark for NVIDIA GPUs. For Google TPUs, JAX 0.4.18 is used to benchmark operators and communication primitives. Due to the hardware feature of TPUs, Matmul is benchmarked in bfloat16 (BF16) and all the other operators are in FP32. For AMD GPU, ROCm 5.4.2 and PyTorch 2.0 are used along with FP16 for Matmul and FP32 for other operators. The kernel launch overhead including the framework overhead is measured by running the operator with an input of size 1.

As shown in Figure 5, for Matmul, Softmax, LayerNorm, GELU, and all-reduce, LLMCompass achieves an average error rate of 9.0%, 12.0%, 11.3%, 5.0%, and 14.9% respectively. For LLM inference, LLMCompass achieves an average error rate of 0.69% and 7.5% for prefill and decoding respectively. On average, LLMCompass achieves a 10.4% error rate for different operators at various input sizes and a 4.1% error rate across the prefill and decoding stages.

Although not a perfect match to real-world hardware, LLMCompass is able to show a similar trend that a naive roofline model fails to show. For example, in Figure 4(d), as the reduction dimension of LayerNorm increases to an extreme, the throughput should drop due to the increasing reduction cost. LLMCompass is able to catch this trend.

LLMCompass’ results are totally interpretable without incorporating any fudge factor and we believe this interpretability is more important than perfectly matched results. Here are some possible causes of the mismatch between LLMCompass and real hardware:

• Lack of hardware knowledge. We have little knowledge about the micro-architecture of GPUs and TPUs (e.g., hardware pipeline design or scheduler design). With a large input size, the hardware is well utilized and some overhead can be hidden. However, with a small input size, it’s hard to hide the overhead and micro-architecture details affect performance significantly. Also, the Tensor Cores in NVIDIA GPUs and Matrix Cores in AMD GPUs are simulated as systolic arrays in LLMCompass, which may not be true in reality.

• Lack of software knowledge. We don’t know how operators and communication primitives are implemented on these platforms as they are closed-source libraries. We conduct a thorough parameter search for each input size to maximize performance, but in reality those libraries probably use heuristics to determine mapping and scheduling, which may not be optimal at all input sizes (e.g., we find that for a Matmul with $M=64$ and $N=K=12288$, AMD MI210 is less than 25% of its roofline performance while a NVIDIA A100 can achieve 50% of its roofline performance.). Also, some key information is not available. For example, we cannot find the packet format for TPU-TPU communication and have to use the NVLink packet format instead.

• Non-ideal hardware. LLMCompass assumes a fixed frequency, but when testing real-world hardware, we have no control over the frequency of the datacenter GPU or TPU nodes. LLMCompass also assumes bandwidth can be utilized at full rate, but in reality there may be some other overhead (e.g., error correction code).

III-D Area and Cost Model

As chip designers increase die area to improve single chip performance, fewer chips fit per wafer and may also risk decreased yield, leading to increased costs. LLMCompass incorporates area and cost models to allow designers to reason about these performance-area trade-offs. These models use the provided hardware description with estimated transistor counts and/or die areas from known components to find the total device die area - our methodology is explained as follows.

Within each core’s lanes, we estimate the vector units’ and systolic arrays’ transistor counts from open-source designs, tape-outs, and generators [75, 36, 18]. We estimate each lane’s register file’s area overhead using an empirical area model [54]. For the local buffer shared amongst lanes in each core as well as the global buffer shared amongst cores, we model them as SRAM caches and derive their areas using CACTI [39] and scale results down to a 7nm process. For memory and device-device interconnect, we estimate PHY and controller area based on annotated A100 and MI210 die photos [50, 60]. In our calculations, the controller area scales based on the process node, but the PHY area remains fixed as they do not scale well due to internal analog devices.

TABLE II: A Sample of Area Model Parameters (7nm)

Parameter	Transistor Count	7nm Area ($\mu{}m^{2}$)
64 Bit Floating Point Unit	685300	7116
32 Bit Int ALU	177000	1838
Per Lane Overhead	996200	10344
Per Core Overhead	44300000	460000
1024 Bit HBM2e Control	552743000	5740000
1024 Bit HBM2e PHY	-	10450000

(a) Die Area Breakdown of NVIDIA GA100 and AMD Aldebaran.

(b) Core Area Breakdown (Stream Multiprocessor for NVIDIA GPUs and Compute Unit for AMD GPUs).

Figure 6: Area Model Validations.

We account for extra per lane overheads (e.g., control signals) by calculating the core area using our model and taking the difference from the expected die areas taken from annotated photos. We then divide the overhead per lane, per scheduler width (32 in A100s, 16 in MI210). Similarly, we account for extra per core overheads (e.g., core-to-core crossbars) by calculating the expected die area with our model and splitting the area between the cores. These per-lane and per-core overhead estimates are averaged between AMD and NVIDIA chips.

To estimate cost, LLMCompass uses supply chain modeling [42] for wafer costs to calculate per-die costs. These per-die costs do not incorporate any IP, masks, or packaging costs. For memory costs, we use average DRAM spot prices for DDR [65] and consumer estimates for HBM2e [33].

Table II shows a sample of the transistor counts and corresponding 7nm die areas of the parameters used in the area model. Using their respective architecture white papers, we model GA100 [46] (the die used in NVIDIA A100) and Aldebaran [2] (the die used in AMD MI210) dies to estimate their total die areas, shown in Fig. 5(a). For the accounted-for components, LLMCompass’ area model estimates for GA100 and Aldebaran dies achieve a 5.1% and 8.1% error respectively. We attribute these differences to the core’s microarchitecture and core-to-core communication overheads which are proprietary and difficult to estimate. Our model also allows users to break down a single core’s area into its individual components, shown in Fig. 5(b).

IV Architectural Implications

With LLMCompass, we are able to conduct a design space exploration and shed light on how to design efficient hardware systems for LLM inference. In this section, we use LLMCompass to study how different compute system configurations, memory bandwidth, and buffer sizes affect LLM inference performance and draw architectural implications. These insights inspire us to propose new designs as in Section V.

IV-A Experimental Setup

For all the unmentioned specifications, we use the specifications of an NVIDIA A100 (as in Table I) and 4-way tensor parallelism. Prefill latency is measured by running one GPT-3 layer with batch size 8 (a balancing point between latency and throughput) and input sequence length 2048 (a medium-long sequence for GPT-3). Decoding latency is measured as the latency of generating the 1024th output token when running one GPT-3 layer with batch size 8 and input sequence length 2048. We use FP16 for all the operators.

IV-B Compute System

We test five different compute system designs as shown in Table III. From A to E, we increase each core’s systolic array, vector unit, and local buffer capacities. B represents a full GA100. We keep B, C, D, and E to have the same total compute capability and total buffer size to compare the design choice of fewer big cores or more tiny cores. Configuration A only has a quarter of the compute capability compared to others. All the designs have the same amount of total buffer size and register file size scales with vector width.

Figure 7 shows prefill and decoding latencies for these designs. Compared to the GA100, design A has 3.25x higher prefill latency but is only 0.1% slower at decoding and uses only 57.8% of the area. Design E with the largest cores see prefill and decoding latency increase by 12.4% and 30.8% respectively, but can reduce die area up to 7.7%.

Analysis: For the prefill stage, B is much faster than A because prefill is compute-bound. As per core systolic arrays and vector units scale, the tile size needs to increase to fully utilize larger computing units. Bigger tiles can cause more padding as the problem size needs to be quantized to the tile size and hardware size. Although large systolic arrays and vector units can be more area-efficient, they are harder to schedule and fully utilize.

Since decoding is IO-bound, increasing compute capability barely helps, which explains why A and B have similar performance. As the matrix multiplications during decoding are narrow (e.g. $16\times{}12288$), it is even harder to fully utilize larger systolic arrays and performance degrades.

TABLE III: Five Compute System Designs.

Specifications	A	B	C	D	E
Core count	128	128	128	32	8
Lane count	4	4	1	1	1
Vector width	8	32	128	512	2048
Systolic array	$8\times 8$	$16\times 16$	$32\times 32$	$64\times 64$	$128\times 128$
Local buffer (KB)	192	192	192	768	3072

(a) Prefill Latency per GPT-3 Layer.

(b) Decoding Latency per GPT-3 Layer per Output Token.

Figure 7: Impact of Compute System Design on Performance.

IV-C Main Memory

As main memory capacity is considered more of a constraint (enough capacity is required to hold the parameters and KV cache), we will focus on the impact of main memory bandwidth. Figure 8 details the performance results for sweeping memory bandwidth from 400 to 3200 GB/s. For prefill, increasing memory bandwidth from 800GB/s to 2000GB/s reduces latency by 14.3%, and further increasing to 3200GB/s has a marginal performance gain of 3.5%. For decoding, increasing from 800GB/s to 2000GB/s has a speedup of 1.88x, and further increasing to 3200GB/s brings another 26% gain.

(a) Prefill Latency per GPT-3 Layer.

(b) Decoding Latency per GPT-3 Layer per Output Token.

Figure 8: Impact of Memory Bandwidth on Performance.

Analysis: In the prefill stage, Matmuls are significantly faster when increasing memory bandwidth from 400GB/s to 800GB/s. Further increasing bandwidth does not significantly affect Matmul performance as it becomes compute-bound. For IO-bound GELU, LayerNorm, and Softmax, larger memory bandwidth realizes significant speedup.

In the decoding stage, Matmuls are significantly faster with increased memory bandwidth, mainly because they are narrow (turn into a vector-matrix multiplication at batch size 1) and IO-bound. In this stage, GELU, LayerNorm, and Softmax have a small input size. They are dominated by kernel launch overhead and barely affected by memory bandwidth.

IV-D Local and Global Buffer

Local Buffer. We fix the hardware specifications to an NVIDIA A100 (as in Table I) and sweep local buffer size. The results are shown in Figure 9. For prefill, increasing the local buffer size from 64KB to 192KB improves the performance by 18.0% while increasing the area by 5.8%. Further increasing to 1024KB has a negligible performance gain of only 0.2% at the cost of 28.8% bigger area. For the decoding stage, increasing the local buffer size from 64KB to 1024KB only increases the performance by 0.5%.

(a) Prefill Latency per GPT-3 Layer.

(b) Decoding Latency per GPT-3 Layer per Output Token.

Figure 9: Impact of Local Buffer Size on Performance.

Analysis: The reduced prefill latency with larger local buffers is mainly because of reduced matrix multiplication latencies. A larger local buffer enables larger matrix tiles and therefore higher systolic array utilization rate. A local buffer size of 192KB is just enough for matrix multiplication of $128\times 128\times 128$ at FP16 with double buffering technique. It can fully utilize the $16\times 16$ systolic arrays, shedding some insight on the NVIDIA A100’s design choices. Increasing local buffer size when the systolic array is already fully utilized leads to marginal performance gains. For decoding stage, increasing local buffer size does not help because it’s IO-bound.

Global Buffer. The performance trends for global buffer size are similar to Figure 9. Increasing the global buffer size from 10MB to 40MB speeds up prefill by 11.8% while increasing area by 9.6%. Further increasing to 80MB only brings a performance gain of 0.01% at the cost of 11.7% bigger area. For decoding, increasing global buffer size from 10MB to 80MB has a performance gain of only 0.7%.

Analysis: Larger global buffers enable larger matrix tiles, increasing systolic array utilization and data reuse at the global buffer level. Similarly, increasing global buffer size has diminishing returns once the systolic arrays are saturated. The decoding stage is not bounded by computation so it barely benefits from the larger global buffer.

V Efficient Hardware Design with LLMCompass

Ideally, efficient hardware design will optimize for both performance and cost. This section draws from the insights in Section IV and proposes two efficient hardware designs: a latency-oriented design and a throughput-oriented design. Both of these designs aim to reduce hardware costs while maintaining or improving performance. The key specifications are shown in Table IV. All the other specifications (e.g., frequency, register file size, device-device interconnect, kernel launch overhead, and framework overhead etc.) are the same as an NVIDIA GA100 for fair comparison.

TABLE IV: Comparison with NVIDIA GA100

Specifications	Latency	GA100	Throughput
	Design	(Full)	Design
Core count	64	128	64
Lane count	4	4	4
Vector width	32	32	32
Systolic array	$16\times 16$	$16\times 16$	$\mathbf{32\times 32}$
Local buffer (KB)	192	192	768
Global buffer (MB)	24	48	48
Global buffer (bytes/clk)	2560	5120	5120
Memory bandwidth (TB/s)	2	2	1
Memory capacity (GB)	80	80	512
Memory protocol	HBM2E	HBM2E	PCIE 5.0/CXL
Die area (TSMC 7nm, $mm^{2}$)	478	826	787
Normalized performance	0.95	1	1.41
Estimated die cost	$80	$151	$142
Estimated memory cost	$560	$560	$154
Estimated total cost	$640	$711	$296
Normalized performance/cost	1.06	1	3.41

V-A Latency-Oriented Design

LLM inference latency means the total time between receiving the request and generating the last token. It is a critical metric for interactive use cases like chatbots. It is composed of prefill latency, the time to process the input sequence, and decoding latency, the time to generate the output sequence in an auto-regression way. Inference latency is usually dominated by decoding unless the input sequence is much longer than the output sequence. Decoding is IO-intensive and is mostly bounded by reading model parameters and KV cache.

Observation: As latency is mostly IO-bound, memory bandwidth is the key to reducing latency, making HBM the best choice. However, due to the capacity limit of HBM, the batch size cannot be too large: the size of the KV cache and intermediate values is proportional to batch size. Therefore, the massive compute capability is not fully utilized.

Proposal: We propose an efficient latency-oriented design by pruning half of the compute capability while using the same memory system as a GA100, as shown in the left of Table IV.

Results: Compared to an NVIDIA GA100, the die area is reduced by 42.1% while keeping 95.3% of the performance on average. The results are shown in Figure 5.

Figure 10: End-to-End Performance of Latency-Oriented Design Normalized to GA100. Performance metric: inverse of latency (higher is better). Settings: batch size16, 4-way tensor parallelism, running 48 GPT-3 layers (half of GPT-3).

Discussion: Due to the IO-bound decoding stage, the over-provisioned GA100 is not able to realize significantly improved inference performance compared to our latency-oriented design. As shown in Figure 11, our pruned design achieves identical decoding performance as a GA100. The GA100 is an enormous die and is susceptible to yield issues - A100 dies are already binned to have 108 functioning SMs out of 128. Our latency-oriented design shows that even with half the cores and SRAM disabled, the device can still achieve similar performance. This may motivate designers to salvage previously deemed faulty chips and manufacture them into separate products focused on LLM inference.

Figure 11: Comparison of Decoding Latency per GPT-3 Layer.

Pruning the compute capability only hurts the compute-bound prefill performance. As prefill is more dominant at long input sequence and short output sequence, the performance degradation will be more visible under these cases, which explains why we only achieve 80% of the GA100 performance at input length 2048 and output length 256. With a smaller input length and larger output length, our pruned latency-aware design can achieve 99% the performance as GA100.

V-B Throughput-Oriented Design

For background use cases such as form processing or data wrangling, throughput can be more important than latency. There are generally two ways to improve throughput:

• Decrease latency - As latency is mostly IO-bound by reading parameters and KV cache, the best way to improve latency is to further improve memory bandwidth. As HBM is already expensive, this may not be easily achieved without increasing cost.

• Increase batch size - Generally, larger batch sizes are more efficient for throughput because the parameters are only read once for the whole batch. Larger batch sizes can also improve the hardware utilization rate. The downside is that a larger batch size consumes more compute power and increases KV cache accesses.

Observation: Increasing batch size is a more efficient way to improve throughput compared to decrease latency, which requires expensive high-end HBMs or even SRAMs. With a larger batch size, more memory capacity is needed to hold the larger KV cache and intermediate values.

Proposal: We propose a throughput-oriented design as shown in the right of Table IV. To hold larger batches, we use 512GB of DRAM powered by 256 PCIe 5.0 channels with an aggregated memory bandwidth of 1TB/s. (According to our area model, an 800$mm^{2}$ die’s perimeter is able to fit around 400 PCIe 5.0 channels.) Considering the high cost and limited capacity of HBMs, this design is more cost-effective. With larger batch sizes comes a greater need for compute capability, so we quadruple the systolic arrays and the local buffer. We halve the core count and vector unit to maintain a similar die area as GA100.

(a) Throughput of Throughput-Oriented Design (Tokens/s).

(b) Normalized to a 8-GA100 GPU Node.

Figure 12: End-to-End Performance of Throughput-Oriented Design. Performance metric: throughput. Settings: largest batch size within memory capacity, 8-way pipeline parallelism where each device runs 12 GPT-3 layers (1/8 of GPT-3).

Results: Compared to an NVIDIA GA100, the die area is slightly smaller and the throughput is improved by 1.42x on average. The results are shown in Figure 12. By replacing HBMs with traditional DRAMs, the cost is reduced by 58.3%, making a total of 3.41x gain in performance/cost.

Discussion: Our design has 6.4x the memory capacity of a GA100, which allows more than 12x bigger batch size after subtracting the fixed space occupied by model parameters. Ideally, with half the bandwidth of a GA100, this configuration can achieve more than 6x improvement in throughput. However, batching only reduces model parameter accesses but not KV cache reads. With a much larger batch, KV cache accesses become the new bottleneck, which diminishes the benefits of batching. As input length and output length increase, throughput decreases due to longer KV cache reads, as shown in Figure 11(a).

From a latency perspective, this throughput-oriented design may not be promising: the latency is 9.21x worse than GA100 on average. While model parameters are only read once for each batch, a larger batch size means more KV cache and intermediate values to read. In LLM inference, there is no free lunch between latency and throughput.

VI Related Work

VI-A Evaluating Large-scale Hardware Design

Evaluating the various characteristics of a hardware design, including performance, area, and cost, is extremely useful for hardware designers. To this end, the options are as follows:

Roofline Model Analysis [72]. Roofline models are analytical, fast to evaluate, and can be applied to various architectures for performance comparison. However, they can be overly optimistic relative to actual hardware capabilities.

Cycle-level Simulation [5, 6, 23, 61, 19, 31, 21, 66, 30, 48, 57, 56]. With a typical simulation rate of less than 100K instructions per second, cycle-level simulators become infeasible for evaluating LLM scale workloads. As these simulators are often designed for specific architectures, it is hard to describe a hardware design very different from its design purpose (e.g., it’s almost impossible to use GPGPU-sim [5] to evaluate a TPU-like design because it relies on the GPU ISA). These simulators often require the user to provide the program for evaluation. If the software program is not optimized, it may lead to unfair comparisons.

FPGA Emulation. Another way is to implement the design in RTL code and emulate it on FPGAs. The RTL code can be either handwritten or generated by accelerator generators [68, 45, 63, 18]. Although the emulation is fast, the synthesis process may take a long time for a large design capable of running large language models, and we may need multiple FPGAs to hold the design. Additionally, users need to rewrite the RTL code and redo the synthesis to evaluate a new design. For each new design, users are responsible for mapping their workloads efficiently to fully utilize the hardware.

Comparison. As shown in Table V, LLMCompass is more accurate than roofline model analysis, faster and more versatile than cycle-level simulators, and less engineering-intensive than FPGA emulation. For evaluating large-scale hardware design in the era of LLMs, we believe LLMCompass is helpful for the initial design stage to determine high-level hardware characteristics (e.g., number of cores, memory bandwidth, etc.). While this work describes LLMCompass in the context of large Transformer models, it can also be applied to other dense neural network models.

LLMCompass can complement FPGA emulation. Designers can perform initial design space exploration before incurring the heavy costs associated with FPGA emulation and the necessary RTL implementation of the proposed design.

VI-B Accelerator Design Space Exploration

Since the era of CNN, various works have focused on exploring optimal hardware designs as well as mapping [48, 13, 14, 35, 68, 74, 25, 34, 55, 76]. LLMCompass is different from these works in design considerations and emphasis: ① Mainly targeting Convolutional Neural Networks (CNNs), these works focus on loop parallelization, loop order, and data flows (e.g., weight stationary or output stationary), which are not the primary design considerations in Transformer-based LLMs. LLMCompass is more tailored for matrix multiplication tiling and scheduling as well as other Transformer operators such as LayerNorm. ② LLMCompass is designed for GPU-scale designs, which are much larger than CNN accelerators like Eyeriss [9]. LLM workloads are also significantly larger than CNN workloads.

LLMCompass can also complement design space explorations. Implemented as a Python library, LLMCompass can be seamlessly integrated into design space exploration frameworks such as FAST [76]. FAST uses an internal TPU performance simulator, limiting its broader utility. Fast and accurate, we believe the fully open-source LLMCompass can democratize hardware design space exploration research.

TABLE V: Comparison of Hardware Evaluation Methods

Methods	Fast	Accurate	Architecturally	Performance	Cost
			Descriptive◆	Optimal❖	Aware
Roofline	✓	✗	✓	✓	✗
Cycle-level	✗	✓	✗	✼	✗
FPGA	✼	✓	✼	✼	✓
LLMCompass	✓	✓	✓	✓	✓

◆: The ability to describe different hardware designs.

❖: Find the optimal mapping to fully demonstrate hardware capability.

VI-C Accelerating LLM Inference

Many Transformer accelerators have been proposed [62, 70, 69, 24], mainly focusing on accelerating the Transformer with hardware-software co-design such as pruning or approximate-computing. Whether these techniques are effective for the largest of models remains to be seen. Additionally, the major challenge of LLMs today comes from the massive scale of the models, which is the main scope of this paper.

Many efforts have also been made to accelerate LLM inference at the software domain [4, 52, 12, 11, 58]. LLMCompass is compatible with these optimization techniques by modeling their compute and memory access patterns. We don’t discuss techniques like FlashAttention [12] because they are orthogonal to the focus of this paper: They focus on the software domain and are usually implemented on a specific hardware platform such as NVIDIA GPUs.

VII Conclusion

This work introduces LLMCompass, a fast, accurate, and architecturally descriptive hardware evaluation framework for LLM inference workloads. LLMCompass’ hardware description template, mapper, and architectural simulator allow hardware designers to evaluate large-scale chip designs for LLMs, which are infeasible for cycle-level simulators. The incorporated area and cost models can also help designers reason about performance-cost trade-offs. With the aid of LLMCompass, we draw implications on how hardware designs affect LLM inference. Based on these findings, we propose a latency-oriented design and a throughput-oriented design that achieve 1.06x and 3.41x performance per cost improvements respectively, compared to NVIDIA GA100. We plan to extend LLMCompass to support more machine learning workloads as well as LLM fine-tuning in the future.