Shesha^††thanks: Shesha or Sheshanaga is a serpentine demigod in Hindu mythology with multiple heads and serves as the celestial bed of Lord Vishnu.  : Multi-head Microarchitectural Leakage Discovery
in new-generation Intel Processors

High-performance CPUs, such as Intel Core and Xeon processors, utilize multiple cores, high-speed memory units, and support parallel processing with efficient isolation and security measures. Memory isolation is achieved through individual per-process virtual address spaces. These virtual addresses are translated to page table entries (PTEs) containing data location, access control, and status bits for access control. These modern CPUs employ Simultaneous Multithreading (SMT) that allows multiple threads to execute on the same core, providing architectural isolation. Intel CPUs support two threads to run simultaneously per physical core. Another important feature called speculative execution enables the CPU core to execute instructions in the pipeline even when there are dependencies on prior unresolved operations. In case of incorrect predictions, the CPU corrects itself by re-executing instructions. While architecturally invisible, there may be observable side effects due to microarchitectural state changes. In this context, such an instruction is referred to as a transient instruction [10, 31, 35]. Transient execution attacks capitalize on these transient instructions to leak sensitive data across security boundaries. Single Instruction, Multiple Data (SIMD) enables data-level parallelism, performing the same operation on multiple sets of data. AVX2 and AVX-512 are key SIMD extensions for x86 architecture. Finally, the CPU includes a shared last-level cache (LLC) across execution cores and an interconnect bus linking the LLC, cores, and DRAM. Additionally, each core has its own private caches - L1 and L2 caches. Temporal buffers are employed to optimize micro-operations, such as the fill buffer for fetching cache line data bits and the store buffer for holding data before committing it to the cache.

A Particle Swarm Optimizer [29], abbreviated PSO, is a generic evolutionary method suited to solving optimization problems. Unlike Genetic Algorithms that rely upon evolution of genetic information, PSOs work on the delicate interplay of individual and swarm behaviours in nature (like ant swarm, bee swarm etc). Abstractly, a PSO balances two opposing forces- exploration and exploitation in order to optimize an objective function. Here, exploration abstracts the random choices that encourage the PSO for a random walk across the search space in hopes of finding newer paths to global optimums. On the other hand, exploitation abstracts the greedy choices, encouraging the PSO to focus on a direct path to currently known optimums. As evident, too much of exploration is no better than a random search, while too much of exploitation risks getting stuck in local optimums. A PSO balances the two forces by switching back-and-forth between particle and swarm behaviour. A particle performs random searches across the search space, and if optimal paths are discovered, broadcasts that information to other particles, such that the entire swarm (where swarm can be interpreted as the set of all particles) starts to use the information from that one particle. Consequently, the particle behaviour abstracts exploration, while the swarm behaviour abstracts exploitation.

Syntactically, a PSO is characterized by a fitness function $f:\mathcal{R}^{n}\rightarrow\mathcal{R}$, mapping an $n$-dimensional vector (termed particle position, or $\mathbf{x}$) to a real-value (termed fitness) which expresses how close the particle position is wrt. global optimum. Initially, the PSO samples a random population of $N$ particles, denoted by a set of position vectors $\mathbb{X}=\{\mathbf{x}_{1},\mathbf{x}_{2},\mathbf{x}_{3},...,\mathbf{x}_{N}\}$. For an iteration $i$ of the PSO algorithm and particle $j:\{1\leq j\leq N\}$, the terms velocity and position are defined as:

where $\alpha$ is the inertial weight, $\beta$ is the cognitive acceleration coefficient, $\gamma$ is the social acceleration coefficient, $\mathbf{p}_{j}$ is the best position (where the fitness function is maximally optimal, up to iteration $i$) of particle $j$, and $\mathbf{p}_{g}$ is the best position for the entire swarm. Abstractly, in every iteration $i+1$, each particle $j$ attempts to update its position $\mathbf{x}_{j}(i+1)$ based on the position in the previous iteration $\mathbf{x}_{j}(i)$ and a computed velocity $\mathbf{v}_{j}(i+1)$. Now, this velocity vector is computed by combining three pieces of information:

1. 1.

   Velocity in the previous iteration: denoted by $\mathbf{v}_{j}(i)$

2. 2.

   Exploration / Particle behaviour: Velocity computed by considering the best position explored by the particle so far (denoted by the vector $\mathbf{p}_{j}(i)$).

3. 3.

   Exploitation / Swarm behaviour: Velocity computed by considering the best position across the entire swarm (denoted by the vector $\mathbf{p}_{g}(i)$).

The parameters $\alpha$, $\beta$, and $\gamma\in[0,1]$ control the importance given to each of these three information points, and directly affect how a particle moves across the search space.

Since the objective of Shesha is to explore possible avenues of transient execution attacks, we assume Ring 3 sudo privileges. However, for attack demonstration, we do not require sudo privilege and only assume user-level privilege. We further assume x86-64-based target systems with latest microcode patches and secure (patched) operating system, with all state-of-the-art mitigations against transient execution attacks. We assume no publicly known exploitable vulnerability is present in the system. We specifically focus on Intel CPUs with newer generation microarchitecture, both on client and server platforms. For our experiments, we execute Shesha on Intel CPUs from four generations: Alder Lake(12th gen Core i7) and Comet Lake (11th gen Core i7) on the client side and Sapphire Rapids (4th gen Xeon) and Ice Lake (3rd gen Xeon) on the server side. All subsequent discussions are contextual to these aforementioned machines.

As discussed in Section 2.2, the objective of a PSO is to optimize a well-defined fitness function $f:\mathcal{R}^{n}\rightarrow\mathcal{R}$. Since our search space is instruction sequences enabling bad speculation over ISEs, our fitness function must be a combination of indicators that capture the existence or absence of bad speculation. For a code sequence $\mathcal{C}$, in our context, $f$ can be defined as $f:\mathcal{C}\rightarrow\{0,1\}$, or $f$ provides a binary decision whether bad speculation was observed for a given code sequence $\mathcal{C}$. We defer discussion on the generation of $\mathcal{C}$ to Section 3.4.1 and focus on the binary decision of bad speculation here.

First, we detail on when bad speculation happens in modern Intel processors. Intel’s Software Developer Manual [24] states bad speculation occurs from three major causes - branch misprediction, microcode assists, and machine clears. Branch misprediction related transient execution occurs due to the need to squash micro-operations issued from the mispredicted branch and re-issue the ones from the correct branch. Likewise, microcode assists are needed when the hardware needs to rely upon microcode for execution. In such a scenario, the speculatively issued micro-operations from the micro-operation decoder are flushed from the pipeline, and are replaced by the micro-operations from the microcode sequencer (which in turn fetches these micro-operations from microcode ROM). Finally, machine clears also cause transient execution for events like memory ordering violations, self-modifying code execution, and so on.

Given this context, to implement the fitness function $f:\mathcal{C}\rightarrow\{0,1\}$, we require an interface to capture when any of branch mis-prediction, microcode assist, or machine clear happens. All modern Intel systems expose a Performance Monitoring Unit [25] that logs different types of events occurring on the physical cores for analysis. Such performance counters are useful in understanding occurrences of bad speculation, given any code sequence $\mathcal{C}$. Table 1 summarizes the exact performance counters we used for Shesha. Therefore, given a code sequence $\mathcal{C}$, the objective of the PSO will be to maximize occurrences of bad speculation as reported by the respective performance counters.

We now discuss the nature of bad speculation events that we term as disjointedly fragmented sub-spaces which allows us to achieve faster convergence for Shesha and efficiently scale as the number of ISE instruction pool increases. As evident from Table 1, each of the performance counters pertains to bad speculation scenarios, which are disjoint from each other from a causal perspective. For instance, bad speculation due to memory disambiguation issues has completely different cause (incorrect speculation while resolving load/store addresses) than bad speculation due to self-modifying code execution (incorrect speculation in writing to instruction cache). Additionally, the disjoint classes of bad speculation also have effects on disjoint architectural and micro-architectural assets. For example, memory disambiguation issues will mainly cause speculation in data forwarding in load/store buffers. On the other hand, self-modifying code execution issues will affect the instruction cache. Additionally, AVX-SSE transition issues shall affect the SIMD registers and temporal buffers. Therefore, we make the following observation:

Such a definition of equivalence classes (over set of instructions) ensures that the search space for bad speculation is split into disjointedly fragmented sub-spaces, such that the portion of the PSO’s swarm working on one equivalence class need not rely upon the entirety of the PSO swarm for exploitation (cf. Section 2.2). For example, without loss of generality, consider a portion of the PSO swarm working with machine clears related to self-modifying code. Then, the velocity vector computations related to swarm behaviour (i.e. involving $\mathbf{p}_{g}(i)$, cf. Section 2.2) need not consider the portion of the swarm working with bad speculation due to branch mispredictions, simply because code sequences triggering branch misprediction (for instance, heavy use of loops to train the branch predictor) have no direct bearing with code sequences triggering self-modifying code (for instance, use of store operations in address ranges belonging to .text section of the process binary, thereby requiring write operations on the instruction cache). Based on this description of equivalence classes, we now detail the construction of Shesha in the next subsection.

With the idea of equivalence classes established, we now detail Shesha - an automatic analysis framework for exploring the disjointedly fragmented equivalence classes of bad speculation in ISEs. Shesha is divided mainly into three phases: Swarm initialization, Cognitive phase, and Mixed phase. In phase , a population pool of N particles is created where each particle represents a randomly sampled instruction sequence from the instruction pools of different ISEs. Then, in phase , the particles are evolved to settle into sub-swarms (i.e. one swarm for one equivalence class). In this phase, since the sub-swarms are still being created, there is no exploitative behaviour (i.e. all particles evolve independently of each other). This design decision is intentional, following our discussion on equivalence classes in Section 3.3. Until the sub-swarms are ready, Shesha does not have reliable global best solutions to use for updating particle velocity (cf. Section 2.2 for PSO semantics). Finally, upon end of phase , the sub-swarms would be ready for each of the equivalence classes, which is when phase kicks in and uses both local best and global best solutions to update particle velocity. We now explain these phases in detail.

SIMD and Vector instruction extensions like AVX, AVX2, AVX512EVEX, AVX512VEX, SSE, SSE2, SSE3, SSE4, SSE4a, and SSSE3 are central to several performance gains experienced in modern Intel processors. These ISEs allow greater than $64$ bit registers holding more than one data point, to be operated in parallel by the same instruction (hence the name SIMD type ISEs). To do so, modern IA-64 architectures define a register bank with sixteen $512$-bit wide registers. Interestingly, ISEs like AVX2/SSE allow differing representations of the same hardware register in two dimensions: ① width $w$ of the register, and ② the SIMD representation of the data contained in that width $w$. To elaborate on dimension ①, the same hardware registers can be interpreted as $512$-bit wide registers zmm0, zmm1, zmm2, ..., zmm15 (by ISEs like AVX512EVEX, AVX512VEX), as $256$-bit wide registers ymm0, ymm1, ymm2, ..., ymm15 (by ISEs like AVX2), and as $128$-bit wide regisers xmm0, xmm1, xmm2, ..., xmm15. Likewise, regarding dimension ②, the same data in any zmm/ymm/xmm register can be interpreted as differently packed integer/floating-point value. This discussion raises a concerning question: Given a processor allowing executions of ISEs requiring different representations of the same hardware, how are transitions between such ISEs handled? According to the Intel Software Developer Manual [24], the exact implementation to handle such transitions depends upon the exact micro-architecture under question. We defer the details of these implementations and our reverse-engineering approach to Appendix B. We make the following observation as the root cause of bad speculation (as in Table 2) encountered in modern Intel processors when transitioning from SIMD to Vector instructions (and vice-versa).

Modern ISEs usually work upon a register bank of $16$ registers of width $512$ bits, and the same data on these registers can be interpreted in different ways. While it is straightforward to switch between integer representations, switching between single-precision and double-precision representations is an involved process. Through the course of our testing campaign, Shesha uncovered a previously unknown source of transient execution paths: non-explicit switching between single and double precision floating-point numbers in a Read-after-Write (RAW) data dependency. Concretely, a sequence of two instructions linked by a RAW dependency causes novel transient execution paths on Intel CPUs if they require contrasting representations (single precision vs double precision) of the same floating-point number. Interestingly, three independent, completely novel transient execution paths are uncovered by Shesha as a result of single/double precision intermixing with RAW data dependency- hardware assist (HW), self-modifying code (SMC) based machine clear, and memory ordering (MO) violation based machine clear. For all subsequent discussions, we consider the following abstract instruction sequence (SP: single precision instruction, DP: double precision instruction):

1. 1.

   SP xmm0, xmm1, xmm2 followed by DP xmm3, xmm4, xmm0     OR

2. 2.

   DP xmm0, xmm1, xmm2 followed by SP xmm3, xmm4, xmm0

Precisely, for the discussed transient paths to kick in, the requirement is a set of at least two instructions with a Read-after-Write (RAW) dependency connecting them (using the xmm0 register in the above examples), without occurrence of an explicit conversion instruction (like CVTPD2PS or CVTPS2PD) in between. We note here that Write-after-Write (WAW) or Write-after-Read (WAR) dependencies do not trigger any of the following transient paths, simply because there is no need for any implicit precision conversion as the second instruction is writing to the dependent register and not reading from it. Writes from a single-precision instruction to a register holding a double-precision value are architecturally simply overwriting the registers, and thus incur no transient executions. We defer the detailed discussion on each of these transient execution paths and our reverse-engineering approach to Appendix C. We make the following observation as the root cause of all the bad speculation events when transitioning between single and double precision.

Transient execution paths guarantee flushing of the in-flight micro-operations from the pipeline and re-steer the execution on the correct path. As such, the width of the transient window for any given transient execution path determines the number of micro-operations an attacker can issue on the transient path before the rollback of the instructions. Practically, a larger transient window allows for executing complex attack vectors in the transient path. Architecturally, no effect of any transient micro-operation is visible. Therefore, we need to rely upon micro-architectural effects to measure the transient window. In our case, we compute the transient window by counting the number of transient loads executed before the pipeline rollback. When no bad speculation (discovered by Shesha) occurs, we observe no activity in the transient path, implying absence of misprediction-based transient execution. Our results are summarized in Table 2. In comparison to the machine clears presented in [42], we conclude that the SIMD-Vector transition-based SMC exhibits a larger transient window than all other variants of transient executions through machine clears or assists (both uncovered here as well as in [42]). This observation directly ties in with the number of repetitions (for warming up the execution pipeline) an attacker needs to do on the transient path before the actual attack can be mounted (cf. Section 7.1). As evident, due to the largest transient window with minimum number of repetitions, SIMD-Vector transition-based SMC provides a very practical transient execution path (with effectively no repetitions required), thereby becoming an actual choice for an adversary to mount in-the-wild attacks.

We demonstrate a covert channel using contention in the affected hardware assets. Two processes are pinned to two logical cores of the same physical core. Hence, we assume Simultaneous Multi-Threading (SMT) or HyperThreading to be enabled for the covert channel construction to work since the affected assets identified (in Table 2) are shared in an SMT setting. The construction of the covert channel relies upon two actors: a sender and a receiver, executing on the same physical core (cf. Algo. 3 and Algo. 4 in Appendix). The receiver times the execution of a code sequence $\mathcal{C}$ ($\mathcal{C}$ has no possibility of any transient execution). In contrast, depending on whether the sender has to send bit $1$ or $0$, it performs an additional execution involving transient execution. The presence or absence of transient execution causes contention in the concerned hardware assets, leading to a perceivable timing difference on the receiver side. Such a covert channel instantiated with SIMD-Vector transition transient executions has a bandwidth of $2$ KB/s with an accuracy of $87\%$.

In addition to the Leak attack type summarized in Table 2 and detailed in Section 7.1, the Floating Point operations relying upon an assist to carry out the denormalization operation forward incorrect data to subsequent transient instructions, before denormalization finishes. This scenario is a floating point flavor of classic load value injection (LVI) in transient executions [42]. However, unlike Section 7.1, the exact transient value injected into the transient operations is not architecturally visible. Hence, the adversary needs to mount LVI in two phases: ① offline phase where the adversary fuzzes the transient operation using different inputs and constructs the transient output, and ② online phase where the adversary uses the specially crafted inputs from ① that give a transient output of adversarial interest. On comparing the architectural result of the computation against the constructed LVI output, we observed both to be different, implying a successful transient value injection attack.

According to IEEE-754 standard, any floating point representation of a number consists of three parts: ① sign bit, ② biased exponent $e$, and ③ mantissa $m$. The standard also defines denormalization: a choice to trade precision for wider range of floating point numbers that can be represented. Concretely, if $e=0$ and $m\neq 0$, denormalization can allow for a gradual underflow by appending enough leading zeroes to the mantissa until the minimal exponent is achieved.

As stated in [42], denormalization requires a Floating Point assist (denoted by ASSISTS.FP in Table 1), which kicks in micro-operations to gradually underflow a floating point number close to $0$. This causes a flush of the current in-flight micro-operations from the execution pipeline, thereby creating a transient window where non-denormalized floating-point values are forwarded to subsequent instructions speculatively, resulting in a transient window of executions on incorrect data, while the floating-point assist is denormalizing the concerned value. In [42], this transient window is used to construct a flavor of load-value injection exploits on the SSE2 instruction extension. Through Shesha however, we uncover existence of this transient execution path in other ISEs as well ³³3Shesha also uncovers previously unreported combinations of AES and SSE instruction sequences (like AESDECLAST and MULSS sharing a Read-after-Write register dependency) to also incur Floating-Point assists., most notably in Fused-Multiply-Add (FMA) instructions like VFMADD132SD, VFMADD132PD, or exponentiation instructions like VSCALEFSS.

FMA instructions have been traditionally introduced to accelerate arithmetic capabilities of Intel processors. Such instruction extensions are capable of performing fused arithmetic operations in one instruction execution. The classic FMA family of instructions can support operations like: ① fused multiply-add, ② fused multiply-subtract, ③ fused multiply add/subtract interleave, ④ signed-reversed multiply on fused multiply-add and multiply-subtract. Moreover, several new “FMA-like” instruction extensions have been introduced to similar effect. For instance, the Integer-FMA extension introduces two instructions that perform FMA-like operations, but on packed integers. Likewise, the 4FMAPS extension allows packing up-to four vectors in one operand (as opposed to a single vector allowed in FMA/IFMA).

Intel processors provide a specialized execution unit for FMA (and its associated sibling) extensions as detailed by the patents [6, 15]. The architecture of the hardware for executing FMA instructions poses an interesting insight: specialized “Memory Access Units” (MAU) reside within the execution cluster of execution engine units dedicated to FMA instruction execution. Concretely, once the frontend unit performs the instruction fetch and instruction decode, it issues the relevant operations to the backend, which consists of several units like the register file, the register renaming unit, the scheduler unit, the retirement unit, the execution unit, and the memory access unit. We emphasize that both the instruction cache unit as well as the instruction translation look-aside buffer (TLB) unit are a part of the frontend, and not the backend. Moreover, the MAU interfaces with both the data cache unit as well as the data TLB unit (which sit outside the execution engine unit, as a separate memory unit) ⁴⁴4We omit a detailed architecture description of the FMA execution unit for want of conciseness. Interested readers can refer to Intel patents [6] and [15] for more details. It must also be mentioned that these architectural descriptions depict the overall idea, and the actual implementation in the processors might differ..

As highlighted in the previous subsection, the FMA execution unit houses MAU whereas the data-cache and dTLB are kept outside the execution unit, in a stand-alone memory unit. We hypothesize that MAUs are essentially meant for buffering optimizations. Recall that almost all FMA (and its associated sibling extensions) have instruction variants that can take memory operations. For example, consider VFMADD132PD zmm1, zmm2, zmm3, which multiples the first and third operands, and then adds the product to the second operand. The other flavor of this instruction is VFMADD132PD zmm1, zmm2, zmmword PTR [rcx] which fetches the third operand directly from memory. Since all FMA instructions (and its sibling extensions) have flavors supporting memory operands, we hypothesize the aforementioned “Memory Access Units” or MAUs are essentially buffers that optimize these memory reads.

From Section 7.4, we recall that Shesha has undiscovered the possibility of speculation in FMA execution. Such speculation, tied with the existence of buffers inside FMA execution engine, could leak to leakages. In this context, we test the following two hypotheses related to the transient behavior shown by the FMA instructions:

With speculative execution in FMA execution engine (cf. Section 7.4), does FMA “Memory Access Units” transiently forward data to speculatively executing FMA instructions?

With speculative execution in SSE-AVX execution engine, does FMA “Memory Access Units” transiently forward data to speculatively executing SSE-AVX instructions?

It must be mentioned that hypothesis draws from our yet another observation about the architecture of FMA execution unit [6, 15]. The register file is shared between SSE-AVX and FMA execution engines. Concretely, the same set of xmm/ymm/zmm registers used by SSE-AVX instructions are also used by FMA instructions. This leads us to investigate whether any cross-interaction exists between these two execution engines.

All experiments are performed on $11^{th}$ Gen Intel(R) Core(TM) i5-11500, microcode version 0x57, with support for FMA and IFMA extensions. We do not assume any privileges for the adversary, except for the userspace privilege to execute code.

We now discuss the broader impacts of our findings. While FMA is usually used to accelerate arithmetic, it is the IFMA instruction pair that has received unprecedented adoption in recent years. This is because of IFMA instructions’ capability to operate fused multiply and add operations on integers, allowing them to accelerate core operations in traditional cryptography (refer to Intel’s guidelines on the same [27]). According to Intel Optimization Manual [23], IFMA is introduced for big number multiplication for acceleration of RSA vectorized software and other Crypto algorithms (Public key) performance. However, as a direct consequence of our findings, such optimized implementations are vulnerable to leakages, which we detail next.