Making Your Program Oblivious: a Comparative

Study for Side-channel-safe Confidential Computing

A K M Mubashwir Alam Keke Chen
Computer Science Computer Science
Marquette University Marquette University
Milwaukee, WI, USA Milwaukee, WI, USA
[email protected] [email protected]
arXiv:2308.06442v1 [cs.CR] 12 Aug 2023

Abstract—Trusted Execution Environments (TEEs) are grad- While TEE performs much more efficiently than software-
ually adopted by major cloud providers, offering a practical based cryptographic approaches, recent studies have also
option of confidential computing for users who don’t fully trust identified several side-channel attacks [8], [11], [40], [44],
public clouds. TEEs use CPU-enabled hardware features to
eliminate direct breaches from compromised operating systems [47]. Although the TEE enclave cannot be directly breached,
or hypervisors. However, recent studies have shown that side- side channels are still there – The enclave interacts with
channel attacks are still effective on TEEs. An appealing solution untrusted memories and file systems, and the CPU cache is still
is to convert applications to be data oblivious to deter many side- shared among processes/virtual machines owned by different
channel attacks. While a few research prototypes on TEEs have users. Thus, attackers can utilize such controlled channel
adopted specific data oblivious operations, the general conversion
approaches have never been thoroughly compared against and attacks, e.g., manipulating page faults and page-table entries
tested on benchmark TEE applications. These limitations make it and exploiting the flaws of modern CPU’s micro-architecture
difficult for researchers and practitioners to choose and adopt a execution optimization. Powerful attacks like Foreshadow [7]
suitable data oblivious approach for their applications. To address and Load Value Injection [43] can combine memory/cache
these issues, we conduct a comprehensive analysis of several footprints and CPU speculative execution to extract the secrets
representative conversion approaches and implement benchmark
TEE applications with them. We also perform an extensive in TEE execution.
empirical study to provide insights into their performance and So far, countermeasures on side-channel attacks are limited
ease of use. to specific applications [2], [28] or firmware fixes at the
Index Terms—Access-Pattern, TEE, SGX, Obliviousness, Side- micro-architectural [43]. Among the candidate solutions, data-
Channel, Confidential Computing oblivious algorithms and applications appear attractive and
promising. Regular programs’ data flow and execution paths
I. I NTRODUCTION vary according to input data, i.e., a specific input value may
trigger different steps to execute. In contrast, data-oblivious
Confidential computing enables users to enjoy public clouds algorithms’ data flow and execution paths are invariant to the
without the need to trust cloud providers’ security infras- input. This data obliviousness property can potentially help
tructure. Researchers are actively developing cryptographic address many side-channel issues, as we will discuss in Section
approaches to secure processing in untrusted platforms, such as II-B.
Homomorphic Encryption [5] and Secure Multiparty Compu- Nevertheless, it is challenging for users to develop an
tation (SMC) [20], [25]. While recent cryptographic methods, oblivious solution for the following reasons. First, it’s unclear
e.g., hybrid protocols [25], [26], [39], are getting more effi- how complex to compose an oblivious data program manually.
cient, pure software-based solutions are still too expensive to Although recent TEE-related studies [12], [29], [36] have indi-
be practical for complex computational tasks or data-intensive cated some oblivious primitives that one can use to compose an
applications [35]. oblivious solution, the complexity and the efforts to develop
More recently, Trusted Execution Environment (TEE) [13] such a solution are unclear. Second, automated approaches
emerges as a more practical solution for confidential comput- can help convert regular programs to oblivious ones, but it’s
ing. TEE utilizes CPUs’ new hardware features to securely unclear how practical they are for TEE applications. Third,
isolate a user’s application from the cloud system. Therefore, the quality of automatically generated oblivious solutions is
even if an entire system, including the operating system or also a concern. Oblivious programs generally cost more than
hypervisor, is compromised, the adversary cannot access the their non-oblivious equivalent in terms of performance and
application. Major CPU manufacturers have implemented the memory. Low-quality conversion may also result in a higher
TEE concept in their recent CPUs, e.g., Intel SGX and AMD performance penalty. There is no systematic study to answer
SEV. Correspondingly, TEE-enabled servers are increasingly these questions for TEE-based applications.
available in major public clouds, e.g., Azure provides SGX- Contributions. We conduct a comprehensive analysis and
enabled servers, and Google adopts AMD SEV. empirical study to compare several data oblivious solutions
for TEE applications. The result will help researchers and is using certified TEE hardware and if the program running
practitioners understand the benefits and limitations of current in an enclave is from a digitally signed binary. During remote
solutions, possibly identify new research topics and assess the attestation, a secret key is generated by Diffie-Hellman key
strategies for adopting the side-channel-safe TEE solutions. exchange between the Enclave and the remote user to establish
Specifically, we first analyze whether and how data obliv- a secure channel for the follow-up communication between the
iousness can address side-channel attacks. Then, we sum- user and the Enclave.
marize four representative solutions: the manual composition Major cloud platforms have provided different types of
approach, the compiler approach, the circuit approach, and TEE-enabled servers. Intel SGX is one of the popular TEE
the application-framework approach and their characteristics implementations. Since 2015, SGX has been available in most
in terms of performance, ease of use, and maturity for appli- Intel CPUs. Unlike Intel SGX, the other two major TEEs,
cation. such as ARM TrustZone and AMD PSP, rely on secure
Finally, we develop an evaluation benchmark that includes hardware and a secure operating system. As AMD integrated
basic oblivious operations, compute-intensive tasks, and data- ARM TrustZone as an extension of CPU [17] and later
intensive tasks. Then, we apply different oblivious program renamed it Platform Security Processor (PSP), the underlying
conversion approaches to the benchmark and evaluate the technology of both systems remains similar. While all TEE
resulting oblivious programs’ performance and ease of use. implementations feature complete memory isolations from the
Our study reveals the strengths and weaknesses of different system components and remote attestation to establish trust,
oblivious solutions and provides guidelines for selecting suit- they still suffer from side-channel attacks.
able techniques under different scenarios.
B. Side Channel Attacks on TEEs
In the remaining sections, we will first present the back-
ground knowledge for our approach (Section II), then dive Since the advent of TEEs, many studies have explored the
in the details of how data oblivious solutions help to protect weaknesses of TEE side channels. While passive adversaries
side-channels (Section III), then discuss different oblivious can exploit some attacks [10], [34], [50] by only observing
approaches (Section IV), and, finally, perform the experimental interactions between TEEs and other system components, the
evalution (Section V) and conclusion (Section VI). assumption of TEEs enables more powerful attacks to be
performed, some of which can even retrieve plaintext infor-
II. P RELIMINARIES mation directly from the Enclave. Based on the attack strate-
This section presents the necessary preliminaries for under- gies, these attacks can be categorized as (i) memory/cache-
standing the paper. We will give the related background knowl- targeted attacks and (ii) microarchitecture-level attacks. In
edge before analyzing oblivious solutions. In the following, we memory/cache-targeted attacks, the attacker exploits the inter-
will introduce the concept of Trusted Execution Environments actions between TEEs and untrusted memory or applications
(TEE), the status of TEE development and deployment, and and observes enclave memory page loading and CPU cache
the effect of side-channel attacks in TEEs. usages. Microarchitecture-level attacks utilize modern CPU
features, such as CPU transient memory execution [7], to
A. Trusted Execution Environment retrieve fine-grained information from the low-level cache
Trusted Execution Environment (TEE) is a hardware-based lines. We will discuss more details in the next section.
solution for executing code in a secure environment where
III. DATA O BLIVIOUS S OLUTIONS FOR S IDE C HANNEL
powerful adversaries cannot access code or data within this
P ROTECTION
secure area. Using TEEs, a user can run their sensitive
computations in a TEE called Enclave, which uses a hardware- A. Threat Model
assisted mechanism to preserve the privacy and integrity of en- Users may run confidential computation tasks in an un-
clave memory. With TEEs, users can pass encrypted data into trusted cloud server, where the server’s OS or hypervisor can
the Enclave, decrypt it, compute with plaintext data, encrypt be compromised. The goal is to preserve data and program’s
the result, and return it to the untrusted cloud components. integrity and confidentiality while availability is out of con-
TEEs isolate private reserved memory for secure applications cern. A typical TEE, such as Intel SGX, provides a hardware-
from other system components, such as operating systems and protected memory area, i.e., the enclave [13], and guarantees
hypervisors. When the operating system or other system appli- the integrity of the data and computation running inside the
cations want to access the dedicated private memory, the CPU enclave. While adversaries cannot directly access the enclave,
restricts the access and redirects to some abort memory page. they can still glean information via side channels, such as
Therefore, TEE applications can perform plaintext calculations memory access patterns and CPU caches. However, cache-
without compromising privacy and security. based attacks target all CPUs (regardless of having TEEs or
However, without verifying the correctness of the cloud not) and thus need manufacturers’ micro-architecture level
hardware and the user binary, the remote user still cannot fixes. In contrast, the exposure of memory access patterns
trust the TEE. The Remote Attestation procedure establishes is inevitable as enclaves have to interact with the untrusted
the trust between the TEE hardware and the user. Using memory area. It’s also reasonable to assume that attackers
remote attestation, the user can verify if the cloud provider cannot access the cloud server physically, e.g., attaching a
 CPU //if (a < b) x = a else x = b
Trusted Cache
CPU CMOVL x, a
Core
Enclave CMOVGE x, b
A few studies [3], [30], [33] have used CMOV instruc-
tions to provide code-level obliviousness for branching
Encrypted
Side-Channel statements. Without specific conditional jumps, CMOV
Attacks
Data instructions move the source operand to the destination
Untrusted Memory when a conditional flag is set. However, regardless the
Untrusted Server flag is set or not, it reads the source operand. Therefore,
the access to the source operand cannot be used to infer
Fig. 1: TEE, side channels, and the threat model. whether the source is copied to the destination or not.
Ohrimenko et al. [30] also designed library functions
omove and ogreater to wrap up the CMOV instructions
device to the server or touching the motherboard, which for conveniently converting the branching statements. No-
excludes all attacks based on physical accesses. Figure 1 tably, a completely oblivious branching execution needs
illustrates the threat model. to run both branches and select the desired result with
the above method, which often leads to very high costs.
B. Data Obliviousness • Circuit. Circuits are considered a natural way to hide
Definition. The execution path and data flow of a (data) access patterns, as the circuit execution activates the
oblivious program do not change with different input data gates in a certain order regardless of the input values
and parameter settings. When all the steps of an algorithm or [19]. The branching statement is readily implemented
mechanism do not depend on input data, one cannot determine with a bitwise multiplexer. However, oblivious memory
the nature of the data by observing the steps of that algorithm. access imposes significant challenges. Many solutions
Thus, oblivious solutions can effectively protect from attacks implement linear scan so far [9], [32], [41], which incurs
depending on data-dependent access patterns. very high costs.
Oblivious Primitives. The goal of developing data obliv- • Oblivious algorithms. Task-specific oblivious algorithms
ious programs is to eliminate any data-dependent operations. are methods specifically designed to work with a specific
We list the primitives that data oblivious programs heavily task or a data structure. They work more efficiently
depend on. than solutions composed of general primitives such as
ORAM. For example, MergeSort can be converted to be
• Address-based Access. This operation includes array el-
oblivious by simply replacing every memory interaction
ement access or data block access. Exposing the position
of the merge phase with ORAM and unwinding data-
of accessed data is the fundamental access pattern. A
dependent loops with fixed iteration loops. However, this
naive solution is to iterate over the whole data structure to
direct conversion can be much more expensive than a
hide the actual accessed position. In contrast, Oblivious
specially designed oblivious sorting algorithm, such as
RAM (ORAM) [18] has been a well-accepted primitive
BitonicSort [4]. Similarly, frequently used data-intensive
for more efficiently hiding accessed addresses. It can ef-
operations, such as join and group by, can have more
fectively reduce the cost of oblivious access to O(log N )
efficient dedicated oblivious versions.
for a structure of N data items. ORAM has been used in
a few TEE-based solutions to hide access patterns. C. Can Data Obliviousness Address TEE Side-channel At-
• Data-dependent Branching. Most programs contain tacks?
data-dependent branching statements. Depending on the Against Memory Targeted Attacks. Memory-targeted at-
different inputs, a program execution may choose dif- tacks glean and utilize access patterns within the system
ferent paths, resulting in distinct access patterns. The memory. TEE applications store data in an encrypted form
following code snippet shows how an attacker can utilize outside the TEE. When encrypted messages are accessed from
the branching access pattern. memory, i.e., between TEE and untrusted memory and even
if (a >= b){ within TEE, an adversary who controls the operating system
// swap a and b, and can observe the data access patterns and possibly extract
// the page access can be observed. sensitive information by manipulating page-fault interrupts
}else{ [10], [40]. Page-table entries [8]. For distributed data-intensive
// no page access. applications, Ohrimenko et al. [34] also demonstrated how
} sensitive information, such as age group, birthplace, and
marital status, can be extracted from MapReduce programs
The common method uses the CPU’s conditional move by only observing the network flow and memory skew.
(CMOV) instructions to eliminate the branching state- These attacks all depend on the differential access patterns
ments. A simplified example is shown as follows: observed via the side channels. For example, an important
step in KMeans clustering is to find the nearest centroid and IV. M AKING YOUR P ROGRAM O BLIVIOUS
update the temporal cluster information for each training data. While the fundamentals of data-oblivious operations are
A straightforward way of accessing cluster information creates clear, developing a practical solution is challenging for several
data-dependent branching based on cluster ids. If each cluster reasons. First, it requires the developer to have basic knowl-
object resides in separate memory pages, an attacker can edge of every data-dependent part of their programs and the
exploit cluster id-dependent branches by observing memory risk of leaking a particular access pattern. Second, converting
page access patterns. Thus, the adversary can estimate the the program to an oblivious one can be complex and error-
cluster size, which may be sensitive to the user. However, prone. We investigate the existing candidate approaches and
a simplified oblivious version of this step hides the secret summarize the following four most representative ones for
dependent branch by accessing each cluster object with a developing oblivious solutions: (i) manual composition (or
CMOV operation. Thus, the attacker cannot distinguish which manual approach), (ii) compiler approach, (iii) circuit ap-
cluster-id is being updated for the training data. proach, and (iv) framework approach.
Against Cache Attacks. Cache-based side-channel attacks A. Manual Composition
[27] had been long exploited before TEE became popular. In manual composition, developers need to learn all the
The basic mechanism of cache attacks remains the same for knowledge of sensitive access patterns and the methods of
systems with or without TEE. The main idea of the cache converting them to be oblivious. These approaches may vary
attack is to load the system memory into the CPU cache and depending on the applications’ related access pattern problems.
perform a time analysis by loading different byte values to The key challenges of this approach are to manually analyze
retrieve the value of the previously loaded memory, such as the access pattern problem for every line of the code and
Prime+Probe [24] and Flush+Reload [48] methods. replace the vulnerable parts with their oblivious alternatives.
Like regular applications, TEE is also vulnerable to cache- Developers may also need to experiment with different obliv-
based side-channel attacks. Since the last level cache (LLC) is ious primitives to determine the most efficient one. It’s also
a shared resource, an attacker can exploit fine-grained infor- necessary to verify whether the conversion is complete with a
mation at a specific stage of the program by probing the data tool such as ObliCheck [42].
access time in each cache line. However, in a data-oblivious Several problem-specific manual compositions have been
algorithm, all the steps and data accesses are fixed. Thus, an reported to address the access-pattern based attacks on TEE
attacker cannot distinguish the secret value and dummy access applications. To protect the random access over block data
from the cache-level timing at a given time. For example, a in untrusted memory, the developer can implement Oblivious
cache attack cannot distinguish the secret-dependent block IDs RAM that works with TEE, e.g., ZeroTrace [2], [36], that
or block data from dummy ones if accessed through oblivious hides which block is read or written by shuffling memory
RAM [2], [36]. blocks during each access. Other problem-specific algorithms
have also been used to hide access patterns of specific tasks,
Against Micro-architectural Attacks. Some powerful at- such as Oblivious Sorting [4], Oblivious Filter [50], Oblivious
tacks exploit the CPU’s micro-architecture to retrieve secrets Join [21], etc. CMOV-based oblivious branching is also ac-
from TEE applications. Foreshadow [7] exploits meltdown- tively applied in developing specific machine algorithms [29]
type [22] attacks on TEE applications. Load Value Injection and addressing the in-enclave access patterns [3], [36], with
(LVI) [43] is the most recent attack on Intel SGX that suc- wrapped functions such as oblivious move, greater, swap, etc.
cessfully retrieves the secrets from the victim’s Enclave within Manually applying oblivious solutions enables the designing
the victim’s address space. The CPU’s micro-architectural of both memory and performance-efficient application. How-
buffer must be prepared with some attacker-controlled secret ever, manually analyzing sensitive data access and code vul-
value to perform the LVI attack. These attacks are powerful nerabilities and applying oblivious solution is time-consuming,
enough to extract plain text information from the TEE without domain-expertise demanding, and sometimes error prone. De-
physical access. Manufacturers have issued microarchitectural- velopers may utilize existing oblivious libraries [12] to reduce
level firmware patches for some [7], [43] of these attacks. manual efforts.

However, not all micro-architectural attacks can be pre- B. Compiler Approach

vented from firmware-level patches. Some micro-architectural- Compiler techniques (e.g., static code analysis) can be used
level attacks still utilize access patterns. For example, to minimize manual efforts. The current compiler approaches
Bulck et al. [45] show that, by exploiting the timing of for generating oblivious code follow two directions: automate
micro-architectural instructions, attackers can observe secret- the manual composition approach, and hide code and data
dependent branches at the CPU instruction level. This type access patterns via randomization.
of micro-architectural-level attacks cannot succeed if the ap- The first category of approaches includes Raccoon [33]
plication developer hides the data-dependent branches with and Ghostrider [23] for automating the manual process. They
oblivious solutions. Therefore, oblivious programs can still utilize static code analysis to detect vulnerabilities, i.e., the
help mitigate these attacks. primitive operations that need to be obfuscated, and then
apply suitable oblivious measures, as we have discussed in the • Loops. In circuits, each gate is executed only once. Thus,
manual approach. They also use a few methods to optimize circuits do not have any loops. Compilers unroll the loops
performance. For example, they often provide an option to to linear execution for bounded iterations. If you do not
allow developers to annotate the parts to be obfuscated; based provide a bound, it will unroll the loop up to the size of
on the array size, the compiler can decide to use linear scan data types. For example, a byte-type loop variable will
or ORAM to hide the accessed element. unroll for 256 iterations.
Another compiler approach is to randomize access patterns • Loops with breaks. Loops with breaks are common in
and program execution paths [1], [6], [31] by using primitives programming. While older compilers do not support
like ORAMs. For example, Obfuscuro [1] divides the memory break statements, modern compilers support break/con-
and code pages into two classes and achieves obliviousness tinue commands. In short, compilers use a similar ap-
via ORAM. Similarly, Dr. SGX [6] and CoSMIX [31] also proach that resolves conditional branching. Expressly,
use data randomization techniques to achieve fully automated compilers turn the loop into a breakable block. When
memory obfuscation. The non-deterministic run-time access there is a breakable block, it will execute all iterations,
patterns successfully hinder the adversary from extracting any except it will use guarded execution to hide which value
information. will be chosen last.
These approaches are mostly experimental, not fully achiev- • Random Array access. That array access depends on the
ing the design goal yet. First, while the compiler approach input. It requires O(n) operations. Since the value is not
avoids most manual efforts, it may not generate the most constant, array access is implemented using an n-sized
efficient oblivious solution. In particular, the randomization multiplexer with the index as the multiplexer selector.
approach often results in significantly high costs. Second, Cryptographic circuit compilers such as CircC [32] and
since the compilers depend on static analysis to identify the HyCC [9] can convert regular C programs to executable C
sensitive code blocks, they may apply unnecessary obfuscation circuit programs. However, very few TEE-related studies [16],
due to a lack of context awareness and the limitation of static [38] have applied circuits as an access pattern protection
analysis tools. Son et al. [42] showed how static analysis-based mechanism. We show in experiments that the performance can
methods failed to recognize standard oblivious algorithms be a significant concern due to three reasons (1) the extensive
and were marked as non-oblivious, leading to unnecessary uses of the above methods to ensure obliviousness; (2) the size
obfuscation in the mitigation phase. of the generated circuit is large, proportional to the input data
The ultimate goal of the compiler approach is to entirely size; (3) executing a circuit in a software mode is inherently
free developers from complex and expensive manual efforts, slow.
which is appealing. Unfortunately, existing compilers have not
reached the goal yet. In fact, we could not find a stable open- D. Framework Approach
source implementation for any of the mentioned approaches. For TEE-based data-intensive applications, the framework
Nevertheless, we still believe this is a promising direction. approach can be a valid candidate [3], [15], [29], [37], [50]. We
refer “framework” here to the well-known big data processing
C. Circuit Approach
frameworks, such as MapReduce [14] and Spark [49]. We have
In many cryptographic approaches [20], [25], circuits have witnessed two types of framework approaches.
been used as a building block for secure evaluation. Boolean The first type aims to extend the big data frameworks to
circuits are naturally oblivious as they execute all the paths take advantage of TEEs, which will have to handle side-
[32], [41]. Since automated program-to-circuit conversion channel attacks, as well. VC3 [37] applied this strategy for
tools have been built for the cryptographic approaches, such modifying the Hadoop [46] system. M2R [15] targets the
as ObliVM [23], CircC [32], and HyCC [9], we wonder problem of access-pattern leakage in the shuffling phase of
whether this approach can also be a candidate for TEE-based VC3 and proposes to use the oblivious schemes for shuffling.
applications. Another significant work, Opaque [50], tries to revise Spark
Cryptographic circuit compilers ensure many things, includ- for SGX. They focus on the data access patterns between
ing variable mutations, conditional branching, loops, loops computing nodes, illustrate how adversaries can use these to
with breaks, early returns, and random array access, are infer sensitive information in the encrypted data and design
oblivious. We briefly describe some of the key features: data-oblivious methods to address these attacks. These frame-
• Variable Mutations. In a regular program, the value of a works have a shared weakness: only a few data processing
variable can be updated; however, in Boolean circuits, components of the framework were moved to the TEE, leaving
values are mutation free. When required to update a most parts of the framework in untrusted areas under serious
variable, it uses versions of the variable. For example, threats. However, re-implementing the frameworks with the
if the previous value of x was zero and then assigned to data obliviousness guarantee is too expensive to be practical.
ten, it will be converted to x0 = 0, x1 = 10. The second type of framework approach utilizes a data-
• Branching. To protect the branching attack, researchers intensive framework to simplify the development of data
implement guarded execution, executing both branches oblivious solutions. Specifically, the framework serves as
and selecting the result based on condition. the middleware to hide the complexity of data-oblivious
processing. The developer only needs to spend much less and 16 GB of DRAM. The TEE environment is Intel SGX
effort implementing data-oblivious TEE applications. SGX- v1.0, and the Linux version is Ubuntu 22.04.
MR [3] utilizes the MapReduce processing model to regulate Approach Implementation. The unprotected approach is
application dataflow so that the framework can integrate the the simple implementation without considering obliviousness.
application-independent data-oblivious protection mechanisms The manual composition approach has used the following
in the framework code to protect any data mining algorithms primitive operations: linear scan and ORAM for random array
that can be cast to the MapReduce framework. Developers accesses, the omove/ogreater functions [3], [29], [36] for
only need to handle a much smaller number of and often oblivious branching, and BitonicSort for sorting, to convert the
simpler access patterns in the application-specific map and implementation. We used ZeroTrace’s open-source implemen-
reduce functions. tation for ORAM on SGX, and the HyCC circuit generator [9]
This framework approach has a few unique benefits: (1) It to convert a plain implementation to a circuit for the circuit-
significantly reduces the complexity of the manual approach based evaluation. For data-intensive applications, we adopted
for data-intensive processing; (2) It can integrate both the latest the SGX-MR framework in the evaluation. Unfortunately, we
big-data processing techniques and data-oblivious solutions at could not find a working open-source implementation for the
the framework level, which is transparent to developers; (3) It published compiler approaches. Due to the sheer complexity
can incorporate data-intensive optimization techniques in the of implementing such a compiler, we have not included the
framework implementation, e.g., most block-based operations. compiler approach in the evaluation.
Thus it can be more efficient than directly applying fully • Oblivious Operations. We evaluate the three basic obliv-
automated compiler or circuit approaches that are unaware of ious operations: random array access (block access in
data-intensive features. data-intensive workloads), branching, and sorting. This
We use Table I to summarize the four approaches qual- evaluation should help understand the performance of
itatively in terms of ease of use, performance, and unique each approach at low-level operations.
features. In experiments, we will see more quantitative results. • Sample Applications. In addition to the basic oblivi-
ous operations, we also include four applications for
TABLE I: Summary of existing data oblivious solutions.
compute-intensive and data-intensive workloads, two for
Method Easy use Performance Other features each category respectively. Each application will be a
Manual Low High Require expertise,
time consuming
certain blend of primitive oblivious operations. (1) The
Compiler High Mid May incur unnecessary compute-intensive applications include the Edit Distance
obfuscation computation for varying lengths of strings, which has a
Circuit Mid Low compilation very slow, complexity of O(N 2 ) for the string length N , and the
circuit size greater than data
F/W High High only domain specific All-Pair Shortest Path algorithm, Floyd-Warshall, with
functionalities a complexity of O(N 3 ) for N nodes in the graph. (2)
The data-intensive applications include the WordCount
and KMeans clustering algorithms.
V. E XPERIMENTAL E VALUATIONS In the following, we will organize the results in terms of
The experimental evaluation has the following goals. compute-intensive and data-intensive workloads. The primitive
• Observe how different conversion approaches perform on operations are discussed under each category correspondingly.
the primitive operations, which are the building blocks
B. Results for Compute-Intensive Workloads
of an oblivious program. These operations include (i)
oblivious data access, (ii) oblivious branching, and (iii) Compute-intensive workloads use a relatively small dataset
basic oblivious algorithms, such as sorting. The primitive that can be fit into the TEE memory. We choose three basic
operations will be discussed under compute-intensive and operations and then two well-known applications to evaluate
data-intensive workloads. the performance of different oblivious methods.
• Understand how the oblivious programs generated with Random Array Access. Accessing array elements is one
different approaches perform. We evaluate the perfor- of the basic operations of most applications. Figure 2(a)
mance with compute-intensive and data-intensive bench- shows the execution time of various oblivious implementa-
mark applications. tion methods. First, the expenses of all oblivious solutions
• Investigate the ease of use, i.e., the developers’ efforts, are significantly higher than the unprotected version. ORAM
via multiple measures, e.g., the line of code (LOC), the shows performance advantages over linear scans for larger
number of sensitive code blocks susceptible to access sizes. For smaller sizes, ORAM’s maintenance cost becomes
pattern leakages, and LOC overhead to achieve mitigation relatively high, resulting in higher overall costs than linear
techniques. scans. The circuit version of array access also uses linear
scans. However, executing a circuit at the software level is
A. Experiment Setup clearly very inefficient regardless of the array size.
The experiments were conducted on a Linux machine with Branching. We design a benchmark program to evaluate
an Intel(R) Core(TM) i7-8700K CPU of a 3.70GHz processor the impact of oblivious branching. In a loop, a branching
 102

Execution time (ms)

Execution time (ms) Unprotected
1065
105 Manual-Linear Scan Circuit
104

Execution Time (ms)

104 Manual-ORAM 1 Manual
103 Circuit 10 103 Unprotected
102 102
101 100 101
100 100
10−1 10
10−2 10−1 10−1
10−3 10−2
101 102 103 104 105 10−2 24 25 26 27 28
Number of records Number of records
Unprotected Manual Circuit
(a) Oblivious array access. Record size 8 bytes.
(b) Oblivious branching. (c) Oblivious sorting. Record size 8 bytes.

Fig. 2: Performance evaluation of in-memory core operations.

statement decides to execute one of the two functions: one operations. We used 1-KB blocks filled with string data. The
with a low cost and the other with a high cost. We repeated the manual approach adopts a block-level BitonicSort algorithm
experiments a few times. Naturally, the unprotected version’s with oblivious in-block operations [3], while the circuit ap-
performance varies over different runs. Figure 2(b) shows proach converts the block-level BitonicSort algorithm. In Fig-
that the manual approach, which uses CMOV instructions, ure 4(b), we observe that the circuit approach is still orders of
is relatively efficient. However, the circuit approach shows magnitude higher than the manual approach, while the manual
multiple orders of magnitude higher costs. approach is about ten times slower than the unprotected one.
Sorting. The manual approach adopts BitonicSort. Since WordCount. For application-level evaluation, we take a
the circuit approach can convert any algorithm to its oblivious fixed amount of input, 500 1KB blocks, each of which is
version, we have considered different sorting algorithms for filled with random text. As the MapReduce-based solution
the circuit approach. It turns out BitonicSort is also the is the most efficient one for the WordCount problem, the
most efficient one in circuit form. Figure 2(c) shows the manual approach essentially duplicates the processing encoded
manual approach is much faster than the circuit approach on in the framework of SGX-MR, which uses BitonicSort for
BitonicSort. the intermediate sorting of word-count pairs. As a result, the
Edit Distance. Edit distance uses dynamic programming to manual approach has an almost identical cost to the SGX-
compute the distance between two sequences, whose complex- MR approach. Certainly, SGX-MR significantly simplifies the
ity is O(N 2 ) for sequences of length N . It’s a typical high- developer’s coding efforts. Again, the circuit approach is too
complexity algorithm working with a relatively small amount expensive to be a practical solution.
of memory. In Figure 3(a), we notice that the manual method is KMeans. We use 4000 1k data blocks consisting of 34×104
close to the unprotected version, but again the circuit approach records and five clusters. Due to the small number of clusters,
is much more expensive than others. we use hash and ORAM for aggregation (check Appendix A),
All-pair shortest path. The all-pair shortest path Floyd- which appears more efficient than sorting-based aggregation in
Warshall algorithm is expensive with a complexity of O(N 3 ) SGX-MR. Figure 5(b) shows this manual approach performs
for N nodes. Figure 3(b) shows the manual approach is signif- best among the candidate techniques.
icantly slower than the unprotected one, but at a manageable D. Developers’ effort to achieve data oblivious solutions
scale. In contrast, the circuit approach is too expensive to
handle larger N . We are also curious about how easy a developer can use
each of these approaches. This evaluation does not include
C. Results for Data-intensive Workloads the extra time learning the different approaches – apparently,
For data-intensive workloads, we evaluate two basic oper- developers need to take a significant amount of time to learn
ations: block-level random access and block-based external the manual approach and the framework approach.
sorting. In application-based evaluations, we also include the Instead, we look at the result of developing the evaluated
framework approach: SGX-MR. applications to understand the difficulty levels of using differ-
Random block access. Similar to the evaluation on array ent approaches. We also assume developers will use a library
access, we include linear scan and ORAM methods for the of oblivious primitives, e.g., ORAM, oblivious branching, and
manual approach. Figure 4(a) clearly shows the manual- oblivious sorting. The use of library will also significantly
ORAM approach performs much better. However, due to the reduce the line of code (LOC) for the manual and framework
large data size, the gap between oblivious approaches and the approaches.
unprotected is also large. Table II summarizes the additional effort a developer need
Block-based External Sorting. Next, we implement block- to achieve data oblivious applications. In the following we will
level sorting to understand the sorting cost in block-level discuss the compared data oblivious strategies.
 106 104
Unprotected
105 103
Execution time (ms)

Execution time (ms)

Manual
104 Circuit 102
103
101
102
101 100
Unprotected
100 10−1 Manual
10−1 10−2
Circuit

10 20 40 60 80 100 0 50 100 200 300 400

Sequence lengths Number of Nodes
(a) Edit distance. (b) Floyd-Warshall all-pair shortest path.

Fig. 3: Performance evaluation of compute-intensive applications.

Circuit Circuit Manual Unprotected

107 Manual-Linear Scan 105
106
Execution time (ms)

Manual-ORAM

Execution time (s)

105 Unprotected
104
104
103
102 103
101
100 102
10−1
10−2 101
10−3
3,000 9,000 15,000 21,000 100
250 500 1,000 1,500 2,000
Number of blocks
Number of blocks
(a) Comparing random Access over blocks. Block size 1KB.
(b) Block-based oblivious sorting. Block size 1KB with 75 words
(records) per block.

Fig. 4: Performance evaluation of core operations for data-intensive applications.

• Manual Composition. Manual composition requires do- count, and for KMeans developer only needs to add six
main knowledge on TEE side channels. The table shows lines of code to solve one access pattern issue.
the manual approach requires identifying one to six Overall, the circuit approach is the easiest to use as it
sensitive code segments and hiding the access pattern does not require any additional effort from the developer.
with data-oblivious alternatives. This approach requires The manual approach involves a lot of efforts in analyzing
the developer to write more lines of code than other the original code and conducting the conversion. In contrast,
approaches, even when the oblivious library is used. the framework approach hides many details with the frame-
• Circuit. The circuit approach is fully automated, and the work implementation and minimizes the developer’s efforts.
developer does not need to do any additional work. However, it does require the developer to learn to use the
• Framework. With a framework like SGX-MR, the devel- framework first.
oper only focuses on small pieces of application-specific
code, such as the map and reduce functions, dramatically VI. C ONCLUSION
reducing the developer’s burden compared to the manual Data oblivious programs provide excellent defenses against
approach. The framework software contains fully opti- several side-channel attacks targeting TEE applications. How-
mized oblivious code that is transparent to developers ever, developing oblivious programs is challenging. We have
and shared by all SGX-MR applications. Table II shows analyzed four representative approaches that can help develop-
by using SGX-MR for framework-level protection, the ers convert non-oblivious programs to oblivious ones. Among
developer does not require any LOC overhead for word these approaches, we consider performance and ease of use
 104 105

Execution Time (s)

Execution Time (s)

104
3
10
103

102
102
101

101 100
Manual SGX-MR Circuit Manual SGX-MR Circuit
(a) Application-level perfromance for Wordcount. (b) Application-level performance for KMeans.
Number of blocks 500 with 75 words/block. 4000 1KB-Blocks with eight bytes per record,
and five clusters.

Fig. 5: Performance evaluation of data-intensive applications.

TABLE II: Summary of developers’ effort to implement the oblivious solutions. The table represents the total line of code as
LOC and the number of access-pattern sensitive segment in the code as AP. LOC-overhead means the lines used to achieve
data obliviousness.
Application Manual Circuit Framework
LOC LOC-Overhead AP LOC LOC-Overhead AP LOC LOC-Overhead AP
Edit Distance 58 28 4 48 0 - - - -
All-Pair Shortest Path 47 15 1 36 0 - - - -
Word Count 277 21 6 155 0 - 22 0 0
KMeans 330 24 4 263 0 - 58 6 1

(and possibly readiness to use) are critical measures. Our [3] A. K. M. M. Alam, S. Sharma, and K. Chen. Sgx-mr: Regulating
experimental results show that: (1) The manual composition dataflows for protecting access patterns of data-intensive sgx applica-
tions. Proceedings on Privacy Enhancing Technologies, 2021(1):5 – 20,
approach gives the best performance guarantee, while devel- 01 Jan. 2021.
opers must fully understand the access pattern of every part [4] K. E. Batcher. Sorting networks and their applications. In Proceedings
of their code and learn the corresponding conversion method; of the April 30–May 2, 1968, Spring Joint Computer Conference, AFIPS
’68 (Spring), pages 307–314, New York, NY, USA, 1968. ACM.
(2) The framework approach for data-intensive applications [5] Z. Brakerski and V. Vaikuntanathan. Fully homomorphic encryption
achieves a good balance between performance and ease of from ring-lwe and security for key dependent messages. In Proceedings
use; (3) The circuit approach is theoretically sound, but of the 31st Annual Conference on Advances in Cryptology, CRYPTO’11,
pages 505–524, Berlin, Heidelberg, 2011. Springer-Verlag.
extremely expensive in practice; and (4) the compiler approach [6] F. Brasser, S. Capkun, A. Dmitrienko, T. Frassetto, K. Kostiainen, and
is promising, but not mature enough for practical use. We A.-R. Sadeghi. Dr. sgx: Hardening sgx enclaves against cache attacks
hope our analysis and evaluation will help both practitioners with data location randomization. arXiv preprint arXiv:1709.09917,
2017.
to decide their solutions and researchers to explore potential
[7] J. V. Bulck, M. Minkin, O. Weisse, D. Genkin, B. Kasikci, F. Piessens,
issues. M. Silberstein, T. F. Wenisch, Y. Yarom, and R. Strackx. Foreshadow:
We consider a few promising research directions. (1) The Extracting the keys to the intel SGX kingdom with transient out-of-order
compiler approach is the most appealing one, as it aims to execution. In 27th USENIX Security Symposium (USENIX Security 18),
page 991–1008, Baltimore, MD, Aug. 2018. USENIX Association.
make the conversion process fully transparent to developers [8] J. V. Bulck, N. Weichbrodt, R. Kapitza, F. Piessens, and R. Strackx.
and the converted program to have a good performance close Telling your secrets without page faults: Stealthy page table-based
to the manual composition approach. (2) Another direction attacks on enclaved execution. In 26th USENIX Security Symposium
(USENIX Security 17), pages 1041–1056, Vancouver, BC, Aug. 2017.
is data oblivious libraries/frameworks and software tools to USENIX Association.
automate the conversion process as possible and minimize [9] N. Büscher, D. Demmler, S. Katzenbeisser, D. Kretzmer, and T. Schnei-
the developers’ manual efforts. The framework approach is der. Hycc: Compilation of hybrid protocols for practical secure computa-
tion. In Proceedings of the 2018 ACM SIGSAC Conference on Computer
an excellent example of this direction. and Communications Security, pages 847–861, 2018.
[10] D. Cash, P. Grubbs, J. Perry, and T. Ristenpart. Leakage-abuse attacks
R EFERENCES against searchable encryption. In Proceedings of the 22nd ACM SIGSAC
[1] A. Ahmad, B. Joe, Y. Xiao, Y. Zhang, I. Shin, and B. Lee. Obfuscuro: A Conference on Computer and Communications Security, CCS ’15, page
commodity obfuscation engine on intel sgx. In Network and Distributed 668–679, New York, NY, USA, 2015. Association for Computing
System Security Symposium, 2019. Machinery.
[2] A. Ahmad, K. Kim, M. I. Sarfaraz, and B. Lee. Obliviate: A data [11] K. Chen and S. Guo. Rasp-boost: Confidential boosting-model learning
oblivious file system for intel sgx. In the Network and Distributed with perturbed data in the cloud. IEEE Transactions on Cloud Comput-
System Security Symposium, 2018. ing, 6(2):584–597, 2018.
[12] S. D. Constable and S. Chapin. libOblivious: A c++ library for oblivious Fei. ImageNet Large Scale Visual Recognition Challenge. International
data structures and algorithms. In Electrical Engineering and Computer Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
Science - Technical Reports. 184, 2018. [35] S. Sagar and C. Keke. Confidential machine learning on untrusted
[13] V. Costan and S. Devadas. Intel sgx explained. IACR Cryptology ePrint platforms: a survey. Cybersecurity, 4(1):1–19, 2021.
Archive, 2016:86, 2016. [36] S. Sasy, S. Gorbunov, and C. W. Fletcher. Zerotrace : Oblivious memory
[14] J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on primitives from intel SGX. In 25th Annual Network and Distributed
large clusters. In OSDI, pages 137–150, 2004. System Security Symposium, NDSS 2018, San Diego, California, USA,
[15] T. T. A. Dinh, P. Saxena, E. Chang, B. C. Ooi, and C. Zhang. M2R: February 18-21, 2018, 2018.
enabling stronger privacy in mapreduce computation. In USENIX [37] F. Schuster, M. Costa, C. Fournet, C. Gkantsidis, M. Peinado, G. Mainar-
Security Symposium, pages 447–462. USENIX Association, 2015. Ruiz, and M. Russinovich. Vc3: Trustworthy data analytics in the cloud
[16] S. Felsen, Á. Kiss, T. Schneider, and C. Weinert. Secure and private using sgx. In 36th IEEE Symposium on Security and Privacy, 2015.
function evaluation with intel sgx. In Proceedings of the 2019 ACM [38] O. A. Selo, M. H. Rachid, A. Shikfa, Y. Wang, and Q. Malluhi. Private
SIGSAC Conference on Cloud Computing Security Workshop, pages function evaluation using intel’s sgx. Security and Communication
165–181, 2019. Networks, 2020:1–10, 2020.
[17] Freundschafter. About amd trustzone, amd platform security processor [39] S. Sharma and K. Chen. Confidential boosting with random linear
(psp), amd secure technology. accessed: Jan. 13, 2020. [Online]. classifiers for outsourced user-generated data. In Computer Security
[18] O. Goldreich and R. Ostrovsky. Software protection and simulation on - ESORICS 2019 - 24th European Symposium on Research in Computer
oblivious ram. Journal of the ACM, 43:431–473, 1996. Security, Luxembourg, September 23-27, 2019, Proceedings, Part I,
[19] D. A. Heath. New Directions in Garbled Circuits. PhD thesis, Georgia pages 41–65, 2019.
Institute of Technology, 2022. [40] S. Shinde, Z. L. Chua, V. Narayanan, and P. Saxena. Preventing page
[20] Y. Huang, D. Evans, J. Katz, and L. Malka. Faster secure two-party faults from telling your secrets. In Proceedings of the 11th ACM on Asia
computation using garbled circuits. In USENIX Conference on Security, Conference on Computer and Communications Security, ASIACCS16,
pages 35–35, 2011. page 317–328, New York, NY, USA, 2016. Association for Computing
[21] S. Krastnikov, F. Kerschbaum, and D. Stebila. Efficient oblivious Machinery.
database joins. Proc. VLDB Endow., 13(12):2132–2145, jul 2020. [41] R. L. Simon. Fair play: The ethics of sport. Routledge, 2018.
[22] M. Lipp, M. Schwarz, D. Gruss, T. Prescher, W. Haas, A. Fogh, [42] J. Son, G. Prechter, R. Poddar, R. A. Popa, and K. Sen. ObliCheck:
J. Horn, S. Mangard, P. Kocher, D. Genkin, Y. Yarom, and M. Hamburg. Efficient verification of oblivious algorithms with unobservable state. In
Meltdown: Reading kernel memory from user space. In 27th USENIX 30th USENIX Security Symposium (USENIX Security 21), 2021.
Security Symposium (USENIX Security 18), 2018. [43] J. Van Bulck, D. Moghimi, M. Schwarz, M. Lipp, M. Minkin, D. Genkin,
[23] C. Liu, X. S. Wang, K. Nayak, Y. Huang, and E. Shi. Oblivm: Y. Yuval, B. Sunar, D. Gruss, and F. Piessens. LVI: Hijacking Transient
A programming framework for secure computation. In 2015 IEEE Execution through Microarchitectural Load Value Injection. In 41th
Symposium on Security and Privacy, pages 359–376, May 2015. IEEE Symposium on Security and Privacy (S&P’20), 2020.
[24] F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee. Last-level cache [44] J. Van Bulck, F. Piessens, and R. Strackx. Sgx-step: A practical attack
side-channel attacks are practical. In Proceedings of the 2015 IEEE framework for precise enclave execution control. In Proceedings of the
Symposium on Security and Privacy, SP ’15, page 605–622, USA, 2015. 2nd Workshop on System Software for Trusted Execution, SysTEX’17,
IEEE Computer Society. New York, NY, USA, 2017. Association for Computing Machinery.
[25] P. Mohassel and Y. Zhang. Secureml: A system for scalable privacy- [45] J. Van Bulck, F. Piessens, and R. Strackx. Nemesis: Studying mi-
preserving machine learning. In 2017 IEEE Symposium on Security and croarchitectural timing leaks in rudimentary cpu interrupt logic. In
Privacy (SP), pages 19–38, 2017. Proceedings of the 2018 ACM SIGSAC Conference on Computer and
[26] V. Nikolaenko, U. Weinsberg, S. Ioannidis, M. Joye, D. Boneh, and Communications Security, pages 178–195, 2018.
N. Taft. Privacy-preserving ridge regression on hundreds of millions of [46] T. White. Hadoop: The Definitive Guide. O’Reilly Media, 2009.
records. In IEEE Symposium on Security and Privacy, pages 334–348, [47] Y. Xu, W. Cui, and M. Peinado. Controlled-channel attacks: Deter-
2013. ministic side channels for untrusted operating systems. In 2015 IEEE
[27] A. Nilsson, P. N. Bideh, and J. Brorsson. A survey of published attacks Symposium on Security and Privacy, pages 640–656, May 2015.
on intel sgx. arXiv preprint arXiv:2006.13598, 2020. [48] Y. Yarom and K. Falkner. Flush+reload: A high resolution, low noise,
[28] O. Ohrimenko, M. Costa, C. Fournet, C. Gkantsidis, M. Kohlweiss, l3 cache side-channel attack. In Proceedings of the 23rd USENIX
and D. Sharma. Observing and preventing leakage in mapreduce. In Conference on Security Symposium, SEC’14, page 719–732, USA, 2014.
Proceedings of the 22nd ACM SIGSAC Conference on Computer and USENIX Association.
Communications Security, CCS ’15, page 1570–1581, New York, NY, [49] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica.
USA, 2015. Association for Computing Machinery. Spark: Cluster computing with working sets. In Proceedings of the 2Nd
[29] O. Ohrimenko, F. Schuster, C. Fournet, A. Mehta, S. Nowozin, USENIX Conference on Hot Topics in Cloud Computing, HotCloud’10,
K. Vaswani, and M. Costa. Oblivious multi-party machine learning pages 10–10, Berkeley, CA, USA, 2010. USENIX Association.
on trusted processors. In T. Holz and S. Savage, editors, 25th USENIX [50] W. Zheng, A. Dave, J. G. Beekman, R. A. Popa, J. E. Gonzalez, and
Security Symposium, USENIX Security 16, Austin, TX, USA, August 10- I. Stoica. Opaque: An oblivious and encrypted distributed analytics
12, 2016, pages 619–636. USENIX Association, 2016. platform. In USENIX Symposium on Networked Systems Design and
[30] O. Ohrimenko, F. Schuster, C. Fournet, A. Mehta, S. Nowozin, Implementation, 2017.
K. Vaswani, and M. Costa. Oblivious multi-party machine learning
on trusted processors. In 25th USENIX Security Symposium, USENIX A PPENDIX A
Security 16, Austin, TX, USA, August 10-12, 2016., pages 619–636, S AMPLE ORAM-H ASH ALGORITHMS
2016.
[31] M. Orenbach, Y. Michalevsky, C. Fetzer, and M. Silberstein. Cosmix: A
compiler-based system for secure memory instrumentation and execution
in enclaves. In USENIX Annual Technical Conference, pages 555–570,
2019.
[32] A. Ozdemir, F. Brown, and R. S. Wahby. Circ: Compiler infrastructure
for proof systems, software verification, and more. In 2022 IEEE
Symposium on Security and Privacy (SP), pages 2248–2266. IEEE,
2022.
[33] A. Rane, C. Lin, and M. Tiwari. Raccoon: Closing digital side-channels
through obfuscated execution. In Proceedings of the 24th USENIX
Conference on Security Symposium, SEC’15, page 431–446, USA, 2015.
USENIX Association.
[34] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-
Algorithm 1 Buffer Management for ORAM-Hash algorithms
1: Buffer contains a working block B for new records, and
a cache of m blocks C.
2: Function GetBlock(block id)
3: if block id not in the cache C then
4: decide a block to overwrite with an algorithm like LRU;
the victim block is written back to the output file.
5: new block ← request oram block(block id)
6: add new block to the cache
7: end if
8: block ref erence ← find the block id in the cache
9: return block ref erence
1: Function AddRecordToBlock(record)
2: if working block B is full then
3: evict LRU and write out the victim block
4: copy B to the cache
5: clear the working block
6: end if
7: add record to the working block
8: return working block id

Algorithm 2 HashMap-based KMeans in Enclave

1: Function KMeans(centroid f ile, coordinates f ile)
2: initialize ORAM and Cache block
3: load initial centroids from centroid f ile
4: for all block in coordinates f ile do
5: points ← ParseCoordinates(block)
6: for each pt in points do
7: centroid index ← FindNearestCentroid(pt,
centroids)
8: LocalMap[ centroid index ] ← pt
9: end for
10: combined points ← aggregate points under same cen-
troid
11: for each (centroid index, point) in
combined points do
12: if centroid index not not in HashM ap then
13: record ← (centroid index, point)
14: id ← AddRecordToBlock(centroid index)
15: HashM ap[centroid index] ← id
16: else
17: id ← HashM ap[centroid index]
18: block ref erence ← GetBlock(id)
19: update block in block ref erence with
(centroid index, point)
20: Write(block)
21: end if
22: end for
23: end for
24: write all blocks from ORAM to centroid f ile