Sampling-Based Binary-Level Cross-Platform

Performance Estimation
Xinnian Zheng, Haris Vikalo, Shuang Song, Lizy K. John, Andreas Gerstlauer
The University of Texas at Austin, TX, USA
{xzheng1, hvikalo, songshuang1990, ljohn, gerstl}@utexas.edu

Abstract—Fast and accurate performance estimation is a require intrusive instrumentation of a compiler-generated inter-
key challenge in modern system design. Recently, machine mediate representation (IR) with profiling calls at basic block
learning-based approaches have emerged that allow predicting boundaries to capture phase-level performance features of a
the performance of an application on a target platform from
executions on a different host. However, existing approaches program on the host as well as, during training, reference
rely on expensive instrumentation that requires source code performance on the target. Instrumentation at the IR basic
to be available. We propose a novel sampling-based, binary- block level leads to three major problems, however: (1)
level cross-platform prediction method that accurately predicts it introduces performance overhead, (2) application source
performance of a workload on a target by relying on various code is required, and (3) prediction is only possible for a
performance statistics sampled on a host using built-in hardware
counters. In our proposed framework, samples acquired from single standalone application at a time. These inherently limit
the host and target do not satisfy straightforward one-to-one practical applicability, e.g. to predict performance of an actual
correspondence that characterizes prior instrumentation-based application and system code mix running on a platform.
approaches. The resulting alignment problem is NP-hard; to solve In this paper, we instead propose a sampling-based, binary-
it efficiently, we develop a stochastic dynamic coupling (SDC) level technique that addresses the drawbacks of existing
algorithm which, under mild assumptions, with high probability
closely approximates optimal alignment. The prediction model instrumentation-based cross-platform prediction methods. Our
constructed using SDC-aligned samples achieves on average approach is source-oblivious and can perform background
96.5% accuracy for 45 benchmarks at speeds of over 3 GIPS. At prediction for arbitrary binary code while maintaining similar
similar accuracies, this is up to 6× faster than instrumentation- prediction accuracy with a significant increase in speed. A
based prediction, and approximately twice the speed of executing key challenge in sampling-based learning methods is proper
the same applications natively on our ARM target.
alignment of samples during training. In instrumentation-based
I. I NTRODUCTION approaches, since the IR is architecture-independent, profiling
Estimating performance of complex software on hardware every fixed number of dynamic basic blocks guarantees an
that is not readily available is among the key challenges in exact correspondence between the performance features on the
modern system-level design. Due to increasing complexity host and reference performance on the target for each captured
of software and hardware, performance modeling and pre- phase. By contrast, when host and target measurements are
diction are difficult tasks. Widely adopted simulation-based sampled with respect to time rather than to fixed block or
approaches, such as cycle-accurate instruction set simulators phase boundaries, the one-to-one correspondence between host
(ISSs), excel in accuracy yet lack significantly in speed. and target measurements is lost. Due to a different pace of
By contrast, traditional analytical models are computationally application progress, sampling an application at the same rate
efficient but their accuracy is limited. To this end, novel on different hardware platforms almost always results in a
machine learning-based cross-platform prediction techniques vastly different number of samples. Hence, the main challenge
have recently been proposed [1], [2]. Compared to traditional is to align and synchronize the host and target samples during
cycle-accurate ISSs, cross-platform methods offer several or- training such that they approximately correspond to the same
ders of magnitude speedup while maintaining similar accuracy. section of executed code. To solve this alignment problem,
The key idea behind such approaches is the simple intuition we propose a novel, efficient and effective stochastic dynamic
that performance of the same application running on two coupling (SDC) heuristic. Crucially, numerical measurements
different hardware platforms is correlated, and that this latent from the host and target can be viewed as two stochastic
correlation can be extracted by supervised learning methods signals, and our aim is to align them such that their cross-
to ultimately predict performance of applications running on a covariance is maximized. Using the aligned samples from
target while executing them natively on a host. Resulting pre- the host and target, we then train prediction models, which
diction models trained by executing small micro-benchmarks correlate performance from the host to the target.
on an early reference implementation or model of the target The rest of the paper is organized as follows: after a
can be used by software and hardware developers to evaluate discussion of related work and an overview of our approach,
large, real-world application behavior that would otherwise Section II presents the formulation of the alignment problem
require target access or be too slow if not infeasible to obtain. and the SDC algorithm as well as theoretical performance
Existing cross-platform prediction approaches achieve high guarantees. Section III discusses the experimental setup and
accuracy using a setup where training and prediction are Section IV presents results. Finally, Section V concludes the
both performed at program phase granularity. Such approaches paper with a summary of key contributions and results.

978-3-9815370-8-6/17/$31.00 2017
c IEEE 1709
 Training
Sample Alignment
App. N
App. 1
Host Sampled

.
.
.
App. 1 Machine Performance Feature Learning
. Aligned
g Reference Algorithm
.

.
.
.
. Performance
Performance
App. N Reference Sampled Reference
Performance

.
.
.
Target Model

Fig. 2: Training sample alignment formulation.

Prediction
Sampled Predicted
Performance Feature Performance
cross-platform prediction model is learned from them. Finally,
Predictive
.
Model during the prediction phase, an arbitrary binary is executed and
App. Host .
New Machine . sampled periodically on the host. Sampled features are then
fed into the model to predict performance on the target.
Fig. 1: Sampling-based cross-platform prediction framework. II. T RAINING AND P REDICTION
In this section, we describe sample alignment employed in
A. Related Work the training phase and formulate the regression model used
Simulation-based approaches such as cycle-accurate or for performance prediction. We demonstrate that the sample
cycle-approximate ISSs [3], [4] have been proposed and alignment problem is NP-hard, which motivates the search
widely adapted for performance estimation. However, through- for computationally efficient heuristics. Moreover, we propose
put of most ISSs is on the order of several hundred KIPS one such heuristic, a linear-time approximation scheme that
to several MIPS. Recently, source-level host-compiled and we refer to as stochastic dynamic coupling, and show it is
transaction level modeling (TLM) techniques [5] have been guaranteed to converge under mild and realistic assumptions.
proposed for improving simulation speed while maintaining A. The Sample Alignment Problem
accuracy close to an ISS, but they often require tedious back- During training, we choose one representative feature on
annotation of source code with static or dynamic performance host and target to be used for sample alignment. Let α ∈ Rn+
estimates, which makes them source dependent and inherently and β ∈ Rm + denote positive vectors whose entries are samples
limits speed and accuracy. of the time series of chosen base measurements acquired on a
Many analytical models based upon statistical methods have host and target, respectively, that need to be aligned. Without a
been proposed [6]–[8]. However, all of these approaches try loss of generality, assume that n < m; this assumption holds
to obtain performance models for some target architecture for the rest of the paper. The goal of sample alignment is
of interest from measurements performed on the same base to find a binary coupling matrix H ∈ {0, 1}n×m such that
architecture. By contrast, in earlier work [1] we introduced the empirical covariance between the host samples α and the
performance and power prediction by establishing analyti- aligned target samples β  = Hβ is maximized. As illustrated
cal models that correlate two distinct architectures. Ardalani in Figure 2, the coupling matrix H belongs to the space
et al. [9] proposed similar concepts for estimating GPU H of binary matrices characterized by descending staircase
performance from CPU executions. These approaches were pattern of consecutive ones and non-overlapping support of
limited to predicting performance of whole programs only, rows. Since the ith row of H specifies how consecutive target
where errors of more than 50% were shown. Phase-based ap- samples in β need to be aggregated to form the ith element
proaches [2] were later proposed to support estimation of both of the aligned vector β  , every element in the original vector
performance and power at finer temporal granularities with β must correspond to only one element of α. Formally, the
more than 95% accuracy when targeting complex benchmarks alignment problem can be expressed as the optimization
and architectures. However, all these instrumentation-based 1
approaches are limited to prediction of single applications for maximize cov(α, β  ) = (α − E[α])T (β  − E[β  ]). (II.1)
H∈H n n
which source code is available. Furthermore, instrumentation
comes with non-negligible execution overhead. mα and β are normalized 1 so
If nthat i=1 αi 
1
= 1 and
n
j=1 β j = 1, then E[α] = ( n i=1 α i , . . ., n i=1 αi )
T
B. Overview 1 1 T
= ( n , . . . , n ) and  similarly, due to the structure of
An overview of our approach is shown in Fig. 1. Like H,  E[β  ] = ( 1 n 1 n
i=1 (Hβ)i , . . . , n i=1 (Hβ)i )
T
n =
in previous instrumentation-based cross-platform prediction 1 m 1 m 1 1 T
( n j=1 βj , . . . , n j=1 βj ) = ( n , . . . , n ) . Therefore,
T
approaches, our sampling-based approach consists of a training expectations E[α] and E[β  ] are constants that do not depend
and a prediction phase. During training, a set of representative on matrix H and the original optimization problem (II.1) can
programs is first executed on both the host and the target hence be simplified as
to obtain a training set. The target reference can be either
maximize αT Hβ. (II.2)
a simulator or real hardware. During the execution of each H∈H
training program, instead of instrumenting profiling calls at Due to the combinatorially high number of potential solutions
the phase level to measure various hardware performance to the alignment problem, optimization (II.2) is NP-hard and
features on the host and reference performance on the target, thus finding its exact solution for large n and m is generally
we sample them periodically. Samples are then aligned and a intractable. As an alternative, we propose a practically feasible

1710 2017 Design, Automation and Test in Europe (DATE)

Algorithm 1 Stochastic Dynamic Coupling Since sampling periods are finite, the total number of hardware
Input: α ∈ Rn+ , β ∈ Rm +;
performance events in each period is bounded. Moreover,
Output: H, total error if we assume application runtime is finite, the number of
1: Initialize Hij = 0 ∀i, j and total error = 0; performance events in each sampling period decreases with
2: Normalize α = nα α , β = m β ;
β sampling frequency. The ratio λ between the total number of
i=1 i j=1 j
3: k = 1; samples on two platforms may therefore serve as a proxy for
4: for i = 1, . . . , n do the ratio of sampling frequencies.
5: residual = αi ; Theorem 1. Under the boundedness Assumption 1, the total
6: for j = k, . . . , m do n
residual error R = i=1 ri of the SDC procedure satisfies
7: if |residual| > |residual − βj | then nC λ2  2
8: residual = residual - βj ; P(R ≥ + ) ≤ exp(− 3 2 ) (II.3)
2λ 2n C
9: Hij = 1; for all  ≥ 0.
10: else
11: ri = |residual|; Intuitively, Theorem 1 states that as the relative sampling
12: total error = total error + ri rate λ increases, the probability of having large total residual
13: k = j; error decreases exponentially with λ2 . Asymptotically, as λ →
14: break; ∞, (II.3) becomes
nC λ2  2
15: end if lim P(R ≥ + ) ≤ lim exp(− 3 2 )
16: end for λ→∞ 2λ λ→∞ 2n C
p
17: end for ⇒ lim P(R ≥ ) → 0 ⇒ R → 0,
λ→∞
18: return H, total error and the total residual SDC error converges in probability to 0.
greedy strategy with time complexity O(max(m, n)), hence Proof. It follows from Assumption 1 that βj ∈ [0, Cλ ]. The
only linear in m and n, and show that its solution converges SDC procedure guarantees that the residual error satisfies ri ∈
asymptotically in probability to the optimal alignment. [0, 2λ
C
]. This is straightforward to prove by a contradiction
B. Stochastic Dynamic Coupling argument. Assume the contrary, which means that at some
The stochastic dynamic coupling (SDC) procedure is for- iteration i of the inner loop (line 7 in Algorithm 1) SDC in
malized as Algorithm 1. SDC starts with the first component the else branch (line 11 in Algorithm 1) assigns ri > 2λ C
.
of α and aligns to it multiple consecutive components (starting However, since all the elements of β are bounded above by
λ , it follows that |ri − βj | ≤ | 2λ − λ | ≤ 2λ ≤ ri , which
C C C C
from the first one) of β; specifically, SDC proceeds to aggre-
gate components of β and align them to α1 for as long as means the condition for the previous if branch (Algorithm 1,
the absolute value of the alignment residual keeps decreasing line 8) is satisfied and the algorithm could not have entered
(line 7 to line 16 in Algorithm 1). The residual represents the the else branch (line 11), hence a contradiction. n
remainder of α1 that has not yet been accounted for by the sum Define a random variable Zi = E[ k=1 rk |r1 , . . . , ri ] and
of aligned β components. Once no more β components can note that the random sequence {Zi : i = 0, . . . , n} forms a
be aligned, SDC moves to α2 and aligns to it the next subset martingale with respect to the sequence r1 , r2 , . . . , rn . From
of consecutive β components, and so on. The SDC procedure the definition of a discrete-time n martingale [10] it follows that
keeps track of the total residual error during the execution E[Z ni+1 |r 1 , . . . ri ] = E[E[ k=1 rk |r1 , . . . , ri+1 ]|r1 , . . . ri ] =
(line 12) as a quality measure of the overall alignment. The E[ k=1 rk |r1 , . . . , ri ] = Zi ∀i = 0, . . . , n, where we ap-
total residual error accumulates the leftover αi residuals that plied the smoothing property of conditional expectation since
are unaccounted for during each iteration of the outer for loop G(r1 , . . . , ri ) ⊆ G(r1 , . . . , ri+1 ). Here, G(S) denotes the σ-
(line 4). Ideally, we want the total residual error to be as close algebra generated by set S. Such a construction is also known
to zero as possible. In the next section we show that the total as a Doob backward martingale [10].
residual error from SDC indeed converges in probability to Now, since ri ∈ [0, 2λ C
], the martingalen difference can be
zero asymptotically with sampling parameters. bounded
n as follows: |Z i − Zi−1 | = |E[n k=1 rk |r1 , . . . , ri ] −
C. Optimality of SDC E[ k=1 rk |r1 , . . . , ri−1 ]| ≤ |E[ k=1 rk |r1 , . . . , ri ]| +
|E[ k=1 rk |r1 , . . . , ri−1 ]| ≤ 2nC
n
Before arguing asymptotic optimality of SDC, we impose 2λ = nC λ , where we used
the following boundedness assumption on vector β. the triangle inequality and the fact that both Zi and Zi−1
are bounded above by nC 2λ . Therefore, sequence {Zi } is a
Assumption 1. There exist a constant C > 0 such that martingale with bounded increments and hence the Azuma-
β∞ ∈ [0, Cλ ], where λ = m n characterizes the relative Heoffding inequality [10] implies that, for all  ≥ 0,
sampling rate between the two input vectors α and β.  n  n
 2 λ2
To see that Assumption 1 is intuitive and realistic, note P(Zn −Z0 ≥ ) = P( rk −E[ rk ] ≥ ) ≤ exp(− ).
2n(nC)2
that in the alignment problem each component of vector β k=1 k=1
n
corresponds to the total number of hardware performance From E[rk ] ≤ 2λ C
it follows E[ k=1 rk ] ≤ nC 2λ . By rearrang-
events of a certain type that occur during a sampling period. ing both sides of the inequality we arrive at (II.3).

2017 Design, Automation and Test in Europe (DATE) 1711

 The martingale analysis outlined above applies to the most TABLE I: Performance counters profiled on the host.
general case where no independence assumption is imposed on Total Cycles Total Instructions
Total Cache References Total Cache Misses
the random variables ri . With certain independence assump- Total Branches Total Branch Misses
tion on ri , we can achieve significantly tighter bounds on the
concentration behavior of R. In what follows, we describe a optimization problem,
special case which yield better concentration results. 1
minimize Xwt̂ − Y  22 + νwt̂ 22
wt̂ 2m (II.4)
Remark 1. (Independence) Under independence assumptions
on random variables ri , the Heoffding inequality [10] for the subject to wt̂ ≥ 0.
sum of independent sub-Gaussian random variables yields The solution wt̂ is then used for prediction as parameter for
the local linear approximation F  )=
 at x̂ (i.e, F(x̂ ) ≈ F(x̂
nC 8λ2 2 t̂ t̂ t̂
P(R ≥ + ) ≤ exp(− ) wt̂ x̂t̂ ). Notice that the optimization problem in (II.4) does not
T
2λ nC 2
for all  ≥ 0. Here we used the fact that ri ∈ [0, 2λ
C
] implies ri have an analytical solution. However, as the objective function
is a sub-Gaussian random variable with parameter σ = 4λ C
. is both smooth and convex, the solution can be computed
efficiently via standard gradiant decent methods [11].
Note that in this case, the multiplicative constant in the In the CLLR, we need to choose two tuning parameters, the
8
exponential term ( nC 2 ) is larger than in the general case regularization penalty ν and the parameter μ for controlling
1
( 2n3 C 2 ), which implies faster convergence of the total residual the size of the neighborhood to explore. We iteratively employ
error R to nC 2λ . In practice, we may not be able to drive a standard cross-validation technique [12] over the training set
parameter λ to extremely large values, which is the setting on to empirically determine their optimal values.
which asymptotic results are predicated. We discuss in Section
III. P REDICTION S ETUP
IV practical aspects of tuning parameter λ by sampling one
platform at a faster rate than the other. We have implemented a prediction setup using the Linux
Perf tool infrastructure v3.16 for timer interrupt based counter
D. Prediction sampling, and Python v2.7.3 and SciPy v0.15.1 as the main
We use sampled feature vectors consisting of selected computation environment. We collect a total of 6 hardware
hardware performance counters from the host for prediction. performance counters, shown in Table I, on an Intel Core i7-
During training, we sample host features and target reference 920 machine with 24GB of memory used as the host. Note
performance for each workload in the training set, and perform that host hardware limits simultaneously collectable counters.
SDC alignment using selected host and target measurements We use fewer features than prior work [1], [2], which relied
to determine the coupling matrix H. Similar to existing cross- on expensive, repeated profiling of each program to collect 14
platform prediction approaches, we then apply a constrained counters in total. We sample at 5 different periods ranging
locally linear regression (CLLR) on the the aligned host from 100ms to 1000ms on the host to study the effect of
feature vectors and the target performance to obtain a local sampling period on the prediction accuracy and speed. On the
LASSO-like prediction model specific to each period. target, we sample only cycles and instructions as reference,
To formalize the prediction procedure, for each sample t let where we choose the sampling period to be 100ms as the
xt ∈ Rd denote the performance feature vector obtained from minimum allowed by the Perf tool.
the host and yt ∈ R the performance on the target. The goal of We use the total number of host and target instructions in
prediction is to extrapolate a mapping F : Rd → R such that each sampling period as α and β base features to perform
for all t, F(xt ) ≈ yt . Results in [1] suggest that performance alignment in our setup. Unlike other performance features
features on one platform and timing on another follow a non- that are both microarchitecture and instruction set architecture
linear relationship. Instead of assuming that F is globally (ISA) dependent, the number of instructions only depends on
linear, one can only impose a differentiability assumption, i.e. the ISA. We otherwise obtain sampled training data consisting
that F is differentiable everywhere in its domain. Although F of host counter vectors xi and corresponding reference target
can no longer be expressed explicitly in closed form as in an performance yj . The aligned training data is then used in
ordinary linear regression case, the differentiability assumption prediction by solving the optimization problem in (II.4).
allows approximating F point-wise via a first-order linear Similar to [2], our training set consists of 284 diverse and
approximation, which we denote here as F.  representative sample programs from the ACM International
Formally, given the feature vector x̂t̂ at sample  t̂ to be Collegiate Programming Contest (ICPC) database. We employ
predicted, let {(xl , yl ), l = 1 . . . q} with yl = j Hlj yj be a larger training set than [2] to account for use of fewer
the set of feature vector and H-aligned reference performance counters and coarser sampling. We apply our trained prediction
pairs in the training set that are close to x̂t based upon the setup to a test set consisting of 19 CPU-bound C++ programs
distance criteria, x̂t̂ − xl 2 ≤ μ, where μ is a parameter for (perlbench, bzip2, gcc, mcf, milc, namd, gobmk, dealII, soplex,
determining the size of the local neighborhood of interest. Let povray, hmmer, sjeng, libquantum, h264ref, lbm, omnetpp, as-
X ∈ Rq×d be the matrix that contains all the xTl as its row tar, sphinx3, xalancbmk) from the SPEC 2006 CPU suite with
vectors, and Y  ∈ Rq be the column vector that contains all “ref” input, 13 Java programs (avrora, eclipse, fop, h2, jython,
the yl as its elements. The CLLR then solves the following luindex, lusearch, pdm, sunflow, tomcat, tradebeans, trades-

1712 2017 Design, Automation and Test in Europe (DATE)

 (a) 19 SPEC CPU C/C++ programs. (b) 13 DaCapo Java and 13 Pybench Python programs.
Fig. 3: Predicted target cycles and prediction accuracy of 45 benchmarks (host sampling period T = 500ms).

(a) 19 SPEC CPU C/C++ programs. (b) 13 DaCapo Java and 13 Pybench Python programs.
Fig. 4: Runtime of 45 benchmarks.
oap, xalan) from the DaCapo Java benchmark v9.12 [13] and case prediction error is around 6%, with average errors of less
13 Python programs (2to3, bzr startup, django v3, fastpickle, than 3.5%. Despite using fewer counters and coarser sampling
hg startup, html5lib, json load, nbody, nqueen, regex v8, but with a larger training set, for the C++ benchmarks for
spectral norm, telco, tornado http) from the commercial Uni- which complete source code is available (Fig. 3a), this is
fied Python benchmark suite (PyBench) [14]. The Java and comparable to results reported by existing instrumentation-
Python benchmarks include significant binary-only library, or based, source-level cross-platform prediction techniques [2],
virtual machine performance components. In our current setup, which achieve on average over 97% accuracy.
we are interested in predicting performance for single-core The total runtime of each test program is shown in Fig. 4. It
workloads excluding disturbances due to host/target operating consists of profiling and prediction times. The profiling time is
system variations. Thus, all programs are restricted to run on the time it takes to collect various counters on the host. Our In-
one core till completion, which minimizes measurement noise tel host supports simultaneous reading of 3 counters at a time.
due to core migration. On both the host and target, the Java Since the Linux Perf tool allows for over-sampled counter
programs are executed with the OpenJDK Runtime v2.6.7, and multiplexing, only one run of each program is necessary to
the Python test programs with Python v2.7.3. collect all 6 counters with little overhead. Compared to prior
To demonstrate the effectiveness of our approach on state- work [2], which incurs instrumentation overhead at every basic
of-the-art, real-world mobile target platforms, we employ a block, samples at faster rates, and requires 5 separate runs of
physical hardware reference as target for training and predic- each program to collect 14 counters, profiling in our approach
tion. We specifically use the ODROID-U3 development board is significantly faster while achieving similar accuracies. The
with a quad-core ARM Cortex-A9 based Samsung Exynos prediction time measures the total duration of solving the
4412 SoC as our target platform for experiments. optimization problem (II.4) for all the samples obtained from
the program. Solving time is governed by the dimension of
IV. E XPERIMENTAL R ESULTS the data matrix X and the neighborhood defined by distance
To demonstrate the accuracy of the proposed approach, we threshold μ in II.4. Prior work [2] uses a convex but non-
apply our sampling-based prediction framework to the 45 test smooth objective function. By contrast, our objective function
programs on the Intel host in order to predict performance of is both convex and smooth, which results in faster solving
each benchmark on the ARM target. We compare predictions speed overall. Combined, shorter profiling and prediction times
against actual measurements obtained from the U3 hardware. significantly increase the speed of our approach as compared to
Fig. 3 shows the accuracy of predicting whole program perfor- prior work [1], [2]. Overall simulation speed of our approach
mance with a host sampling period of T = 500ms. Predicted is on the order of 3 giga target instructions per second (GIPS).
cycles are very close to actual hardware measurements. Worst- Related work runs at 500-800 MIPS counting instrumented

2017 Design, Automation and Test in Europe (DATE) 1713

 Fig. 6: Convergence behavior of total Fig. 7: Alignment example
Fig. 5: Speed and accuracy tradeoff. residual in training. (447.dealII-train).
instructions, with a peak of around 1 GIPS excluding any V. S UMMARY AND C ONCLUSIONS
prediction. Furthermore, as the Intel host is significantly more This paper proposes a novel sampling-based approach for
powerful than our ARM target, performance estimation using source-oblivious, binary-level cross-platform performance pre-
our approach is in fact on average twice as fast than running diction. We introduce a stochastic dynamic coupling (SDC)
the test application natively on the target itself. algorithm to address the training sample alignment problem
For the same sampling period T , comparing runtimes of unique to the proposed approach. Although solving such
the SPEC programs (Fig. 4b) with Java and Python programs alignment exactly is NP-hard, we show that our algorithm ap-
(Fig. 4a), we see that as programs execute longer, the number proximates the optimal alignment with high probability under
of samples increases proportionally, which results in more mild boundedness assumptions. Using SDC-aligned samples
time spend in solving the optimization problem (II.4). As for prediction achieves on average over 96.5% accuracy across
sampling periods become smaller, since Perf introduces only a 45 diverse academic and commercial test programs in C/C++,
very small sampling overhead, the profiling time stays roughly Java and Python. With a host sampling period of 500ms,
the same. The prediction time, however, increases due to an the average simulation speed is over 3 GIPS, which, at the
increase in the total number of samples. same accuracy, is up to six times faster than state-of-the-
Fig. 5 shows the prediction accuracy and speed tradeoff with art instrumentation-based cross-platform prediction, and twice
respect to the choice of host sampling period T . For reasons the speed of executing the test applications natively on our
outlined above, as sampling period increases, speed increases ARM A9 target. In the future we plan to extend our approach
proportionally. Prediction accuracy also increases initially until to support both performance and power prediction for more
the sampling period reaches 500ms. Increasing the sampling complex multi-threaded and IO-intensive workloads, as well
period causes the relative sampling ratio λ to also increase, as other architectures, such as GPUs.
which results in a smaller alignment error as discussed in R EFERENCES
Section II-C. This is empirically confirmed in Fig. 6, which [1] X. Zheng et al. Learning-based analytical cross-platform performance
shows the convergence behavior of the total residual R with prediction. In SAMOS, 2015.
respect to λ for the training data. In general, an increase in host [2] X. Zheng et al. Accurate phase-level cross-platform power and perfor-
mance estimation. In DAC, 2016.
sampling rate T results in an increase of the effective λ. As [3] Nathan Binkert et al. The gem5 simulator. SIGARCH Computer
λ increases, the total residual error deviates significantly less Architecture News, 39(2):1–7, 2011.
often from its mean, which in turn roughly scales as O( nC 2λ )
[4] T. E. Carlson et al. Sniper: Exploring the level of abstraction for scalable
and accurate parallel multi-core simulation. In SC, 2011.
and thus converges to zero as λ → ∞. This agrees with our [5] O. Bringmann et al. The next generation of virtual prototyping: Ultra-
analysis of SDC optimality in Section II-C. In other words, fast yet accurate simulation of HW/SW systems. In DATE, 2015.
the training data from the host and target are better aligned at [6] W. L. Bircher et al. Runtime identification of microprocessor energy
saving opportunities. In ISLPED, 2005.
larger sampling periods. However, this trend does not continue [7] B. C. Lee et al. CPR: Composable performance regression for scalable
beyond a sampling period of 500ms. When the host sampling multiprocessor models. In MICRO, 2008.
period becomes too large, the total amount of samples obtained [8] J. C. McCullough et al. Evaluating the effectiveness of model-based
power characterization. In USENIX, 2011.
during training decreases, and prediction accuracy suffers due [9] N. Ardalani et al. Cross-architecture performance prediction (XAPP)
to a lack of training data. using CPU code to predict GPU performance. In MICRO, 2015.
Finally, Fig. 7 illustrates SDC alignment during training [10] S. Boucheron et al. Concentration Inequalities: A Nonasymptotic Theory
of Independence. Oxford university press, 2013.
using SPEC program 447.dealII with “train” input as an [11] A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding
example. As mentioned before, we perform SDC alignment on algorithm for linear inverse problems. SIAM Journal on Imaging
the total number of instructions due to their microarchitecture Sciences, 2(1):183–202, 2009.
[12] R. Kohavi. A study of cross-validation and bootstrap for accuracy
independence. The aligned samples measured on the ARM estimation and model selection. In IJCAI, 1995.
target couple closely with the reference samples obtained on [13] S. M. Blackburn et al. The DaCapo benchmarks: Java benchmarking
the Intel host. Even when the underlying application exhibits development and analysis. In OOPSLA, 2006.
[14] The Grand Unified Python Benchmark Suite. https://www.openhub.net/
drastic temporal variations, SDC is able to accurately track and p/python-benchmarks.
map the application’s behavior from the target to the host.

1714 2017 Design, Automation and Test in Europe (DATE)