Towards a Green Ranking for Programming Languages ∗

Marco Couto, Rui Pereira Francisco Ribeiro, Rui Rua João Saraiva
HASLab/INESC TEC HASLab/INESC TEC HASLab/INESC TEC
Universidade do Minho Universidade do Minho Universidade do Minho
Portugal Portugal Portugal
[email protected], {fribeiro,rrua}@di.uminho.pt [email protected]
[email protected]

ABSTRACT and software developers. In this century, however, the focus on

While in the past the primary goal to optimize software was the computer performance is changing: program’s execution time is
run time optimization, nowadays there is a growing awareness of no longer the only concern in terms of computer performance. The
the need to reduce energy consumption. Additionally, a growing quick adoption of non-wired computer devices in recent years is
number of developers wish to become more energy-aware when making energy consumption one of the main bottlenecks when
programming and feel a lack of tools and the knowledge to do so. building computers and developing their software.
In this paper we define a ranking of energy efficiency in pro- While computer manufacturers began developing energy effi-
gramming languages. We consider a set of computing problems cient hardware since the very beginning of the world wide adoption
implemented in ten well-known programming languages, and mon- of mobile computing devices, only recently has the software engi-
itored the energy consumed when executing each language. Our neering community started to become concerned about the energy
preliminary results show that although the fastest languages tend efficiency of software systems [23]. Today, this topic has become
to be the lowest consuming ones, there are other interesting cases an intensive area of research, where techniques to monitor energy
where slower languages are more energy efficient than faster ones. consumption by programs have been defined, such as the use of
consumption models in mobile devices [34], the use of estimations
CCS CONCEPTS as provided by the Intel RAPL [6, 8, 18], the Qualcomm TrepN
frameworks 1 , or even the use of external energy measurement de-
• Software and its engineering → Software performance; Gen-
vices [2, 19]. Using such monitoring frameworks, several techniques
eral programming languages;
have been proposed to reason about energy consumption in pro-
KEYWORDS gramming languages, for example, analyzing the energy efficiency
of Java and Haskell data structures [16, 22, 25], analyzing how differ-
Energy Efficiency, Programming Languages, Language Benchmark- ent programming coding practices influence energy consumption
ing, Green Software [28], studying the impact of testing techniques in software energy
ACM Reference format: consumption [14], etc.
Marco Couto, Rui Pereira, Francisco Ribeiro, Rui Rua, and João Saraiva. 2017. An interesting question that frequently arises in the area of
Towards a Green Ranking for Programming Languages . In Proceedings of software energy efficiency is whether a faster program is also an
SBLP 2017, Fortaleza, CE, Brazil, September 21–22, 2017, 8 pages. energy efficient program, or not. Indeed, there are research papers
https://doi.org/10.1145/3125374.3125382
pointing in completely different directions, for example [1, 16, 22,
24, 31, 33].
1 INTRODUCTION This question also arises when comparing the performance of dif-
The performance of computers was a main concern since their ferent programming languages. However, comparing programming
creation in the previous century. Today, this is no exception and languages performance is an extremely complex task. Programs
performance is still the main goal for both computer manufacturers written in different languages which implement exactly the same
∗ This
computing problem may use different algorithms. Moreover, the
work is financed by the ERDF – European Regional Development
reused language libraries, the quality of the compiler, and its (ag-
Fund through the Operational Programme for Competitiveness and Interna-
gressive) optimizations all greatly influence the performance of the
tionalisation - COMPETE 2020 Programme and by National Funds through
the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnolo- resulting programs. Thus, a programming language may become
gia within project POCI-01-0145-FEDER-016718. The second author is also faster, not by changing its programs, but by "just" improving its li-
sponsored by FCT grant SFRH/BD/112733/2015 braries and/or its compiler (or virtual machine). Nevertheless there
is some work aiming at comparing performance of programming
Permission to make digital or hard copies of all or part of this work for personal or languages, such as the "Computer Language Benchmarks Game"
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation (CLBG) project 2 which compares computer language performance,
on the first page. Copyrights for components of this work owned by others than ACM in terms of execution time and memory consumption. It includes
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
a repository of programs written in different languages which im-
fee. Request permissions from [email protected]. plement solutions to a set of predefined computing problems. This
SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil
1 https://developer.qualcomm.com/software/trepn-power-profiler
© 2017 Association for Computing Machinery.
2 Moreinformation about the CLBG can be found here: http://benchmarksgame.alioth.
ACM ISBN 978-1-4503-5389-2/17/09. . . $15.00
https://doi.org/10.1145/3125374.3125382 debian.org
SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil M. Couto, R. Pereira, F. Ribeiro, R. Rua, J. Saraiva

language benchmark project was developed to provide a runtime • The source code for the different program solutions is
ranking of programming languages. available as open source, and consequently the solutions
In this paper we reuse the computing problems and solutions pro- (and their authors) are known. Thus, it is possible to see
posed in CLBG to define a ranking of energy efficiency in program- that each program follows common programming practices
ming languages. We consider the computing problems of CLBG used by the respective programming language community.
implemented in ten well-known programming languages (C, C#, • The solutions use the same underlying algorithm. There-
Fortran, Go, Java, JRuby, Lua, OCaml, Perl, and Racket) in order to fore they may ignore programming language features that
answer the following questions: Is the fastest language, the most would improve performance using a different algorithm.
energy efficient one? Are there languages which run slower, while • The inputs provided to the programs are known, and all
consuming less energy than others? programs produce the same output for similar inputs (in
We developed a framework using an energy measurement tool other words the programs are correct).
provided by Intel (RAPL), in order to monitor the the consumption • Finally, the compiler version and options to compile/exe-
when executing a program. We have used this monitoring frame- cute the program’s source code in each language is also
work to measure the energy of all executable programs included in specified in the CLBG.
CLBG. Our preliminary results show that the C language is both The considered programs in the CLBG cover different computing
the fastest and greenest language, while also showing interesting problems, as described in Table 1.
cases where slower languages are greener than others: for example,
when considering all the benchmarks, Lua is slower by, roughly Table 1: CLBG corpus of programs used in our study.
12%, than Perl, while consuming 53% less energy!
The remaining of this paper is structured as follows: Section 2 Benchmark Description Input
presents the computer language benchmark game. Section 3 de- Double precision N-body
scribes the methodology we use to monitor the energy consumption n-body 50M
simulation
of the ten languages we consider in our study. We present the en- Indexed access to tiny integer
fannkuch-redux 12
ergy consumption of each program and we discuss the obtained sequence
results. Section 4 details the threats to the validity of our study. In Eigenvalue using the power
spectral norm 5,500
Section 5 we describe related work, and in Section 6 we present the method
conclusions of our work. Generate Mandelbrot set portable
mandelbrot 16,000
bitmap file
2 COMPUTER LANGUAGE BENCHMARK Streaming arbitrary precision
pidigits 10,000
arithmetic
GAME Match DNA 8mers and substitute fasta
regex redux
In order to compare the performance, both in terms of execution magic patterns output
time and energy consumption, of different programming languages, Generate and write random DNA
fasta 25M
we need solutions/programs for the same problems expressed in sequences
each of those languages. Developing programs written in different Hashtable update and k-nucleotide fasta
k nucleotide
programming languages (and programming paradigms) is both a strings output
complex and time consuming task. Thus, in our study we reused reverse Read DNA sequences, write their fasta
complement reverse-complement output
the repository of programs available in the "Computer Language
Allocate, traverse and deallocate
Benchmarks Game3 " (CLBG) project. The project compares pro- binary trees 21
many binary trees
gramming language performance, in terms of runtime and memory
chameneos Symmetrical thread rendezvous
consumption, by implementing a solution to a set of computing 6M
redux requests
problems in different programming languages. Search for solutions to shape
The CLBG includes solutions for thirteen different problems, meteor contest 2,098
packing puzzle
written in twenty eight different programming languages. These Switch from thread to thread
thread ring 50M
benchmarks have been used in several research works, to study passing one token
the dynamic behavior of non-Java JVM languages [15], to analyze
dynamic scripting languages [32] and compiler optimizations [30], Unfortunately, in the current state of the CLBG, not all imple-
to benchmark a JIT compiler for PHP [11], and in the design of the mentations for these thirteen computing problems are available. In
R language [20]. In fact, the repository of programs included in our study we only consider the first ten computing problems, as
CLBG are also particularly suitable to analyze the energy efficiency their implementations are the most commonly available.
of programming languages due to the following characteristics:
• The solutions do represent the state-of-the-art on how to 3 ENERGY EFFICIENCY IN PROGRAMMING
solve the thirteen problems in each of the programming LANGUAGES
language. Indeed, the programs in the repository were In this section, we present our case study for comparing the energy
developed by experts in each of the languages considered. efficiency of different computer languages.
3 The word game in this benchmark means that programmers from different languages We start by describing the followed methodology and the tools
contribute programs that compete (but try to remain comparable) with each others. used to compare the energy efficiency of 10 different programming
Towards a Green Ranking for Programming Languages SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil

languages (Section 3.1). We chose to analyze the 10 following lan- (such as Perl) or languages which require a virtual machine to be
guages: C, C#, Fortran, Go, Java, JRuby, Lua, OCaml, Perl, and Racket. executed (such as Java), the path to the interpreter/virtual machine
We used the program solutions available in the CLBG, implemented is also provided. To execute the fasta benchmark in Perl, the variable
in these 10 languages, as the objects for our case study. Finally, we command would be "/usr/bin/perl fasta.perl 25000000".
present the obtained results for all 10 languages (Section 3.2). Before advancing, we needed to certify that there would be
minimal or no overhead from our measuring framework using the
3.1 Design and Execution system function. To do so, we measured the energy consumption
The CLBG proposes 13 benchmark problems to be solved in various of both a C and Java language program, using RAPL and jRAPL
different languages. Of these 13 we chose the first 10 shown in respectively, and compared the results to the measurements from
Table 1, as explained in the previous section. We gathered the source our C language energy measuring framework. The differences were
code of the most efficient (i.e., fastest) benchmark solutions for the insignificant, consistent, and negligible. Therefore, we may use
10 benchmark problems, for each of our considered 10 programming our C language energy measurement framework without worry of
languages. As the CLBG lists the performance of each proposed overheads or imprecise measurements.
solution, we already knew which ones were the most efficient. In a few cases, we were not able to generate the results for
These benchmark solutions were compiled and executed accord- all of the 10 benchmarks solutions. This was due to one of three
ing to the information provided by the submission’s documentation reasons: (1) there was no available source code for the missing
page. These documentation pages also provided information on the benchmark, (2) the implementation contained errors, or (3) there
compiler versions, and compilation/executions options. were missing libraries which could not be found. Nevertheless,
To obtain energy measurements, we used Intel’s Running Aver- by choosing these 10 languages, we assure that every benchmark
age Power Limit (RAPL) tool [6], which has already been proven to problem is implemented in at least 70% of our chosen languages,
provide very accurate energy measurements [8, 26]. Each bench- and that all programming paradigms are included.
mark was measured 10 times, both energy consumption and execu- We ran these studies on a desktop with the following specifica-
tion time. By following this approach we aim to reduce the impact tions: Linux Ubuntu Server 16.10 operating system, kernel version
of cold starts and cache effects, while also analyzing the consistency 4.8.0-22-generic, with 16GB of RAM, a Haswell Intel(R) Core(TM)
of the measured results to check consistency and avoid outliers. i5-4460 CPU @ 3.20GHz. After measuring each benchmark solution
Since the RAPL tool is currently only usable within the C and Java 10 times, we extracted and treated the CPU energetic consumption
(through jRAPL [18]) languages, we needed to develop a measuring values and carefully analyzed the results to check for consistency
framework to allow the energy measurement of every language. Our and accurate measurements as described in Section 4.
goal was to measure the energy consumed by the whole program,
and not by independent code blocks (such as methods), in order 3.2 Results and Discussion
to properly compare the languages. Therefore, we developed a
This section presents the obtained results in our study, followed by
measuring framework written in the C language which measures
an analysis and discussion of these results. Each table, from Table 2
the execution of an external program, using the system function.
to Table 6, represents the results from a particular benchmark,
The overall process of this framework is described in Listing 1.
where each row represents one of the 10 considered languages. The
... values in each line, from left to right, include the average values
int main () {
... for the Energy consumed (Joules), Time of execution (milliseconds),
for (i = 0 ; i < N ; i ++) {
time_before = getTime (...) ;
and Ratio (ratio between Energy and Time). This ratio can also be
// performs initial energy measurement seen as the average Power consumed in Kilowatts (kW). In these
rapl_before (...) ;
tables, the programming languages are ordered in regards to their
// executes the program energy consumption, from lowest to highest.
system ( command );
Table 7 shows the global results (summation) for Energy and
// computes the difference between Time normalized to the C language. So for example, C# consumes
// this measurement and the initial one
rapl_after (...) ; 2.21x more energy than C, while taking 2.44x more to finish. In this
time_elapsed = getTime (...) - time_before ;
...
table, the left side orders the programming language in regards to
} their energy consumption while the right orders the language in
...
} regards to their execution time.
In some cases, across all tables from Table 2 to Table 7, the
Listing 1: Overall process of the energy measuring framework.
programming language name will have an ↑x or ↓x symbol. This
As we can see, both the execution time and energy are mea- states that the language would go up by x positions (↑x ) or down
sured for every execution of command. This variable holds a string by x positions (↓x ) if ordered by the execution time. For example
describing how to execute a particular benchmark, in a particu- in Table 3, Fortran is the most energy efficient language, but would
lar language. For example, for executing the binary trees bench- fall off 3 positions down if ordered by the execution time.
mark in the C language, the value of the command variable will be To further understand these values, we also generated graphs
"./binarytrees.o 21", where binarytrees.o is the name of the ex- for each of the benchmarks (Figures 1 - 8). Each graph contains the
ecutable file resulting of prior compilation, and 21 is the argument results of each language relative for a benchmark, consisting of a
provided to the binary trees benchmark. For interpreted languages combination of three parts: a bar chart, a line chart and a scatter plot.
SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil M. Couto, R. Pereira, F. Ribeiro, R. Rua, J. Saraiva

Table 2: Results for binary-trees and fannkuch-redux Table 7: Normalized global results for Energy and Time

binary-trees fannkuch-redux Total

Energy Time Ratio Energy Time Ratio Energy Time
C 36.06 1124.67 0.032 C 201.11 6076.28 0.033 C 1.00 1.00 C
Fortran 63.56 2112.17 0.030 Ocaml 258.20 7895.43 0.033 Java 1.68 1.65 Java
Ocaml ↓1 84.63 3525.47 0.024 Java 291.46 8240.84 0.035 Ocaml ↓1 2.13 2.44 C# ↓2
Java ↑1 96.09 3305.65 0.029 Fortran ↓1 295.56 8665.26 0.034 Fortran ↓2 2.20 2.48 Ocaml ↑1
Racket ↓1 115.45 11260.66 0.010 Go ↑1 298.01 8487.00 0.035 C# ↑2 2.21 2.63 Go ↓1
C# ↑1 155.19 10797.15 0.014 C# 373.13 10839.74 0.034 Go ↑1 3.04 3.91 Fortran ↑2
Go 588.14 16291.66 0.036 Racket 1836.00 43680.23 0.042 Racket 7.35 10.29 Racket
Jruby 617.96 19276.14 0.032 Lua ↓2 6732.44 634877.63 0.011 Lua ↓2 39.54 44.68 Jruby ↓1
Lua ↓1 1841.62 209217.00 0.009 Jruby ↑1 7242.76 219148.25 0.033 Jruby ↑1 45.35 68.45 Perl ↓1
Perl ↑1 3276.56 96097.28 0.034 Perl ↑1 10526.19 249357.50 0.042 Perl ↑1 84.89 77.10 Lua ↑2

Table 3: Results for fasta and k-nucleotide

the average time in milliseconds. By combining the two charts we
fasta k-nucleotide can easily understand the relationship between energy and time.
Energy Time Ratio Energy Time Ratio
Fortran ↓3 23.54 1661.43 0.014 C 81.20 2958.04 0.027
Finally, the scatter plot on top of both represents the ratio be-
C ↑1 25.22 973.27 0.026 Java 127.32 4116.17 0.031 tween energy consumed and execution time. This plot allows us
Java ↑1 32.73 1248.89 0.026 C# 189.55 7138.50 0.027
Ocaml ↓2 32.97 3170.54 0.010 Go 247.79 8004.35 0.031 to observe if the relation between energy and time is consistent
Go 35.93 1838.39 0.020 Ocaml 294.12 13847.16 0.021 between languages. A variation in these values indicates that the
C# ↑3 41.47 1549.05 0.027 Fortran ↓1 393.77 41656.29 0.009
Racket 100.64 8254.71 0.012 Racket ↓1 511.60 44238.51 0.012 energy consumed is not directly proportional to the execution time,
Lua 287.82 24616.91 0.012 Lua ↓1 981.73 88074.58 0.011 but also dependent on the language and/or benchmark solution.
Jruby 775.08 49508.94 0.016 Perl ↑3 1227.12 35615.50 0.034
Perl 2535.80 61462.63 0.041 Jruby 2269.92 88523.14 0.026 From the data presented in the aforementioned tables and plots,
we can indeed draw a set of relevant and interesting observations,
Table 4: Results for pidigits and regex-redux and answer our previous questions. For example, a common percep-
tion to the energy consumption problem is that it is a performance
pidigits regex-redux issue, whereby reducing execution time would bring about energy
Energy Time Ratio Energy Time Ratio efficiency. While some works do support that claim [33], the op-
C 22.60 804.82 0.028
C 6.19 546.23 0.011 Ocaml ↓2 133.62 12977.91 0.010 posite has also been observed [16, 24, 31], where an increase in
Racket 8.43 731.89 0.012 Java ↑1 177.79 5693.63 0.031
Go 11.27 772.52 0.015 Perl ↑1 218.66 7163.59 0.031
execution time brings about a decrease in the energy consumed.
C# 11.44 943.64 0.012 Racket ↓2 289.46 26152.36 0.011 We have observed both cases, as do [1, 22].
Perl 16.11 1306.91 0.012 Jruby ↑1 313.13 13477.41 0.023
Jruby 121.26 8198.40 0.015 C# ↑1 485.58 14722.95 0.033 • The results clearly show that the C language is both the
fastest and greenest language. However, in certain cases
Table 5: Results for mandelbrot and n-body there are other more energy efficient solutions.
• From our study, we can see several examples where lan-
mandelbrot n-body guage A consumes more energy to run a benchmark than
Energy Time Ratio Energy Time Ratio
C 32.60 1141.53 0.029 Fortran 41.69 3580.94 0.012
language B, while taking less time to do so. Such an exam-
Java ↓1 105.34 3657.16 0.029 C 49.24 4190.34 0.012 ple can be seen in Figure 1, for instance, where the Java
Go ↑1 107.53 3451.28 0.031 Java 51.03 5839.28 0.009
C# 140.49 3960.59 0.035 Ocaml 51.60 5856.66 0.009 language has higher energy consumption then the OCaml
Fortran ↓1 146.60 8633.04 0.017 Go 52.98 5899.14 0.009 language, yet a slightly lower execution time. The same
Ocaml ↑1 194.56 6863.35 0.028 C# 53.38 6117.06 0.009
Racket 418.33 44516.96 0.009 Racket 227.73 22259.99 0.010 can be observed in the Lua and Perl languages, for exam-
Lua 3392.11 100441.83 0.034 Jruby 1432.98 98407.31 0.015 ple, in Figure 1, 2, and 8. In these cases Lua consumes less
Jruby 6972.23 217158.63 0.032 Lua 3462.55 177251.13 0.020
Perl 16042.30 390013.25 0.041 Perl 3853.19 335390.75 0.011 energy while having a higher execution time.
• Energy consumption is not always directly proportional
Table 6: Results for reverse-complement and spectral-norm to execution time. In almost every graph/table, we can
observe that execution time does not behave in the same
reverse-complement spectral-norm way the energy consumption does in proportion to one
Energy Time Ratio Energy Time Ratio another. For instance, in Table 2 we can see that the C# lan-
C 5.64 227.90 0.030 Fortran 19.37 667.03 0.011
Ocaml 7.58 286.78 0.022 C 20.02 676.51 0.025 guage consumes roughly 4 times more energy to execute
Go 7.91 366.17 0.024 Go 32.16 1331.62 0.022
Fortran ↓2 9.99 937.56 0.029 C# 32.36 1374.42 0.018
the binary-trees benchmark then the C language. How-
C# ↑1 10.68 581.44 0.024 Ocaml ↓1 36.57 1683.41 0.026 ever, in terms of execution time, it takes the C# language
Java ↑1 14.59 628.79 0.022 Java ↑1 37.16 1658.64 0.023
Perl 17.50 1187.04 0.041 Racket 79.31 2341.39 0.012 around 9 times more to finish then the C language.
Racket 25.17 2112.06 0.034 Perl 814.68 19781.86 0.015 • By observing the ratio values (or in other words the av-
Jruby 114.38 4533.48 0.013 Lua ↓1 1019.49 95405.64 0.013
Lua 119.38 9305.06 0.011 Jruby ↑1 1031.88 79449.25 0.025 erage Power in kW), the previous discussion is further
strengthened. While performance optimization would in-
deed (in most cases) reduce the energy consumed, the fact
The bars represent the energy consumed by the languages, with of the matter is we still have the Power variable in the
the left y-axis representing the average Joules. The execution time Energy equation (Joules = Watts x Second). For this to
is represented by the line chart, with the right y-axis representing be purely a performance issue, we would have to assume
Towards a Green Ranking for Programming Languages SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil

Figure 1: Graphical data for binary-trees

Figure 2: Graphical data for fannkuch-redux

Figure 3: Graphical data for fasta

Figure 4: Graphical data for mandelbrot

that the Power is a constant, which we clearly see that is less than half the ratio (average kW) compared to JRuby,
not the case, and may even vary quite heavily in between and in turn half the energy consumed also.
different languages. For example in Figure 8, JRuby and • We can easily see that the three interpreted languages Lua,
Lua have almost identical execution times, while Lua has JRuby, and Perl have a tendency to not only be slower,
SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil M. Couto, R. Pereira, F. Ribeiro, R. Rua, J. Saraiva

Figure 5: Graphical data for n-body

Figure 6: Graphical data for pidigits and regex-redux

Figure 7: Graphical data for spectral-norm

Figure 8: Graphical data for k-nucleotide and reverse-complement

but much more energy consuming (such that, we even had to Perl (Table 7, we can see Lua is slower by roughly 12%
to show the visual data in different scales). Even so, out than Perl, while consuming 53% less energy!
of the three, Lua tends to be the slowest one while also • Looking at Table 7 we have a better view of the overall
being the lowest consuming one. When compared overall results. When comparing all the languages together, we
Towards a Green Ranking for Programming Languages SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil

clearly see that C and Java are the best two, in both en- we set up the study. As the CLBG is an ongoing "competition", we
ergy consumption and execution time. We also observe expect that more advanced and more efficient solutions will substi-
languages such as OCaml, Fortran, and Lua as being much tute the ones we obtained as time goes on, and even the languages’
more energy efficient than they are performance efficient, compilers might evolve. Thus this, along with measurements in
with the opposite observed for C#, Go, Jruby, and Perl. different systems, might produce slightly different resulting val-
ues if replicated. Nevertheless, unless there is a huge leap within
4 THREATS TO VALIDITY the language, the comparisons might not greatly differ. The actual
The goal of our study was to both measure and understand the approach and methodology we used also favors easy replications.
energetic behavior of several programming languages, allowing us This can be attributed to the CLBG containing most of the impor-
to bring about a greater insight on how certain languages compare tant information needed to run the experiments, these being: the
to each other in terms of energy consumption. We present in this source code, compiler version, and compilation/execution options.
subsection some threats to the validity of our study, divided into Thus we believe these results can be further generalized, and other
four categories [4], namely: conclusion validity, internal validity, researchers can replicate our methodology for future work.
construct validity, and external validity.
5 RELATED WORK
Conclusion Validity. From our experiment it is clear that different
programming paradigms and even languages within the same par- For years, the main reference when it comes to classifying a pro-
adigm have a completely different impact on energy consumption. gram’s performance and efficiency has been its execution time.
We also see interesting cases where the most energy efficient is not Indeed, there is a long lasting series of engineering techniques
the fastest, and believe these results are useful for programmers. provided to software developers which aim at helping them build
Nevertheless, the observed energy consumption is only attributed correct programs, while also indicating how to reduce the execu-
to CPU usage, and while we have shown that energy and perfor- tion time [30, 32]. With the improvements on hardware devices,
mance are sometimes related in non-predictable ways, the impact of programs became more complex and robust, and another perfo-
other hardware components (such as memory usage and its energy mance aspect became relevant: the memory usage. Soon, software
impact) deserve further analysis. developers were supplied with tools and techniques for analyzing
the memory usage of their programs [3].
Internal Validity. This category concerns itself with what factors With the emergence of different languages, platforms and pro-
may interfere with the results of our study. When measuring the en- gramming paradigms, the interest in analyzing performance aspects
ergy consumption of the various different programming languages, in software has indeed increased, and execution time is no longer
other factors alongside the different implementations and actual the only relevant performance aspect. Software related energy con-
languages themselves may contribute to variations. To avoid this, sumption has been gaining an increasing interest in recent years as
we executed every language and benchmark solution equally. In well, especially among researchers. In fact, this interest is being ex-
each, we measured the energy consumption and execution time tended to software developers [23], who have been questioning how
10 times, removed the furthest outliers, and calculated the median, to optimize energy consumption through software improvements.
mean, standard deviation, min, and max values. This allowed us Several studies have been aimed at understanding how the en-
to minimize the particular states of the tested machine, including ergy consumption of software systems can be influenced by develop-
uncontrollable system processes and software. However, the results ment aspects. In fact, it is already known that several factors, such as
measured are quite consistent, and thus reliable. In addition, the the use of different design patterns/coding practices [16, 17, 27, 28],
used measurement tool has also been proven to be very accurate. different data structures [10, 16, 18, 22], or even code obfusca-
tion [29] can significatively influence the energy consumed by soft-
Construct Validity. We analyzed 10 different programming lan- ware. This awareness of energy consumption is notorious within
guages, each with roughly 10 solutions to the proposed problems, the software testing area, where some works aim at reducing the
totaling out to almost 100 different cases. These solutions were overall consumption in the testing phase, by reducing the number
developed by experts in each of the programming languages, with of tests while maintaining the code coverage [12, 14].
the main goal of "winning" by producing the best solution for per- Other energy related research works offer a more thorough en-
formance time. While the different languages contain different ergy consumption analysis of programs. While some have been fo-
implementations, all produced the same exact output and each are cused on offering techniques able to identify, within a program, code
implemented to be the fastest and most efficient as possible. Having blocks responsible for excessive energy consumption [5, 13, 21],
these different yet efficient solutions for the same scenarios allows others try to predict the energy consumption of a program by stati-
us to compare the different programming languages in a quite just cally analyzing it [7, 9]. The result of the latter studies is an estimate
manner as they were all placed against the same problem. Albeit of the energy consumed by the program in a specific scenario.
certain paradigms or languages could have an advantage for certain Regardless of the interest in the energy consumption area, there
problems, while others may be implemented in a not so traditional is no study, to the best of our knowledge, which compares the en-
sense. Nevertheless, where is no basis to suspect that these projects ergy consumption behavior of different programming languages.
are best or worst than any other kind we could of have used. With this study, we aim at creating the basis for exploring the
External Validity. We concern ourselves with the generalization subject in more detail, by providing evidence that the energy con-
of the results. The obtained solutions were the best ones at the time sumption behaves differently depending not only on the program
SBLP 2017, September 21–22, 2017, Fortaleza, CE, Brazil M. Couto, R. Pereira, F. Ribeiro, R. Rua, J. Saraiva

but also on the programming language, while also having different [14] Ding Li, Yuchen Jin, Cagri Sahin, James Clause, and William GJ Halfond. 2014.
behaviors in comparison to its execution time analysis. Integrated energy-directed test suite optimization. In Proc. of the 2014 Int. Sym-
posium on Software Testing and Analysis. ACM, 339–350.
[15] Wing Hang Li, David R. White, and Jeremy Singer. 2013. JVM-hosted Languages:
They Talk the Talk, but Do They Walk the Walk?. In Proc. of the 2013 Int. Conf. on
6 CONCLUSIONS AND FUTURE WORK Principles and Practices of Programming on the Java Platform: Virtual Machines,
Languages, and Tools (PPPJ ’13). ACM, 101–112.
In this paper, we described a study on the energy efficiency of ten [16] Luís Gabriel Lima, Gilberto Melfe, Francisco Soares-Neto, Paulo Lieuthier,
different programming languages, considering a corpus of ten com- João Paulo Fernandes, and Fernando Castor. 2016. Haskell in Green Land: Ana-
puting problems included in the CLBG project. We monitored the lyzing the Energy Behavior of a Purely Functional Language. In Proc. of the 23rd
IEEE Int. Conf. on Software Analysis, Evolution, and Reengineering (SANER’2016).
energy consumed and execution time of almost hundred computer IEEE, 517–528.
programs. [17] Mario Linares-Vásquez, Gabriele Bavota, Carlos Bernal-Cárdenas, Rocco Oliveto,
Our results show that compiled languages are, as we expected, Massimiliano Di Penta, and Denys Poshyvanyk. 2014. Mining energy-greedy API
usage patterns in Android apps: an empirical study. In Proc. of the 11th Working
both more energy efficient and faster than interpreted ones, while Conf. on Mining Software Repositories. ACM, 2–11.
also showing that C is the fastest and greenest language in our [18] Kenan Liu, Gustavo Pinto, and Yu David Liu. 2015. Data-oriented characterization
of application-level energy optimization. In Fundamental Approaches to Software
study. However, the results also show that energy consumption is Engineering. Springer, 316–331.
not always directly proportional to execution time. In fact, there are [19] Dustin McIntire, Thanos Stathopoulos, Sasank Reddy, Thomas Schmidt, and
greener programming languages, while being slower than others. William J. Kaiser. 2012. Energy-Efficient Sensing with the Low Power, Energy
Aware Processing (LEAP) Architecture. ACM Trans. Embed. Comput. Syst. 11, 2,
In this study we only monitored and related the CPU energy Article 27 (July 2012), 36 pages.
consumption to execution time. As future work we plan to study [20] Floréal Morandat, Brandon Hill, Leo Osvald, and Jan Vitek. 2012. Evaluating
the impact of memory consumption on the energy consumption, the Design of the R Language: Objects and Functions for Data Analysis. In
Proc. of the 26th European Conf. on Object-Oriented Programming (ECOOP’12).
namely in terms of the energy consumed by the computer’s RAM Springer-Verlag, 104–131.
and hard drive. We also plan to extend our study with more pro- [21] Rui Pereira, Tiago Carção, Marco Couto, Jácome Cunha, João Paulo Fernandes,
and João Saraiva. 2017. Helping Programmers Improve the Energy Efficiency
gramming languages, namely, languages using different execution of Source Code. In Proceedings of the 39th International Conference on Software
models (for example, lazy evaluated languages). Engineering Companion (ICSE-C ’17). 238–240.
[22] Rui Pereira, Marco Couto, João Saraiva, Jácome Cunha, and João Paulo Fernan-
des. 2016. The Influence of the Java Collection Framework on Overall Energy
REFERENCES Consumption. In Proc. of the 5th Int. Workshop on Green and Sustainable Software
(GREENS ’16). ACM, 15–21.
[1] Sarah Abdulsalam, Ziliang Zong, Qijun Gu, and Meikang Qiu. 2015. Using the [23] Gustavo Pinto, Fernando Castor, and Yu David Liu. 2014. Mining questions
Greenup, Powerup, and Speedup metrics to evaluate software energy efficiency. about software energy consumption. In Proc. of the 11th Working Conf. on Mining
In Proc. of the 6th Int. Green and Sustainable Computing Conf. IEEE, 1–8. Software Repositories. ACM, 22–31.
[2] Tarsila Bessa, Pedro Quintão, Michael Frank, and Fernando Magno Quin- [24] Gustavo Pinto, Fernando Castor, and Yu David Liu. 2014. Understanding energy
tão Pereira. 2016. JetsonLeap: A Framework to Measure Energy-Aware Code behaviors of thread management constructs. In Proc. of the 2014 ACM Int. Conf. on
Optimizations in Embedded and Heterogeneous Systems. In Proc. of the 20th Object Oriented Programming Systems Languages & Applications. ACM, 345–360.
Brazilian Symposium on Programming Languages, Fernando Castor and Yu David [25] G. Pinto, K. Liu, F. Castor, and Y. D. Liu. 2016. A Comprehensive Study on
Liu (Eds.). Springer Int. Publishing, 16–30. the Energy Efficiency of Java’s Thread-Safe Collections. (Oct 2016), 20–31.
[3] Wei-Ngan Chin, Huu Hai Nguyen, Shengchao Qin, and Martin Rinard. 2005. https://doi.org/10.1109/ICSME.2016.34
Memory Usage Verification for OO Programs. In Static Analysis: 12th Int. Sympo- [26] Efraim Rotem, Alon Naveh, Avinash Ananthakrishnan, Eliezer Weissmann, and
sium, SAS 2005, London, UK, September 7-9, 2005. Proceedings, Chris Hankin and Doron Rajwan. 2012. Power-Management Architecture of the Intel Microarchi-
Igor Siveroni (Eds.). Springer Berlin Heidelberg, 70–86. tecture Code-Named Sandy Bridge. IEEE Micro 32, 2 (2012), 20–27.
[4] Thomas D Cook and Donald T Campbell. 1979. Quasi-experimentation: design & [27] Cagri Sahin, Furkan Cayci, Irene Lizeth Manotas Gutierrez, James Clause, Fouad
analysis issues for field settings. Houghton Mifflin. Kiamilev, Lori Pollock, and Kristina Winbladh. 2012. Initial explorations on
[5] Marco Couto, Tiago Carção, Jácome Cunha, João Paulo Fernandes, and João design pattern energy usage. In Green and Sustainable Software (GREENS), 2012
Saraiva. 2014. Detecting Anomalous Energy Consumption in Android Applica- First Int. Workshop on. IEEE, 55–61.
tions. In Programming Languages: 18th Brazilian Symposium, SBLP 2014, Maceio, [28] Cagri Sahin, Lori Pollock, and James Clause. 2014. How do code refactorings
Brazil, October 2-3, 2014. Proceedings, Fernando Magno Quintão Pereira (Ed.). affect energy usage?. In Proc. of the 8th ACM/IEEE Int. Symposium on Empirical
Springer Int. Publishing, 77–91. Software Engineering and Measurement. ACM, 36.
[6] Martin Dimitrov, Carl Strickland, Seung-Woo Kim, Karthik Kumar, and Kshitij [29] Cagri Sahin, Philip Tornquist, Ryan McKenna, Zachary Pearson, and James
Doshi. 2015. Intel® Power Governor. https://software.intel.com/en-us/articles/ Clause. 2014. How Does Code Obfuscation Impact Energy Usage?. In Software
intel-power-governor. (2015). Accessed: 2015-10-12. Maintenance (ICSM), 2013 29th IEEE Int. Conf. on. IEEE.
[7] Neville Grech, Kyriakos Georgiou, James Pallister, Steve Kerrison, Jeremy Morse, [30] Vincent St-Amour, Sam Tobin-Hochstadt, and Matthias Felleisen. 2012. Opti-
and Kerstin Eder. 2015. Static Analysis of Energy Consumption for LLVM mization Coaching: Optimizers Learn to Communicate with Programmers. In
IR Programs. In Proc. of the 18th Int. Workshop on Software and Compilers for Proc. of the ACM Int. Conf. on Object Oriented Programming Systems Languages
Embedded Systems (SCOPES ’15). ACM, 12–21. and Applications (OOPSLA ’12). ACM, 163–178.
[8] Marcus Hähnel, Björn Döbel, Marcus Völp, and Hermann Härtig. 2012. Measuring [31] Anne E Trefethen and Jeyarajan Thiyagalingam. 2013. Energy-aware software:
energy consumption for short code paths using RAPL. SIGMETRICS Performance Challenges, opportunities and strategies. Journal of Computational Science 4, 6
Evaluation Review 40, 3 (2012), 13–17. (2013), 444–449.
[9] Shuai Hao, Ding Li, William G. J. Halfond, and Ramesh Govindan. 2013. Estimat- [32] Kevin Williams, Jason McCandless, and David Gregg. 2010. Dynamic Interpreta-
ing Mobile Application Energy Consumption Using Program Analysis. In Proc. tion for Dynamic Scripting Languages. In Proc. of the 8th Annual IEEE/ACM Int.
of the 2013 Int. Conf. on Software Engineering (ICSE ’13). IEEE Press, 92–101. Symposium on Code Generation and Optimization (CGO ’10). ACM, 278–287.
[10] Samir Hasan, Zachary King, Munawar Hafiz, Mohammed Sayagh, Bram Adams, [33] Tomofumi Yuki and Sanjay Rajopadhye. 2014. Folklore confirmed: Compiling for
and Abram Hindle. 2016. Energy profiles of java collections classes. In Proc. of speed= compiling for energy. In Languages and Compilers for Parallel Computing.
the 38th Int. Conf. on Software Engineering. ACM, 225–236. Springer, 169–184.
[11] Andrei Homescu and Alex Şuhan. 2011. HappyJIT: A Tracing JIT Compiler for [34] Lide Zhang, Birjodh Tiwana, Zhiyun Qian, Zhaoguang Wang, Robert P. Dick,
PHP. SIGPLAN Not. 47, 2 (Oct. 2011), 25–36. Zhuoqing Morley Mao, and Lei Yang. Accurate online power estimation and
[12] Reyhaneh Jabbarvand, Alireza Sadeghi, Hamid Bagheri, and Sam Malek. 2016. automatic battery behavior based power model generation for smartphones. In
Energy-aware Test-suite Minimization for Android Apps. In Proc. of the 25th Int. Proc. of the 8th Int. Conf. on Hardware/Software Codesign and System Synthesis,
Symposium on Software Testing and Analysis (ISSTA 2016). 425–436. CODES+ISSS 2010, part of ESWeek ’10 Sixth Embedded Systems Week, Scottsdale,
[13] Ding Li, Shuai Hao, William GJ Halfond, and Ramesh Govindan. 2013. Calculating AZ, USA, October 24-28, 2010.
source line level energy information for android applications. In Proc. of the 2013
Int. Symposium on Software Testing and Analysis. ACM, 78–89.