A Comparative Study of Programming Languages

in Rosetta Code
Sebastian Nanz · Carlo A. Furia
Chair of Software Engineering, Department of Computer Science, ETH Zurich, Switzerland
[email protected]

Abstract—Sometimes debates on programming languages are and types of tasks solved, and by the use of novice program-
more religious than scientific. Questions about which language is mers as subjects. Real-world programming also develops over
more succinct or efficient, or makes developers more productive far more time than that allotted for short exam-like program-
are discussed with fervor, and their answers are too often based ming assignments; and produces programs that change features
on anecdotes and unsubstantiated beliefs. In this study, we use and improve quality over multiple development iterations.
arXiv:1409.0252v4 [cs.SE] 22 Jan 2015

the largely untapped research potential of Rosetta Code, a code

repository of solutions to common programming tasks in various At the opposite end of the spectrum, empirical studies
languages, which offers a large data set for analysis. Our study based on analyzing programs in public repositories such as
is based on 7’087 solution programs corresponding to 745 tasks
GitHub [2], [22], [25] can count on large amounts of mature
in 8 widely used languages representing the major programming
paradigms (procedural: C and Go; object-oriented: C# and Java; code improved by experienced developers over substantial
functional: F# and Haskell; scripting: Python and Ruby). Our time spans. Such set-ups are suitable for studies of defect
statistical analysis reveals, most notably, that: functional and proneness and code evolution, but they also greatly complicate
scripting languages are more concise than procedural and object- analyses that require directly comparable data across different
oriented languages; C is hard to beat when it comes to raw languages: projects in code repositories target disparate cate-
speed on large inputs, but performance differences over inputs gories of software, and even those in the same category (such
of moderate size are less pronounced and allow even interpreted as “web browsers”) often differ broadly in features, design,
languages to be competitive; compiled strongly-typed languages, and style, and hence cannot be considered to be implementing
where more defects can be caught at compile time, are less prone minor variants of the same task.
to runtime failures than interpreted or weakly-typed languages.
We discuss implications of these results for developers, language The study presented in this paper explores a middle ground
designers, and educators. between highly controlled but small programming assignments
and large but incomparable software projects: programs in
I. I NTRODUCTION Rosetta Code. The Rosetta Code repository [27] collects
solutions, written in hundreds of different languages, to an
What is the best programming language for. . .? Questions open collection of over 700 programming tasks. Most tasks
about programming languages and the properties of their are quite detailed descriptions of problems that go beyond
programs are asked often but well-founded answers are not simple programming assignments, from sorting algorithms to
easily available. From an engineering viewpoint, the design pattern matching and from numerical analysis to GUI program-
of a programming language is the result of multiple trade- ming. Solutions to the same task in different languages are
offs that achieve certain desirable properties (such as speed) thus significant samples of what each programming language
at the expense of others (such as simplicity). Technical aspects can achieve and are directly comparable. The community of
are, however, hardly ever the only relevant concerns when contributors to Rosetta Code (nearly 25’000 users at the time
it comes to choosing a programming language. Factors as of writing) includes programmers that scrutinize, revise, and
heterogeneous as a strong supporting community, similarity improve each other’s solutions.
to other widespread languages, or availability of libraries are
often instrumental in deciding a language’s popularity and how Our study analyzes 7’087 solution programs to 745 tasks
it is used in the wild [16]. If we want to reliably answer in 8 widely used languages representing the major program-
questions about properties of programming languages, we have ming paradigms (procedural: C and Go; object-oriented: C#
to analyze, empirically, the artifacts programmers write in and Java; functional: F# and Haskell; scripting: Python and
those languages. Answers grounded in empirical evidence can Ruby). The study’s research questions target various program
be valuable in helping language users and designers make features including conciseness, size of executables, running
informed choices. time, memory usage, and failure proneness. A quantitative
statistical analysis, cross-checked for consistency against a
To control for the many factors that may affect the prop- careful inspection of plotted data, reveals the following main
erties of programs, some empirical studies of programming findings about the programming languages we analyzed:
languages [9], [21], [24], [30] have performed controlled ex-
periments where human subjects (typically students) in highly • Functional and scripting languages enable writing more
controlled environments solve small programming tasks in concise code than procedural and object-oriented lan-
different languages. Such controlled experiments provide the guages.
most reliable data about the impact of certain programming • Languages that compile into bytecode produce smaller
language features such as syntax and typing, but they are also executables than those that compile into native machine
necessarily limited in scope and generalizability by the number code.
 • C is hard to beat when it comes to raw speed on large B. Task selection
inputs. Go is the runner-up, also in terms of economical
memory usage. Whereas even vague task descriptions may prompt well-
• In contrast, performance differences between languages written solutions, our study requires comparable solutions to
shrink over inputs of moderate size, where languages with clearly-defined tasks. To identify them, we categorized tasks,
a lightweight runtime may be competitive even if they are based on their description, according to whether they are
interpreted. suitable for lines-of-code analysis (LOC), compilation (COMP),
• Compiled strongly-typed languages, where more defects and execution (EXEC); TC denotes the set of tasks in a
can be caught at compile time, are less prone to runtime category C. Categories are increasingly restrictive: lines-of-
failures than interpreted or weakly-typed languages. code analysis only includes tasks sufficiently well-defined
that their solutions can be considered minor variants of a
Beyond the findings specific to the programs and program- unique problem; compilation further requires that tasks de-
ming languages that we analyzed, our study paves the way mand complete solutions rather than sketches or snippets;
for follow-up research that can benefit from the wealth of execution further requires that tasks include meaningful inputs
data in Rosetta Code and generalize our findings to other and algorithmic components (typically, as opposed to data-
domains. To this end, Section IV discusses some practical structure and interface definitions). As Table 1 shows, many
implications of these findings for developers, language de- tasks are too vague to be used in the study, but the differences
signers, and educators, whose choices about programming between the tasks in the three categories are limited.
languages can increasingly rely on a growing fact base built
ALL LOC COMP EXEC PERF SCAL
on complementary sources. # TASKS 745 454 452 436 50 46
The bulk of the paper describes the design of our empirical
study (Section II), and its research questions and overall results Table 1: Classification and selection of Rosetta Code tasks.
(Section III). We refer to a detailed technical report [18] for
the complete fine-grain details of the measures, statistics, and Most tasks do not describe sufficiently precise and varied
plots. To support repetition and replication studies, we also inputs to be usable in an analysis of runtime performance. For
make the complete data available online1 , together with the instance, some tasks are computationally trivial, and hence
scripts we wrote to produce and analyze it. do not determine measurable resource usage when running;
others do not give specific inputs to be tested, and hence
solutions may run on incomparable inputs; others still are
II. M ETHODOLOGY well-defined but their performance without interactive input
is immaterial, such as in the case of graphic animation tasks.
A. The Rosetta Code repository To identify tasks that can be meaningfully used in analyses of
performance, we introduced two additional categories (PERF
Rosetta Code [27] is a code repository with a wiki inter- and SCAL) of tasks suitable for performance comparisons:
face. This study is based on a repository’s snapshot taken on 24 PERF describes “everyday” workloads that are not necessarily
June 20142 ; henceforth “Rosetta Code” denotes this snapshot. very resource intensive, but whose descriptions include well-
Rosetta Code is organized in 745 tasks. Each task is a defined inputs that can be consistently used in every solution;
natural language description of a computational problem or in contrast, SCAL describes “computing-intensive” workloads
theme, such as the bubble sort algorithm or reading the JSON with inputs that can easily be scaled up to substantial size and
data format. Contributors can provide solutions to tasks in their require well-engineered solutions. For example, sorting algo-
favorite programming languages, or revise already available rithms are computing-intensive tasks working on large input
solutions. Rosetta Code features 379 languages (with at least lists; “Cholesky matrix decomposition” is an “everyday” task
one solution per language) for a total of 49’305 solutions working on two test input matrices that can be decomposed
and 3’513’262 lines (total lines of program files). A solution quickly. The corresponding sets TPERF and TSCAL are disjoint
consists of a piece of code, which ideally should accurately subsets of the execution tasks TEXEC ; Table 1 gives their size.
follow a task’s description and be self-contained (including
test inputs); that is, the code should compile and execute in a
proper environment without modifications. C. Language selection

Tasks significantly differ in the detail, prescriptiveness, and Rosetta Code includes solutions in 379 languages. Analyz-
generality of their descriptions. The most detailed ones, such ing all of them is not worth the huge effort, given that many
as “Bubble sort”, consist of well-defined algorithms, described languages are not used in practice or cover only few tasks. To
informally and in pseudo-code, and include tests (input/output find a representative and significant subset, we rank languages
pairs) to demonstrate solutions. Other tasks are much vaguer according to a combination of their rankings in Rosetta Code
and only give a general theme, which may be inapplicable and in the TIOBE index [32]. A language’s Rosetta Code
to some languages or admit widely different solutions. For ranking is based on the number of tasks for which at least
instance, task “Memory allocation” just asks to “show how to one solution in that language exists: the larger the number
explicitly allocate and deallocate blocks of memory”. of tasks the higher the ranking; Table 2 lists the top-20
languages (LANG) in the Rosetta Code ranking (ROSETTA)
1 https://bitbucket.org/nanzs/rosettacodedata with the number of tasks they implement (# TASKS). The
2 Cloned into our Git repository1 using a modified version of the Perl module TIOBE programming community index [32] is a long-standing,
RosettaCode-0.0.5 available from http://cpan.org/. monthly-published language popularity ranking based on hits

2
in various search engines; Table 3 lists the top-20 languages Twenty-four languages satisfy both criteria. We assign
in the TIOBE index with their TIOBE score (TIOBE). scores to them, based on the following rules:
ROSETTA LANG # TASKS TIOBE
R1. A language ` receives a TIOBE score τ` = 1 iff it is in
#1 Tcl 718 #43
the top-20 in TIOBE (Table 3); otherwise, τ` = 2.
#2 Racket 706 –3
#3 Python 675 #8 R2. A language ` receives a Rosetta Code score ρ` corre-
#4 Perl 6 644 – sponding to its ranking in Rosetta Code (first column in
#5 Ruby 635 #14 Table 2).
#6 J 630 –
#7 C 630 #1 Using these scores, languages are ranked in increasing lexi-
#8 D 622 #50
#9 Go 617 #30
cographic order of (τ` , ρ` ). This ranking method sticks to the
#10 PicoLisp 605 – same rationale as C1 (prefer popular languages) and C2 (ensure
#11 Perl 601 #11 a statistically significant base for analysis), and helps mitigate
#12 Ada 582 #29 the role played by languages that are “hyped” in either the
#13 Mathematica 580 – TIOBE or the Rosetta Code ranking.
#14 REXX 566 –
#15 Haskell 553 #38 To cover the most popular programming paradigms, we
#16 AutoHotkey 536 –
#17 Java 534 #2
partition languages in four categories: procedural, object-
#18 BBC BASIC 515 – oriented, functional, scripting. Two languages (R and MAT-
#19 Icon 473 – LAB) mainly are special-purpose; hence we drop them. In
#20 OCaml 471 – each category, we rank languages using our ranking method
and pick the top two languages. Table 4 shows the overall
Table 2: Rosetta Code ranking: top 20. ranking; the shaded rows contain the eight languages selected
for the study.
TIOBE LANG # TASKS ROSETTA
#1 C 630 #7 PROCEDURAL OBJECT- ORIENTED FUNCTIONAL SCRIPTING
#2 Java 534 #17 ` (τ` , ρ` ) ` (τ` , ρ` ) ` (τ` , ρ` ) ` (τ` , ρ` )
#3 Objective-C 136 #72 C (1,7) Java (1,17) F# (1,8) Python (1,3)
#4 C++ 461 #22 Go (2,9) C# (1,21) Haskell (2,15) Ruby (1,5)
#5 (Visual) Basic 34 #145 Ada (2,12) C++ (1,22) Common Lisp (2,23) Perl (1,11)
#6 C# 463 #21 PL/I (2,30) D (2,50) Scala (2,25) JavaScript (1,28)
#7 PHP 324 #36 Fortran (2,39) Erlang (2,26) PHP (1,36)
#8 Python 675 #3 Scheme (2,47) Tcl (2,1)
#9 JavaScript 371 #28 Lua (2,35)
#10 Transact-SQL 4 #266
#11 Perl 601 #11 Table 4: Combined ranking: the top-2 languages in each
#12 Visual Basic .NET 104 #81 category are selected for the study.
#13 F# 341 #33
#14 Ruby 635 #5
#15 ActionScript 113 #77
#16 Swift –4 – D. Experimental setup
#17 Delphi/Object Pascal 219 #53
#18 Lisp –5 – Rosetta Code collects solution files by task and language.
#19 MATLAB 305 #40 The following table details the total size of the data considered
#20 Assembly –5 – in our experiments (LINES are total lines of program files).
Table 3: TIOBE index ranking: top 20. C C# F# Go Haskell Java Python Ruby ALL
TASKS 630 463 341 617 553 534 675 635 745
FILES 989 640 426 869 980 837 1’319 1’027 7’087
A language ` must satisfy two criteria to be included in LINES 44’643 21’295 6’473 36’395 14’426 27’891 27’223 19’419 197’765
our study:
Our experiments measure properties of Rosetta Code solu-
C1. ` ranks in the top-50 positions in the TIOBE index; tions in various dimensions: source-code features (such as lines
C2. ` implements at least one third (≈ 250) of the Rosetta of code), compilation features (such as size of executables),
Code tasks. and runtime features (such as execution time). Correspond-
ingly, we have to perform the following actions for each
Criterion C1 selects widely-used, popular languages. Criterion solution file f of every task t in each language `:
C2 selects languages that can be compared on a substantial
number of tasks, conducing to statistically significant results. • Merge: if f depends on other files (for example, an
Languages in Table 2 that fulfill criterion C1 are shaded application consisting of two classes in two different
(the top-20 in TIOBE are in bold); and so are languages in files), make them available in the same location where
Table 3 that fulfill criterion C2. A comparison of the two tables f is; F denotes the resulting self-contained collection of
indicates that some popular languages are underrepresented source files that correspond to one solution of t in `.
in Rosetta Code, such as Objective-C, (Visual) Basic, and
Transact-SQL; conversely, some languages popular in Rosetta 3 No rank means that the language is not in the top-50 in the TIOBE index.
Code have a low TIOBE ranking, such as Tcl, Racket, and 4 Not represented in Rosetta Code.
Perl 6. 5 Only represented in Rosetta Code in dialect versions.

3
 • Patch: if F has errors that prevent correct compilation did not change the inputs themselves from those suggested in
or execution (for example, a library is used but not the task descriptions. In contrast, we inspected all solutions to
imported), correct F as needed. tasks TSCAL and patched them by supplying task-specific inputs
• LOC: measure source-code features of F. that are computationally demanding. A significant example of
• Compile: compile F into native code (C, Go, and computing-intensive tasks were the sorting algorithms, which
Haskell) or bytecode (C#, F#, Java, Python); executable we patched to build and sort large integer arrays (generated
denotes the files produced by compilation.6 Measure on the fly using a linear congruential generator function with
compilation features. fixed seed). The input size was chosen after a few trials so
• Run: run the executable and measure runtime features. as to be feasible for most languages within a timeout of 3
minutes; for example, the sorting algorithms deal with arrays
Actions merge and patch are solution-specific and are of size from 3 · 104 elements for quadratic-time algorithms to
required for the actions that follow. In contrast, LOC, compile, 2 · 106 elements for linear-time algorithms.
and run are only language-specific and produce the actual
experimental data. To automate executing the actions to the LOC. For each language `, we wrote a script `_loc that
extent possible, we built a system of scripts that we now inputs a list of files, calls cloc7 on them to count the lines of
describe in some detail. code, and logs the results.

Merge. We stored the information necessary for this step Compile. For each language `, we wrote a script `_compile
in the form of makefiles—one for every task that requires that inputs a list of files and compilation flags, calls the appro-
merging, that is such that there is no one-to-one correspon- priate compiler on them, and logs the results. The following
dence between source-code files and solutions. A makefile table shows the compiler versions used for each language,
has one target for every task solution F, and a default all as well as the optimization flags. We tried to select a stable
target that builds all solution targets for the current task. compiler version complete with matching standard libraries,
Each target’s recipe calls a placeholder script comp, passing and the best optimization level among those that are not too
to it the list of input files that constitute the solution together aggressive or involve rigid or extreme trade-offs.
with other necessary solution-specific compilation files (for
LANG COMPILER VERSION FLAGS
example, library flags for the linker). We wrote the makefiles C gcc (GNU) 4.6.3 -O2
after attempting a compilation with default options for all C# mcs (Mono 3.2.1) 3.2.1.0 -optimize
solution files, each compiled in isolation: we inspected all F# fsharpc (Mono 3.2.1) 3.1 -O
Go go 1.3
failed compilation attempts and provided makefiles whenever Haskell ghc 7.4.1 -O2
necessary. Java javac (OracleJDK 8) 1.8.0_11
Python python (CPython) 2.7.3/3.2.3
Patch. We stored the information necessary for this step in Ruby ruby 2.1.2 -c

the form of diffs—one for every solution file that requires cor-
rection. We wrote the diffs after attempting a compilation with C_compile tries to detect the C dialect (gnu90, C99, . . . )
the makefiles: we inspected all failed compilation attempts, and until compilation succeeds. Java_compile looks for names
wrote diffs whenever necessary. Some corrections could not be of public classes in each source file and renames the files
expressed as diffs because they involved renaming or splitting to match the class names (as required by the Java compiler).
files (for example, some C files include both declarations and Python_compile tries to detect the version of Python (2.x or
definitions, but the former should go in separate header files); 3.x) until compilation succeeds. Ruby_compile only performs
we implemented these corrections by adding shell commands a syntax check of the source (flag -c), since Ruby has no
directly in the makefiles. (standard) stand-alone compilation.

An important decision was what to patch. We want to have Run. For each language `, we wrote a script `_run that
as many compiled solutions as possible, but we also do not inputs an executable name, executes it, and logs the results.
want to alter the Rosetta Code data before measuring it. We Native executables are executed directly, whereas bytecode
did not fix errors that had to do with functional correctness is executed using the appropriate virtual machines. To have
or very solution-specific features. We did fix simple errors: reliable performance measurements, the scripts: repeat each
missing library inclusions, omitted variable declarations, and execution 6 times; discard the timing of the first execution
typos. These guidelines try to replicate the moves of a user (to fairly accommodate bytecode languages that load virtual
who would like to reuse Rosetta Code solutions but may not machines from disk: it is only in the first execution that
be fluent with the languages. As a result of following this the virtual machine is loaded from disk, with corresponding
process, we have a reasonably high confidence that patched possibly significant one-time overhead; in the successive ex-
solutions are correct implementations of the tasks. ecutions the virtual machine is read from cache, with only
limited overhead); check that the 5 remaining execution times
Diffs play an additional role for tasks for performance are within one standard deviation of the mean; log the mean
analysis (TPERF and TSCAL in Section II-B). Solutions to these execution time. If an execution does not terminate within a
tasks must not only be correct but also run on the same inputs time-out of 3 minutes it is forcefully terminated.
(tasks TPERF ) and on the same “large” inputs (tasks TSCAL ). We
checked all solutions to tasks TPERF and patched them when Overall process. A Python script orchestrates the whole
necessary to ensure they work on comparable inputs, but we experiment. For every language `, for every task t, for each
action act ∈ {loc, compile, run}:
6 For Ruby, which does not produce compiled code of any kind, this step is
replaced by a syntax check of F. 7 http://cloc.sourceforge.net/

4
 1) if patches exist for any solution of t in `, apply them; As statistical test, we normally8 use the Wilcoxon signed-
2) if no makefile exists for task t in `, call script `_act rank test, a paired non-parametric difference test which as-
directly on each solution file f of t; sesses whether the mean ranks of xM α and of yα differ. We
M
3) if a makefile exists, invoke it and pass `_act as command display the test results in a table, under column labeled with
compto be used; the makefile defines the self-contained language X at row labeled with language Y , and include
collection of source files F on which the script works. various measures:

Since the command-line interface of the `_loc, `_compile, and 1) The p-value, which estimates the probability that chance
can explain the differences between xM α and yα . Intu-
`_run scripts is uniform, the same makefiles work as recipes M
for all actions act. itively, if p is small it means that there is a high chance
that X and Y exhibit a genuinely different behavior with
respect to metric M.
E. Experiments 2) The effect size, computed as Cohen’s d, defined as the
standardized mean difference: d = (xM α − yα )/s, where
M
The experiments ran on a Ubuntu 12.04 LTS 64bit V is the mean of a vector V , and s is the pooled
GNU/Linux box with Intel Quad Core2 CPU at 2.40 GHz and standard deviation of the data. For statistically significant
4 GB of RAM. At the end of the experiments, we extracted differences, d estimates how large the difference is.
all logged data for statistical analysis using R. 3) The signed ratio
α f α
α max(xM , yM )
f
M − yM )
α
R = sgn(xf
F. Statistical analysis f
α f α
M , yM )
min(xf
The statistical analysis targets pairwise comparisons be-
tween languages. Each comparison uses a different metric M of the largest median to the smallest median (where Ve is
including lines of code (conciseness), size of the executable the median of a vector V ), which gives an unstandardized
(native or bytecode), CPU time, maximum RAM usage (i.e., measure of the difference between the two medians.9
maximum resident set size), number of page faults, and number Sign and absolute value of R have direct interpretations
of runtime failures. Metrics are normalized as we detail below. whenever the difference between X and Y is significant:
Let ` be a programming language, t a task, and M a metric. if M is such that “smaller is better” (for instance, running
M − yM ) indicates that the
`M (t) denotes the vector of measures of M, one for each time), then a positive sign sgn(xfα f α
solution to task t in language `. `M (t) may be empty if there average solution in language Y is better (smaller) with
are no solutions to task t in `. The comparison of languages X respect to M than the average solution in language X; the
and Y based on M works as follows. Consider a subset T of the absolute value of R indicates how many times X is larger
tasks such that, for every t ∈ T , both X and Y have at least one than Y on average.
solution to t. T may be further restricted based on a measure-
dependent criterion; for example, to check conciseness, we Throughout the paper, we will say that language X: is
may choose to only consider a task t if both X and Y have at significantly different from language Y , if p < 0.01; and that it
least one solution that compiles without errors (solutions that tends to be different from Y if 0.01 ≤ p < 0.05. We will say that
do not satisfy the criterion are discarded). the effect size is: vanishing if d < 0.05; small if 0.05 ≤ d < 0.3;
medium if 0.3 ≤ d < 0.7; and large if d ≥ 0.7.
Following this procedure, each T determines two data vec-
α and yα , for the two languages X and Y , by aggregating
tors xM M G. Visualizations of language comparisons
the measures per task using an aggregation function α; as
aggregation functions, we normally consider both minimum Each results table is accompanied by a language relation-
and mean. For each task t ∈ T , the t-th component of the two ship graph, which helps visualize the results of the the pairwise
α and yα is:
vectors xM M language relationships. In such graphs, nodes correspond to
programming languages. Two nodes `1 and `2 are arranged
α
xM (t) = α(XM (t))/νM (t, X,Y ) , so that their horizontal distance is roughly proportional to
α
yM (t) = α(YM (t))/νM (t, X,Y ) , the absolute value of ratio R for the two languages; an exact
proportional display is not possible in general, as the pairwise
where νM (t, X,Y ) is a normalization factor defined as: ordering of languages may not be a total order. Vertical
 distances are chosen only to improve readability and carry no
min (XM (t)YM (t)) if min(XM (t)YM (t)) > 0 , meaning.
νM (t, X,Y ) =
1 otherwise , A solid arrow is drawn from node X to Y if language Y
is significantly better than language X in the given metric,
where juxtaposing vectors denotes concatenating them. Thus, and a dashed arrow if Y tends to be better than X (using the
the normalization factor is the smallest value of metric M terminology from Section II-F). To improve the visual layout,
measured across all solutions of t in X and in Y if such a edges that express an ordered pair that is subsumed by others
value is positive; otherwise, when the minimum is zero, the are omitted, that is if X → W → Y the edge from X to Y is
α (t)
normalization factor is one. This definition ensures that xM
and yαM (t) are well-defined even when a minimum of zero 8 Failure analysis (RQ5) uses the U test, as described there.
occurs due to the limited precision of some measures such 9 The definition of R uses median as average to lessen the influence of
as running time. outliers.

5
omitted. The thickness of arrows is proportional to the effect average 2.2–2.9 times longer than programs in functional and
size; if the effect is vanishing, no arrow is drawn. scripting languages.
Within the two groups, differences are less pronounced.
III. R ESULTS Among the scripting languages, and among the functional
RQ1. Which programming languages make for more con- languages, no statistically significant differences exist. Python
cise code? tends to be the most concise, even against functional languages
(1.2–1.6 times shorter on average). Among procedural and
To answer this question, we measure the non-blank non- object-oriented languages, Java tends to be slightly more
comment lines of code of solutions of tasks TLOC marked for concise, with small to medium effect sizes.
lines of code count that compile without errors. The require-  
ment of successful compilation ensures that only syntactically Functional and scripting languages provide signifi-
correct programs are considered to measure conciseness. To cantly more concise code than procedural and object-
check the impact of this requirement, we also compared these oriented languages.
results with a measurement including all solutions (whether 
they compile or not), obtaining qualitatively similar results. RQ2. Which programming languages compile into smaller
executables?
For all research questions but RQ5, we considered both
minimum and mean as aggregation functions (Section II-F). To answer this question, we measure the size of the
For brevity, the presentation describes results for only one executables of solutions of tasks TCOMP marked for compilation
of them (typically the minimum). For lines of code measure- that compile without errors. We consider both native-code ex-
ments, aggregating by minimum means that we consider, for ecutables (C, Go, and Haskell) and bytecode executables (C#,
each task, the shortest solution available in the language. F#, Java, Python). Ruby’s standard programming environment
does not offer compilation to bytecode and Ruby programs are
Table 5 shows the results of the pairwise comparison,
therefore not included in the measurements for RQ2.
where p is the p-value, d the effect size, and R the ratio, as
described in Section II-F. In the table, ε denotes the smallest Table 7 shows the results of the statistical analysis, and
positive floating-point value representable in R. Figure 8 the corresponding language relationship graph.
LANG C C# F# Go Haskell Java Python LANG C C# F# Go Haskell Java
C# p 0.543 C# p <ε
d 0.004 d 2.669
R -1.1 R 2.1
F# p <ε <ε F# p <ε < 10-15
d 0.735 0.945 d 1.395 1.267
R 2.5 2.6 R 1.4 -1.5
Go p 0.377 0.082 < 10-29 Go p < 10-52 < 10-39 < 10-31
d 0.155 0.083 0.640 d 3.639 2.312 2.403
R 1.0 1.0 -2.5 R -153.3 -340.7 -217.8
Haskell p <ε <ε 0.168 <ε Haskell p < 10-45 < 10-35 < 10-29 <ε
d 1.071 1.286 0.085 1.255 d 2.469 2.224 2.544 1.071
R 2.9 2.7 1.3 2.9 R -111.2 -240.3 -150.8 1.3
Java p 0.026 < 10-4 < 10-25 0.026 < 10-32 Java p <ε < 10-4 <ε <ε <ε
d 0.262 0.319 0.753 0.148 1.052 d 3.148 0.364 1.680 3.121 1.591
R 1.0 1.1 -2.3 1.0 -2.9 R 2.3 1.1 1.7 341.4 263.8
Python p <ε <ε < 10-4 <ε 0.021 <ε Python p <ε < 10-15 <ε <ε <ε < 10-5
d 0.951 1.114 0.359 0.816 0.209 0.938 d 5.686 0.899 1.517 3.430 1.676 0.395
R 2.9 3.6 1.6 3.0 1.2 2.8 R 2.9 1.3 2.0 452.7 338.3 1.3
Ruby p <ε <ε 0.013 <ε 0.764 <ε 0.015
d 0.558 0.882 0.103 0.742 0.107 0.763 0.020 Table 7: Comparison of size of executables (by minimum).
R 2.5 2.7 1.2 2.5 -1.2 2.2 -1.3

Table 5: Comparison of lines of code (by minimum).

Python

Java
Go Haskell
C#
C# Java Ruby Python
F#
C F#
Go Haskell C
Figure 6: Comparison of lines of code (by minimum). Figure 8: Comparison of size of executables (by minimum).
Figure 6 shows the corresponding language relationship It is apparent that measuring executable sizes determines
graph; remember that arrows point to the more concise a total order of languages, with Go producing the largest and
languages, thickness denotes larger effects, and horizontal Python the smallest executables. Based on this order, three
distances are roughly proportional to average differences. consecutive groups naturally emerge: Go and Haskell compile
Languages are clearly divided into two groups: functional to native and have “large” executables; F#, C#, Java, and
and scripting languages tend to provide the most concise Python compile to bytecode and have “small” executables; C
code, whereas procedural and object-oriented languages are compiles to native but with size close to bytecode executables.
significantly more verbose. The absolute difference between
the two groups is major; for instance, Java programs are on Size of bytecode does not differ much across languages:

6
F#, C#, and Java executables are, on average, only 1.3–2.0 Table 10 shows the results of the statistical analysis, and
times larger than Python’s. The differences between sizes Figure 11 the corresponding language relationship graph.
of native executables are more spectacular, with Go’s and
LANG C C# F# Go Haskell Java Python
Haskell’s being on average 153.3 and 111.2 times larger than C# p 0.001
C’s. This is largely a result of Go and Haskell using static d 0.328
linking by default, as opposed to gcc defaulting to dynamic R -7.5
F# p 0.012 0.075
linking whenever possible. With dynamic linking, C produces d 0.453 0.650
very compact binaries, which are on average a mere 2.9 R -8.6 -2.6
times larger than Python’s bytecode. C was compiled with Go p < 10-4 0.020 0.016
d 0.453 0.338 0.578
level -O2 optimization, which should be a reasonable middle R -1.6 6.3 5.6
ground: binaries tend to be larger under more aggressive speed Haskell p < 10-4 0.084 0.929 < 10-3

optimizations, and smaller under executable size optimizations d 0.895 0.208 0.424 0.705
R -24.5 -2.9 -1.6 -13.1
(flag -Os). Java p < 10-4 0.661 0.158 0.0135 0.098
  d 0.374 0.364 0.469 0.563 0.424
Languages that compile into bytecode have signifi- R -3.2 1.8 4.9 -2.4 6.6
Python p < 10-5 0.033 0.938 < 10-3 0.894 0.082
cantly smaller executables than those that compile into d 0.704 0.336 0.318 0.703 0.386 0.182
 native machine code. R -29.8 -4.3 -1.1 -15.5 1.1 -9.2
Ruby p < 10-3 0.004 0.754 < 10-3 0.360 0.013 0.055
RQ3. Which programming languages have better running- d 0.999 0.358 0.113 0.984 0.250 0.204 0.020
time performance? R -34.1 -6.2 -1.4 -18.5 -1.4 -7.9 -1.6

To answer this question, we measure the running time of Table 10: Comparison of running time (by minimum) for
solutions of tasks TSCAL marked for running time measurements computing-intensive tasks.
on computing-intensive workloads that run without errors or
timeout (set to 3 minutes). As discussed in Section II-B F#
and Section II-D, we manually patched solutions to tasks Ruby Java
in TSCAL to ensure that they work on the same inputs of
substantial size. This ensures that—as is crucial for running
C# Go C
time measurements—all solutions used in these experiments
run on the very same inputs.
Python
NAME INPUT Haskell
1 9 billion names of God the integer n = 105
2–3 Anagrams 100 × unixdict.txt (20.6 MB) Figure 11: Comparison of running time (by minimum) for
4 Arbitrary-precision integers
32
54

computing-intensive tasks.
25
5 Combinations 10
6 Count in factors n = 106 C is unchallenged over the computing-intensive tasks
7 Cut a rectangle 10 × 10 rectangle TSCAL . Go is the runner-up, but significantly slower with
8 Extensible prime generator 107 th prime
9 Find largest left truncatable prime 107 th prime medium effect size: the average Go program is 1.6 times slower
10 Hamming numbers 107 th Hamming number than the average C program. Programs in other languages are
11 Happy numbers 106 th Happy number much slower than Go programs, with medium to large effect
12 Hofstadter Q sequence # flips up to 105 th term
13–16 Knapsack problem/[all versions] from task description size (2.4–18.5 times slower than Go on average).
17
18
Ludic numbers
LZW compression
from task description
100 × unixdict.txt (20.6 MB)
 
19 Man or boy test n = 16 C is king on computing-intensive workloads. Go is the
20 N-queens problem n = 13 runner-up. Other languages, with object-oriented or
21 Perfect numbers first 5 perfect numbers functional features, incur further performance losses.
22 Pythagorean triples perimeter < 108 
23 Self-referential sequence n = 106
24 Semordnilap 100 × unixdict.txt The results on the tasks TSCAL clearly identified the proce-
25 Sequence of non-squares non-squares < 106
26–34 Sorting algorithms/[quadratic] n ' 104
dural languages—C in particular—as the fastest. However, the
35–41 Sorting algorithms/[n log n and linear] n ' 106 raw speed demonstrated on those tasks represents challenging
42–43 Text processing/[all versions] from task description (1.2 MB) conditions that are relatively infrequent in the many classes of
44 Topswops n = 12
45 Towers of Hanoi n = 25 applications that are not algorithmically intensive. To find out
46 Vampire number from task description performance differences under other conditions, we measure
Table 9: Computing-intensive tasks. running time on the tasks TPERF , which are still clearly defined
and run on the same inputs, but are not markedly computa-
tionally intensive and do not naturally scale to large instances.
Table 9 summarizes the tasks TSCAL and their inputs. It is Examples of such tasks are checksum algorithms (Luhn’s
a diverse collection which spans from text processing tasks credit card validation), string manipulation tasks (reversing the
on large input files (“Anagrams”, “Semordnilap”), to combi- space-separated words in a string), and standard system library
natorial puzzles (“N-queens problem”, “Towers of Hanoi”), accesses (securing a temporary file).
to NP-complete problems (“Knapsack problem”) and sorting
algorithms of varying complexity. We chose inputs sufficiently The results, which we only discuss in the text for brevity,
large to probe the performance of the programs, and to are definitely more mixed than those related to tasks TSCAL ,
make input/output overhead negligible w.r.t. total running time. which is what one could expect given that we are now looking

7
into modest running times in absolute value, where every Haskell F# Java
language has at least decent performance. First of all, C loses
its undisputed supremacy, as it is not significantly faster than
Go and Haskell—even though, when differences are statisti- Python C# Go C
cally significant, C remains ahead of the other languages. The
procedural languages and Haskell collectively emerge as the Ruby
fastest in tasks TPERF ; none of them sticks out as the fastest Figure 13: Comparison of maximum RAM used (by mini-
because the differences among them are insignificant and may mum).
sensitively depend on the tasks that each language implements
in Rosetta Code. Among the other languages (C#, F#, Java, Java), lazy evaluation, pattern matching (Haskell and F#),
Python, and Ruby), Python emerges as the fastest. Overall, we dynamic typing, and reflection (Python and Ruby).
confirm that the distinction between TPERF and TSCAL tasks—
which we dub “everyday” and “computing-intensive”—is quite Differences between languages in the same category (pro-
important to understand performance differences among lan- cedural, scripting, and functional) are generally small or in-
guages. On tasks TPERF , languages with an agile runtime, such significant. The exception are object-oriented languages, where
as the scripting languages, or with natively efficient operations Java uses significantly more RAM than C# (on average, 2.4
on lists and string, such as Haskell, may turn out to be efficient times more). Among the other languages, Haskell emerges
in practice. as the least memory-efficient, although some differences are
 insignificant.
The distinction between “everyday” and “computing-
intensive” workloads is important when assessing While maximum RAM usage is a major indication of
running-time performance. On “everyday” workloads, the efficiency of memory usage, modern architectures in-
languages may be able to compete successfully regard- clude many-layered memory hierarchies whose influence on
less of their programming paradigm. performance is multi-faceted. To complement the data about
 maximum RAM and refine our understanding of memory
RQ4. Which programming languages use memory more usage, we also measured average RAM usage and number
efficiently? of page faults. Average RAM tends to be practically zero
To answer this question, we measure the maximum RAM in all tasks but very few; correspondingly, the statistics are
usage (i.e., maximum resident set size) of solutions of tasks inconclusive as they are based on tiny samples. By contrast,
TSCAL marked for comparison on computing-intensive tasks the data about page faults clearly partitions the languages
that run without errors or timeout; this measure includes in two classes: the functional languages trigger significantly
the memory footprint of the runtime environments. Table 12 more page faults than all other languages; in fact, the only
shows the results of the statistical analysis, and Figure 13 the statistically significant differences are those involving F# or
corresponding language relationship graph. Haskell, whereas programs in other languages hardly ever trig-
ger a single page fault. The difference in page faults between
LANG C C# F# Go Haskell Java Python Haskell programs and F# programs is insignificant. The page
C# p < 10-4 faults recorded in our experiments indicate that functional
d 2.022
R -2.5 languages exhibit significant non-locality of reference. The
F# p 0.006 0.010 overall impact of this phenomenon probably depends on a
d 0.761 1.045
R -4.5 -5.2
machine’s architecture; RQ3, however, showed that functional
Go p < 10-3 < 10-4 0.006 languages are generally competitive in terms of running-time
d 0.064 0.391 0.788 performance, so that their non-local behavior might just denote
R -1.8 4.1 3.5
Haskell p < 10 -3 0.841 0.062 < 10 -3 a particular instance of the space vs. time trade-off.
d 0.287 0.123 0.614 0.314  
R -14.2 -3.5 2.5 -5.6 Procedural languages use significantly less memory
Java p < 10-5 < 10-4 0.331 < 10-5 0.007 than other languages. Functional languages make
d 0.890 1.427 0.278 0.527 0.617
distinctly non-local memory accesses.
R -5.4 -2.4 1.4 -2.6
Python p < 10-5 0.351 0.034 < 10-4
2.1
0.992 0.006

d 0.330 0.445 0.096 0.417 0.010 0.206 RQ5. Which programming languages are less failure
R -5.0 -2.2 1.0 -2.7 2.9 -1.1
Ruby p < 10-5 0.002 0.530 < 10-4 0.049 0.222 0.036 prone?
d 0.403 0.525 0.242 0.531 0.301 0.301 0.061
R -5.0 -5.0 1.5 -2.5 1.8 -1.1 1.0 To answer this question, we measure runtime failures of
solutions of tasks TEXEC marked for execution that compile
Table 12: Comparison of maximum RAM used (by minimum).
without errors or timeout. We exclude programs that time out
because whether a timeout is indicative of failure depends on
C and Go clearly emerge as the languages that make the the task: for example, interactive applications will time out
most economical usage of RAM. Go’s frugal memory usage in our setup waiting for user input, but this should not be
(on average only 1.8 times higher than C) is remarkable, recorded as failure. Thus, a terminating program fails if it
given that its runtime includes garbage collection. In contrast, returns an exit code other than 0 (for example, when throwing
all other languages use considerably more memory (2.5–14.2 an uncaught exception). The measure of failures is ordinal and
times on average over either C or Go), which is justifiable not normalized: ` f denotes a vector of binary values, one for
in light of their bulkier runtimes, supporting not only garbage each solution in language ` where we measure runtime failures;
collection but also features such as dynamic binding (C# and a value in ` f is 1 iff the corresponding program fails and it is

8
0 if it does not fail. help make compile-time checks effective. By contrast, the
scripting languages tend to be the most failure prone of the lot.
Data about failures differs from that used to answer the
This is a consequence of Python and Ruby being interpreted
other research questions in that we cannot aggregate it by
languages10 : any syntactically correct program is executed, and
task, since failures in different solutions, even for the same
hence most errors manifest themselves only at runtime.
task, are in general unrelated. Therefore, we use the Mann-
Whitney U test, an unpaired non-parametric ordinal test which There are few major differences among the remaining
can be applied to compare samples of different size. For two compiled languages, where it is useful to distinguish between
languages X and Y , the U test assesses whether the two weak (C) and strong (the other languages) type systems [8,
samples X f and Y f of binary values representing failures are Sec. 3.4.2]. F# shows no statistically significant differences
likely to come from the same population. with any of C, C#, and Haskell. C tends to be more failure
prone than C# and is significantly more failure prone than
C C# F# Go Haskell Java Python Ruby
# ran solutions 391 246 215 389 376 297 675 516
Haskell; similarly to the explanation behind the interpreted
% no error 87% 93% 89% 98% 93% 85% 85% 86% languages’ failure proneness, C’s weak type system may be
responsible for fewer failures being caught at compile time
Table 14: Number of solutions that ran without timeout, and
than at runtime. In fact, the association between weak typing
their percentage that ran without errors.
and failure proneness was also found in other studies [25].
Java is unusual in that it has a strong type system and is
Table 15 shows the results of the tests; we do not report
compiled, but is significantly more error prone than Haskell
unstandardized measures of difference, such as R in the pre-
and C#, which also are strongly typed and compiled. Future
vious tables, since they would be uninformative on ordinal
work will determine if Java’s behavior is spurious or indicative
data. Figure 16 is the corresponding language relationship
of concrete issues.
graph. Horizontal distances are proportional to the fraction of  
solutions that run without errors (last row of Table 14). Compiled strongly-typed languages are significantly
less prone to runtime failures than interpreted or
LANG C C# F# Go Haskell Java Python
C# p 0.037 weakly-typed languages, since more errors are caught
d 0.170 at compile time. Thanks to its simple static type system,
F# p 0.500 0.204 Go is the least failure-prone language in our study.
d 0.057 0.119  
Go p < 10-7 0.011 < 10-5
d 0.410 0.267 0.398
Haskell p 0.006 0.748 0.083 0.002 IV. D ISCUSSION
d 0.200 0.026 0.148 0.227
Java p 0.386 0.006 0.173 < 10-9 < 10-3 The results of our study can help different stakeholders—
d 0.067 0.237 0.122 0.496 0.271 developers, language designers, and educators—to make better
Python p 0.332 0.003 0.141 < 10-10 < 10-3 0.952
d 0.062 0.222 0.115 0.428 0.250 0.004
informed choices about language usage and design.
Ruby p 0.589 0.010 0.260 < 10-9 < 10-3 0.678 0.658
d 0.036 0.201 0.091 0.423 0.231 0.030 0.026 The conciseness of functional and scripting programming
languages suggests that the characterizing features of these
Table 15: Comparisons of runtime failure proneness. languages—such as list comprehensions, type polymorphism,
dynamic typing, and extensive support for reflection and list
F#
and map data structures—provide for great expressiveness.
Python In times where more and more languages combine elements
C# belonging to different paradigms, language designers can focus
on these features to improve the expressiveness and raise the
Java C Go
level of abstraction. For programmers, using a programming
Haskell language that makes for concise code can help write software
Ruby with fewer bugs. In fact, some classic research suggests [11],
Figure 16: Comparisons of runtime failure proneness. [14], [15] that bug density is largely constant across pro-
gramming languages—all else being equal [7], [17]; therefore,
C C# F# Go Haskell Java Python Ruby shorter programs will tend to have fewer bugs.
# comp. solutions 524 354 254 497 519 446 775 581
% no error 85% 90% 95% 89% 84% 78% 100% 100% The results about executable size are an instance of the
ubiquitous space vs. time trade-off. Languages that compile
Table 17: Number of solutions considered for compilation, and to native can perform more aggressive compile-time optimiza-
their percentage that compiled without errors. tions since they produce code that is very close to the actual
hardware it will be executed on. In fact, compilers to native
Go clearly sticks out as the least failure prone language. tend to have several optimization options, which exercise
If we look, in Table 17, at the fraction of solutions that different trade-offs. GNU’s gcc, for instance, has a -Os flag
failed to compile, and hence didn’t contribute data to failure that optimizes for executable size instead of speed (but we
analysis, Go is not significantly different from other compiled didn’t use this highly specialized optimization in our experi-
languages. Together, these two elements indicate that the Go ments). However, with the ever increasing availability of cheap
compiler is particularly good at catching sources of failures at and compact memory, differences between languages have
compile time, since only a small fraction of compiled programs
fail at runtime. Go’s restricted type system (no inheritance, 10 Even if Python compiles to bytecode, the translation process only per-
no overloading, no genericity, no pointer arithmetic) may forms syntactic checks (and is not invoked separately normally anyway).

9
significant implications only for applications that run on highly and the measures we take to answer them, target widespread
constrained hardware such as embedded devices(where, in well-defined features (conciseness, performance, and so on)
fact, bytecode languages are becoming increasingly common). with straightforward matching measures (lines of code, running
Interpreted languages such as Ruby exercise yet another trade- time, and so on). A partial exception is RQ5, which targets
off, where there is no visible binary at all and all optimizations the multifaceted notion of failure proneness, but the question
are done at runtime. and its answer are consistent with related empirical work that
approached the same theme from other angles, which reflects
No one will be surprised by our results that C dominates positively on the soundness of our constructs. Regarding
other languages in terms of raw speed and efficient memory conciseness, lines of code remains a widely used metric, but it
usage. Major progresses in compiler technology notwithstand- will be interesting to correlate it with other proposed measures
ing, higher-level programming languages do incur a noticeable of conciseness.
performance loss to accommodate features such as automatic
memory management or dynamic typing in their runtimes. We took great care in the study’s design and execution to
Nevertheless, our results on “everyday” workloads showed that minimize threats to internal validity—are we measuring things
pretty much any language can be competitive when it comes to right? We manually inspected all task descriptions to ensure
the regular-size inputs that make up the overwhelming majority that the study only includes well-defined tasks and comparable
of programs. When teaching and developing software, we solutions. We also manually inspected, and modified whenever
should then remember that “most applications do not actually necessary, all solutions used to measure performance, where
need better performance than Python offers” [26, p. 337]. it is of paramount importance that the same inputs be applied
in every case. To ensure reliable runtime measures (running
Another interesting lesson emerging from our performance time, memory usage, and so on), we ran every executable
measurements is how Go achieves respectable running times as multiple times, checked that each repeated run’s deviation from
well as good results in memory usage, thereby distinguishing the average is moderate (less than one standard deviation),
itself from the pack just as C does (in fact, Go’s developers and based our statistics on the average (mean) behavior. Data
include prominent figures—Ken Thompson, most notably— analysis often showed highly statistically significant results,
who were also primarily involved in the development of C). which also reflects favorably on the soundness of the study’s
Go’s design choices may be traced back to a careful selection data. Our experimental setup tried to use standard tools with
of features that differentiates it from most other language de- default settings; this may limit the scope of our findings, but
signs (which tend to be more feature-prodigal): while it offers also helps reduce biasdue to different familiarity with different
automatic memory management and some dynamic typing, it languages. Exploring different directions, such as pursuing the
deliberately omits genericity and inheritance, and offers only a best optimizations possible in each language [21]for each task,
limited support for exceptions. In our study, we have seen that is an interesting goal of future work.
Go features not only good performance but also a compiler
that is quite effective at finding errors at compile time rather A possible threat to external validity—do the findings
than leaving them to leak into runtime failures. Besides being generalize?—has to do with whether the properties of Rosetta
appealing for certain kinds of software development (Go’s Code programs are representative of real-world software
concurrency mechanisms, which we didn’t consider in this projects. On one hand, Rosetta Code tasks tend to favor
study, may be another feature to consider), Go also shows algorithmic problems, and solutions are quite small on average
to language designers that there still is uncharted territory in compared to any realistic application or library. On the other
the programming language landscape, and innovative solutions hand, every large project is likely to include a small set of
could be discovered that are germane to requirements in certain core functionalities whose quality, performance, and reliability
special domains. significantly influences the whole system’s; Rosetta Code
programs can be indicative of such core functionalities. In
Evidence in our, as well as others’ (Section VI), analysis
addition, measures of performance are meaningful only on
tends to confirm what advocates of static strong typing have
comparable implementations of algorithmic tasks, and hence
long claimed: that it makes it possible to catch more errors
Rosetta Code’s algorithmic bias helped provide a solid base
earlier, at compile time. But the question remains of which
for comparison of this aspect (Section II-B and RQ3,4). The
process leads to overall higher programmer productivity (or,
size and level of activity of the Rosetta Code community
in a different context, to effective learning): postponing testing
mitigates the threat that contributors to Rosetta Code are
and catching as many errors as possible at compile time, or
not representative of the skills and expertise of real-world
running a prototype as soon as possible while frequently going
programmers. However, to ensure wider generalizability, we
back to fixing and refactoring? The traditional knowledge that
plan to analyze other characteristics of the Rosetta Code
bugs are more expensive to fix the later they are detected is
programming community. To sum up, while some quantitative
not an argument against the “test early” approach, since testing
details of our results may vary on different codebases and
early may be the quickest way to find an error in the first place.
methodologies, the big, mainly qualitative, picture, is likely
This is another area where new trade-offs can be explored by
robust.
selectively—or flexibly [1]—combining featuresthat enhance
compilation or execution. Another potential threat comes from the choice of pro-
gramming languages. Section II-C describes how we selected
V. T HREATS TO VALIDITY languages representative of real-world popularity among major
paradigms. Classifying programming languages into paradigms
Threats to construct validity—are we measuring the right has become harder in recent times, when multi-paradigm
things?—are quite limited given that our research questions, languages are the norm(many programming languages offer

10
procedures, some form of object system, and even func- on three tasks. They conclude that Scala’s functional style
tional features such as closures and list comprehensions). leads to more compact code and comparable performance.
11 Nonetheless, we maintain that paradigms still significantly To eschew the limitations of classroom studies—based on
influence the way in which programs are written, and it is the unrepresentative performance of novice programmers (for
natural to associate major programming languages to a specific instance, in [5], about a third of the student subjects fail the
paradigm based on their Rosetta Code programs. For example, parallel programming task in that they cannot achieve any
even though Python offers classes and other object-oriented speedup)—previous work of ours [20], [21] compared Chapel,
features, practically no solutions in Rosetta Code use them. Cilk, Go, and TBB on 96 solutions to 6 tasks that were checked
Extending the study to more languages and new paradigms for style and performance by notable language experts. [20],
belongs to future work. [21] also introduced language dependency diagrams similar to
those used in the present paper.
VI. R ELATED W ORK A common problem with all the aforementioned studies is
Controlled experiments are a popular approach to lan- that they often target few tasks and solutions, and therefore
guage comparisons: study participants program the same tasks fail to achieve statistical significance or generalizability. The
in different languages while researchers measure features such large sample size in our study minimizes these problems.
as code size and execution or development time. Prechelt [24] Surveys can help characterize the perception of program-
compares 7 programming languages on a single task in 80 ming languages. Meyerovich and Rabkin [16] study the rea-
solutions written by studentsand other volunteers. Measures sons behind language adoption. One key finding is that the
include program size, execution time, memory consumption, intrinsic features of a language (such as reliability) are less
and development time. Findings include: the program written important for adoption when compared to extrinsic ones such
in Perl, Python, REXX, or Tcl is “only half as long” as as existing code, open-source libraries, and previous experi-
written in C, C++, or Java; performance results are more ence. This puts our study into perspective, and shows that some
mixed, but C and C++ are generally faster than Java. The features we investigate are very important to developers (e.g.,
study asks questions similar to ours but is limited by the performanceas second most important attribute). Bissyandé et
small sample size. Languages and their compilers have evolved al. [3] study similar questions: the popularity, interoperability,
since 2000 (when [24] was published), making the results and impact of languages. Their rankings, according to lines
difficult to compare; however, some tendencies (conciseness of of code or usage in projects, may suggest alternatives to the
scripting languages, performance-dominance of C) are visible TIOBE ranking we usedfor selecting languages.
in our study too. Harrison et al. [10] compare the code
quality of C++ against the functional language SML’s on 12 Repository mining, as we have done in this study, has
tasks, finding few significant differences. Our study targets become a customary approach to answering a variety of
a broader set of research questions (only RQ5 is related to questions about programming languages. Bhattacharya and
quality). Hanenberg [9] conducts a study with 49 students over Neamtiu [2] study 4 projects in C and C++ to understand
27 hours of development time comparing static vs. dynamic the impact on software quality, finding an advantage in C++.
type systems, finding no significant differences. In contrast to With similar goals, Ray et al. [25] mine 729 projects in 17
controlled experiments, our approach cannot take development languages from GitHub. They find that strong typing is mod-
time into account. estly better than weak typing, and functional languages have
an advantage over procedural languages. Our study looks at a
Many recent comparative studies have targeted program- broader spectrum of research questions in a more controlled
ming languages for concurrency and parallelism. Studying environment, but our results on failures (RQ5) confirm the
15 students on a single problem, Szafron and Schaeffer [31] superiority of statically strongly typed languages. Other studies
identify a message-passing library that is somewhat superior investigate specialized features of programming languages. For
to higher-level parallel programming, even though the latter example, recent studies by us [6] and others [29] investigate
is more “usable” overall. This highlights the difficulty of the use of contracts and their interplay with other language
reconciling results of different metrics. We do not attempt features such as inheritance. Okur and Dig [22] analyze 655
this in our study, as the suitability of a language for certain open-source applications with parallel programming to identify
projects may depend on external factorsthat assign different adoption trends and usage problems, addressing questions that
weights to different metrics. Other studies [4], [5], [12], are orthogonal to ours.
[13] compare parallel programming approaches (UPC, MPI,
OpenMP, and X10) using mostly small student populations. In
the realm of concurrent programming, a study [28] with 237 VII. C ONCLUSIONS
undergraduate students implementing one program with locks, Programming languages are essential tools for the working
monitors, or transactions suggests that transactions leads to computer scientist, and it is no surprise that what is the “right
the fewest errors. In a usability study with 67 students [19], tool for the job” can be the subject of intense debates. To put
we find advantages of the SCOOP concurrency model over such debates on strong foundations, we must understand how
Java’s monitors. Pankratius et al. [23] compare Scala and features of different languages relate to each other. Our study
Java using 13 students and one software engineer working revealed differences regarding some of the most frequently dis-
11 At the 2012 LASER summer school on “Innovative languages for software
cussed language features—conciseness, performance, failure-
engineering”, Mehdi Jazayeri mentioned the proliferation of multi-paradigm proneness—and is therefore of value to software developers
languages as a disincentive to updating his book on programming language and language designers. The key to having highly significant
concepts [8]. statistical results in our study was the use of a large program

11
chrestomathy: Rosetta Code. The repository can be a valuable [14] C. Jones, Programming Productivity. Mcgraw-Hill College, 1986.
resource also for future programming language research that [15] S. McConnell, Code Complete, 2nd ed. Microsoft Press, 2004.
corroborates, or otherwise complements, our findings. Besides [16] L. A. Meyerovich and A. S. Rabkin, “Empirical analysis of program-
using Rosetta Code, researchers can also improve it (by ming language adoption,” in Proceedings of the 2013 ACM SIGPLAN
correcting any detected errors) and can increase its research International Conference on Object Oriented Programming Systems
Languages & Applications, ser. OOPSLA ’13. New York, NY, USA:
value (by maintaining easily accessible up-to-date statistics). ACM, 2013, pp. 1–18.
Acknowledgments. Thanks to Rosetta Code’s Mike Mol [17] P. Mohagheghi, R. Conradi, O. M. Killi, and H. Schwarz, “An em-
for helpful replies to our questions about the repository. Com- pirical study of software reuse vs. defect-density and stability,” in 26th
International Conference on Software Engineering (ICSE 2004). IEEE
ments by Donald “Paddy” McCarthy helped us fix a problem Computer Society, 2004, pp. 282–292.
with one of the analysis scripts for Python. Other comments [18] S. Nanz and C. A. Furia, “A comparative study of programming
by readers of Slashdot, Reddit, and Hacker News were also languages in Rosetta Code,” http://arxiv.org/abs/1409.0252, September
valuable—occasionally even gracious. We thank members of 2014.
the Chair of Software Engineering for their helpful feedback [19] S. Nanz, F. Torshizi, M. Pedroni, and B. Meyer, “Design of an
on a draft of this paper. This work was partially supported by empirical study for comparing the usability of concurrent programming
ERC grant CME #291389. languages,” in Proceedings of the 2011 International Symposium on
Empirical Software Engineering and Measurement, ser. ESEM ’11.
Washington, DC, USA: IEEE Computer Society, 2011, pp. 325–334.
R EFERENCES [20] S. Nanz, S. West, and K. Soares da Silveira, “Examining the expert
gap in parallel programming,” in Proceedings of the 19th European
[1] M. Bayne, R. Cook, and M. D. Ernst, “Always-available static and
Conference on Parallel Processing (Euro-Par ’13), ser. Lecture Notes
dynamic feedback,” in Proceedings of the 33rd International Conference
in Computer Science, vol. 8097. Berlin, Heidelberg: Springer, 2013,
on Software Engineering, ser. ICSE ’11. New York, NY, USA: ACM,
pp. 434–445.
2011, pp. 521–530.
[21] S. Nanz, S. West, K. Soares da Silveira, and B. Meyer, “Benchmarking
[2] P. Bhattacharya and I. Neamtiu, “Assessing programming language
usability and performance of multicore languages,” in Proceedings of
impact on development and maintenance: A study on C and C++,”
the 7th ACM-IEEE International Symposium on Empirical Software
in Proceedings of the 33rd International Conference on Software
Engineering and Measurement, ser. ESEM ’13. Washington, DC, USA:
Engineering, ser. ICSE ’11. New York, NY, USA: ACM, 2011, pp.
IEEE Computer Society, 2013, pp. 183–192.
171–180.
[22] S. Okur and D. Dig, “How do developers use parallel libraries?” in
[3] T. F. Bissyandé, F. Thung, D. Lo, L. Jiang, and L. Réveillère, “Popular-
Proceedings of the ACM SIGSOFT 20th International Symposium on
ity, interoperability, and impact of programming languages in 100,000
the Foundations of Software Engineering, ser. FSE ’12. New York,
open source projects,” in Proceedings of the 2013 IEEE 37th Annual
NY, USA: ACM, 2012, pp. 54:1–54:11.
Computer Software and Applications Conference, ser. COMPSAC ’13.
Washington, DC, USA: IEEE Computer Society, 2013, pp. 303–312. [23] V. Pankratius, F. Schmidt, and G. Garretón, “Combining functional and
imperative programming for multicore software: an empirical study
[4] F. Cantonnet, Y. Yao, M. M. Zahran, and T. A. El-Ghazawi, “Produc-
evaluating Scala and Java,” in Proceedings of the 2012 International
tivity analysis of the UPC language,” in Proceedings of the 18th Inter-
Conference on Software Engineering, ser. ICSE ’12. IEEE, 2012, pp.
national Parallel and Distributed Processing Symposium, ser. IPDPS
123–133.
’04. Los Alamitos, CA, USA: IEEE Computer Society, 2004.
[24] L. Prechelt, “An empirical comparison of seven programming lan-
[5] K. Ebcioglu, V. Sarkar, T. El-Ghazawi, and J. Urbanic, “An experiment
guages,” IEEE Computer, vol. 33, no. 10, pp. 23–29, Oct. 2000.
in measuring the productivity of three parallel programming languages,”
in Proceedings of the Third Workshop on Productivity and Performance [25] B. Ray, D. Posnett, V. Filkov, and P. T. Devanbu, “A large scale study of
in High-End Computing, ser. P-PHEC ’06, 2006, pp. 30–37. programming languages and code quality in GitHub,” in Proceedings of
the ACM SIGSOFT 20th International Symposium on the Foundations
[6] H.-C. Estler, C. A. Furia, M. Nordio, M. Piccioni, and B. Meyer, “Con-
of Software Engineering. New York, NY, USA: ACM, 2014.
tracts in practice,” in Proceedings of the 19th International Symposium
on Formal Methods (FM), ser. Lecture Notes in Computer Science, vol. [26] E. S. Raymond, The Art of UNIX Programming. Addison-Wesley,
8442. Springer, 2014, pp. 230–246. 2003.
[7] N. E. Fenton and M. Neil, “A critique of software defect prediction [27] Rosetta Code, June 2014. [Online]. Available: http://rosettacode.org/
models,” IEEE Trans. Software Eng., vol. 25, no. 5, pp. 675–689, 1999. [28] C. J. Rossbach, O. S. Hofmann, and E. Witchel, “Is transactional pro-
[8] C. Ghezzi and M. Jazayeri, Programming language concepts, 3rd ed. gramming actually easier?” in Proceedings of the 15th ACM SIGPLAN
Wiley & Sons, 1997. Symposium on Principles and Practice of Parallel Programming, ser.
PPoPP ’10. New York, NY, USA: ACM, 2010, pp. 47–56.
[9] S. Hanenberg, “An experiment about static and dynamic type systems:
Doubts about the positive impact of static type systems on develop- [29] T. W. Schiller, K. Donohue, F. Coward, and M. D. Ernst, “Case
ment time,” in Proceedings of the ACM International Conference on studies and tools for contract specifications,” in Proceedings of the
Object Oriented Programming Systems Languages and Applications, 36th International Conference on Software Engineering, ser. ICSE 2014.
ser. OOPSLA ’10. New York, NY, USA: ACM, 2010, pp. 22–35. New York, NY, USA: ACM, 2014, pp. 596–607.
[10] R. Harrison, L. G. Samaraweera, M. R. Dobie, and P. H. Lewis, [30] A. Stefik and S. Siebert, “An empirical investigation into programming
“Comparing programming paradigms: an evaluation of functional and language syntax,” ACM Transactions on Computing Education, vol. 13,
object-oriented programs,” Software Engineering Journal, vol. 11, no. 4, no. 4, pp. 19:1–19:40, Nov. 2013.
pp. 247–254, July 1996. [31] D. Szafron and J. Schaeffer, “An experiment to measure the usability of
[11] L. Hatton, “Computer programming languages and safety-related sys- parallel programming systems,” Concurrency: Practice and Experience,
tems,” in Proceedings of the 3rd Safety-Critical Systems Symposium. vol. 8, no. 2, pp. 147–166, 1996.
Berlin, Heidelberg: Springer, 1995, pp. 182–196. [32] TIOBE Programming Community Index, July 2014. [Online].
[12] L. Hochstein, V. R. Basili, U. Vishkin, and J. Gilbert, “A pilot study Available: http://www.tiobe.com
to compare programming effort for two parallel programming models,”
Journal of Systems and Software, vol. 81, pp. 1920–1930, 2008.
[13] L. Hochstein, J. Carver, F. Shull, S. Asgari, V. Basili, J. K.
Hollingsworth, and M. V. Zelkowitz, “Parallel programmer productivity:
A case study of novice parallel programmers,” in Proceedings of
the 2005 ACM/IEEE Conference on Supercomputing, ser. SC ’05.
Washington, DC, USA: IEEE Computer Society, 2005, pp. 35–43.

12
 C ONTENTS

I Introduction 1

II Methodology 2
II-A The Rosetta Code repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
II-B Task selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
II-C Language selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
II-D Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
II-E Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
II-F Statistical analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
II-G Visualizations of language comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

III Results 6

IV Discussion 9

V Threats to Validity 10

VI Related Work 11

VII Conclusions 11

References 12

VIII Appendix: Pairwise comparisons 24

VIII-A Conciseness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
VIII-B Conciseness (all tasks) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
VIII-C Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
VIII-D Binary size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
VIII-E Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
VIII-F Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
VIII-G Memory usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
VIII-H Page faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
VIII-I Timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
VIII-J Solutions per task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
VIII-K Other comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
VIII-L Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
VIII-M Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
VIII-N Overall code quality (compilation + execution) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
VIII-O Fault proneness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

13
IX Appendix: Tables and graphs 30
IX-A Lines of code (tasks compiling successfully) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
IX-B Lines of code (all tasks) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
IX-C Comments per line of code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
IX-D Size of binaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
IX-E Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
IX-F Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
IX-G Maximum RAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
IX-H Page faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
IX-I Timeout analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
IX-J Number of solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
IX-K Compilation and execution statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

X Appendix: Plots 71
X-A Lines of code (tasks compiling successfully) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
X-B Lines of code (all tasks) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
X-C Comments per line of code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
X-D Size of binaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
X-E Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
X-F Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
X-G Maximum RAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
X-H Page faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
X-I Timeout analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
X-J Number of solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

L IST OF F IGURES
6 Comparison of lines of code (by minimum). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
8 Comparison of size of executables (by minimum). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
11 Comparison of running time (by minimum) for computing-intensive tasks. . . . . . . . . . . . . . . . . . . . . . . 7
13 Comparison of maximum RAM used (by minimum). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
16 Comparisons of runtime failure proneness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
20 Lines of code (min) of tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
21 Lines of code (min) of tasks compiling successfully (normalized horizontal distances) . . . . . . . . . . . . . . . . 31
23 Lines of code (mean) of tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
24 Lines of code (mean) of tasks compiling successfully (normalized horizontal distances) . . . . . . . . . . . . . . . 33
26 Lines of code (min) of all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
27 Lines of code (min) of all tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
29 Lines of code (mean) of all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
30 Lines of code (mean) of all tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . . . . . . . . 37
32 Comments per line of code (min) of all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
33 Comments per line of code (min) of all tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . 39

14
35 Comments per line of code (mean) of all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
36 Comments per line of code (mean) of all tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . 41
38 Size of binaries (min) of tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
39 Size of binaries (min) of tasks compiling successfully (normalized horizontal distances) . . . . . . . . . . . . . . . 43
41 Size of binaries (mean) of tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
42 Size of binaries (mean) of tasks compiling successfully (normalized horizontal distances) . . . . . . . . . . . . . . 45
44 Performance (min) of tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
45 Performance (min) of tasks running successfully (normalized horizontal distances) . . . . . . . . . . . . . . . . . . 47
47 Performance (mean) of tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
48 Performance (mean) of tasks running successfully (normalized horizontal distances) . . . . . . . . . . . . . . . . . 49
50 Scalability (min) of tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
51 Scalability (min) of tasks running successfully (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . 51
52 Scalability (min) of tasks running successfully (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . 51
54 Scalability (mean) of tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
55 Scalability (mean) of tasks running successfully (normalized horizontal distances) . . . . . . . . . . . . . . . . . . 53
56 Scalability (mean) of tasks running successfully (normalized horizontal distances) . . . . . . . . . . . . . . . . . . 53
58 Maximum RAM usage (min) of scalability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
59 Maximum RAM usage (min) of scalability tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . 55
60 Maximum RAM usage (min) of scalability tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . 55
62 Maximum RAM usage (mean) of scalability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
63 Maximum RAM usage (mean) of scalability tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . 57
64 Maximum RAM usage (mean) of scalability tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . 57
66 Page faults (min) of scalability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
67 Page faults (min) of scalability tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . . . . . . 59
69 Page faults (mean) of scalability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
70 Page faults (mean) of scalability tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . . . . . 61
72 Timeout analysis of scalability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
73 Timeout analysis of scalability tasks (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . . . . . . 63
75 Number of solutions per task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
76 Number of solutions per task (normalized horizontal distances) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
78 Comparisons of compilation status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
80 Comparisons of running status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
82 Comparisons of combined compilation and running status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
84 Comparisons of fault proneness (based on exit status) of solutions that compile correctly and do not timeout . . . 69
88 Lines of code (min) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . 73
89 Lines of code (min) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . 74
90 Lines of code (min) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . 75
91 Lines of code (min) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . 76
92 Lines of code (min) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . 77
93 Lines of code (min) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . 78

15
94 Lines of code (min) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . 79
95 Lines of code (min) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . 80
96 Lines of code (min) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . 81
97 Lines of code (min) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . 82
98 Lines of code (min) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . 82
99 Lines of code (min) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . 83
100 Lines of code (min) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . 83
101 Lines of code (min) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . 83
102 Lines of code (min) of tasks compiling successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . 84
103 Lines of code (mean) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . 85
104 Lines of code (mean) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . 86
105 Lines of code (mean) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . 87
106 Lines of code (mean) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . 88
107 Lines of code (mean) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . 89
108 Lines of code (mean) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . 90
109 Lines of code (mean) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . 91
110 Lines of code (mean) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . 92
111 Lines of code (mean) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . 93
112 Lines of code (mean) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . 94
113 Lines of code (mean) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . 94
114 Lines of code (mean) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . 95
115 Lines of code (mean) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . 95
116 Lines of code (mean) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . 95
117 Lines of code (mean) of tasks compiling successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . 96
118 Lines of code (min) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
119 Lines of code (min) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
120 Lines of code (min) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
121 Lines of code (min) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
122 Lines of code (min) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
123 Lines of code (min) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
124 Lines of code (min) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
125 Lines of code (min) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
126 Lines of code (min) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
127 Lines of code (min) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
128 Lines of code (min) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
129 Lines of code (min) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
130 Lines of code (min) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
131 Lines of code (min) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
132 Lines of code (min) of all tasks (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
133 Lines of code (mean) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

16
134 Lines of code (mean) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
135 Lines of code (mean) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
136 Lines of code (mean) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
137 Lines of code (mean) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
138 Lines of code (mean) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
139 Lines of code (mean) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
140 Lines of code (mean) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
141 Lines of code (mean) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
142 Lines of code (mean) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
143 Lines of code (mean) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
144 Lines of code (mean) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
145 Lines of code (mean) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
146 Lines of code (mean) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
147 Lines of code (mean) of all tasks (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
148 Comments per line of code (min) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . 123
149 Comments per line of code (min) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . 124
150 Comments per line of code (min) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 125
151 Comments per line of code (min) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 126
152 Comments per line of code (min) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 127
153 Comments per line of code (min) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 128
154 Comments per line of code (min) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 129
155 Comments per line of code (min) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 130
156 Comments per line of code (min) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 131
157 Comments per line of code (min) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 132
158 Comments per line of code (min) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 132
159 Comments per line of code (min) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 133
160 Comments per line of code (min) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 133
161 Comments per line of code (min) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 133
162 Comments per line of code (min) of all tasks (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
163 Comments per line of code (mean) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 135
164 Comments per line of code (mean) of all tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 136
165 Comments per line of code (mean) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 137
166 Comments per line of code (mean) of all tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 138
167 Comments per line of code (mean) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 139
168 Comments per line of code (mean) of all tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 140
169 Comments per line of code (mean) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 141
170 Comments per line of code (mean) of all tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 142
171 Comments per line of code (mean) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . 143
172 Comments per line of code (mean) of all tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . 144
173 Comments per line of code (mean) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 144

17
174 Comments per line of code (mean) of all tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 145
175 Comments per line of code (mean) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . 145
176 Comments per line of code (mean) of all tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . 145
177 Comments per line of code (mean) of all tasks (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
178 Size of binaries (min) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . 148
179 Size of binaries (min) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . 149
180 Size of binaries (min) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . 150
181 Size of binaries (min) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . 151
182 Size of binaries (min) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . 152
183 Size of binaries (min) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . 153
184 Size of binaries (min) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . 154
185 Size of binaries (min) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . 155
186 Size of binaries (min) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . 155
187 Size of binaries (min) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . 156
188 Size of binaries (min) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . 156
189 Size of binaries (min) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . 156
190 Size of binaries (min) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . 157
191 Size of binaries (min) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . 157
192 Size of binaries (min) of tasks compiling successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . 157
193 Size of binaries (mean) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . 158
194 Size of binaries (mean) of tasks compiling successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . 159
195 Size of binaries (mean) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . 160
196 Size of binaries (mean) of tasks compiling successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . 161
197 Size of binaries (mean) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . 162
198 Size of binaries (mean) of tasks compiling successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . 163
199 Size of binaries (mean) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . 164
200 Size of binaries (mean) of tasks compiling successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . 165
201 Size of binaries (mean) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . 165
202 Size of binaries (mean) of tasks compiling successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . 166
203 Size of binaries (mean) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . 166
204 Size of binaries (mean) of tasks compiling successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . 166
205 Size of binaries (mean) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . 167
206 Size of binaries (mean) of tasks compiling successfully (Python vs. other languages) . . . . . . . . . . . . . . . . 167
207 Size of binaries (mean) of tasks compiling successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . 167
208 Performance (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 169
209 Performance (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 170
210 Performance (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 171
211 Performance (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 172
212 Performance (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 173
213 Performance (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 174

18
214 Performance (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 175
215 Performance (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 176
216 Performance (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . 177
217 Performance (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . 178
218 Performance (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 178
219 Performance (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 179
220 Performance (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . 179
221 Performance (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . 179
222 Performance (min) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
223 Performance (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 181
224 Performance (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 182
225 Performance (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 183
226 Performance (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 184
227 Performance (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 185
228 Performance (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 186
229 Performance (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . 187
230 Performance (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . 188
231 Performance (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . 189
232 Performance (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . 190
233 Performance (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . 190
234 Performance (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . 191
235 Performance (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . 191
236 Performance (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . 191
237 Performance (mean) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
238 Scalability (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 194
239 Scalability (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 195
240 Scalability (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 196
241 Scalability (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 197
242 Scalability (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 198
243 Scalability (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 199
244 Scalability (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 200
245 Scalability (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 201
246 Scalability (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . 202
247 Scalability (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . 203
248 Scalability (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 203
249 Scalability (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 204
250 Scalability (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . 204
251 Scalability (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . 204
252 Scalability (min) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
253 Scalability (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 206

19
254 Scalability (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 207
255 Scalability (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 208
256 Scalability (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 209
257 Scalability (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 210
258 Scalability (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 211
259 Scalability (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 212
260 Scalability (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 213
261 Scalability (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . 214
262 Scalability (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . 215
263 Scalability (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 215
264 Scalability (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 216
265 Scalability (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . 216
266 Scalability (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . 216
267 Scalability (mean) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
268 Maximum RAM usage (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . 219
269 Maximum RAM usage (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . 220
270 Maximum RAM usage (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . 221
271 Maximum RAM usage (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . 222
272 Maximum RAM usage (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . 223
273 Maximum RAM usage (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . 224
274 Maximum RAM usage (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . 225
275 Maximum RAM usage (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . 226
276 Maximum RAM usage (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . 227
277 Maximum RAM usage (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . 228
278 Maximum RAM usage (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . 228
279 Maximum RAM usage (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . 229
280 Maximum RAM usage (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . 229
281 Maximum RAM usage (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . 229
282 Maximum RAM usage (min) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . 230
283 Maximum RAM usage (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . 231
284 Maximum RAM usage (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . 232
285 Maximum RAM usage (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . 233
286 Maximum RAM usage (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . 234
287 Maximum RAM usage (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . 235
288 Maximum RAM usage (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . 236
289 Maximum RAM usage (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . 237
290 Maximum RAM usage (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . 238
291 Maximum RAM usage (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . 239
292 Maximum RAM usage (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . 240
293 Maximum RAM usage (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . 240

20
294 Maximum RAM usage (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . 241
295 Maximum RAM usage (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . 241
296 Maximum RAM usage (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . 241
297 Maximum RAM usage (mean) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . 242
298 Page faults (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 244
299 Page faults (min) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . 245
300 Page faults (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 246
301 Page faults (min) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 247
302 Page faults (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 248
303 Page faults (min) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 249
304 Page faults (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 250
305 Page faults (min) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 251
306 Page faults (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . 252
307 Page faults (min) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . 253
308 Page faults (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 253
309 Page faults (min) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 254
310 Page faults (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . 254
311 Page faults (min) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . 254
312 Page faults (min) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
313 Page faults (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 256
314 Page faults (mean) of tasks running successfully (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 257
315 Page faults (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 258
316 Page faults (mean) of tasks running successfully (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 259
317 Page faults (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 260
318 Page faults (mean) of tasks running successfully (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . 261
319 Page faults (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 262
320 Page faults (mean) of tasks running successfully (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 263
321 Page faults (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . 264
322 Page faults (mean) of tasks running successfully (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . 265
323 Page faults (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 265
324 Page faults (mean) of tasks running successfully (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . 266
325 Page faults (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . 266
326 Page faults (mean) of tasks running successfully (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . 266
327 Page faults (mean) of tasks running successfully (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
328 Timeout analysis of scalability tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
329 Timeout analysis of scalability tasks (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
330 Timeout analysis of scalability tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
331 Timeout analysis of scalability tasks (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
332 Timeout analysis of scalability tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
333 Timeout analysis of scalability tasks (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274

21
334 Timeout analysis of scalability tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
335 Timeout analysis of scalability tasks (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
336 Timeout analysis of scalability tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
337 Timeout analysis of scalability tasks (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
338 Timeout analysis of scalability tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
339 Timeout analysis of scalability tasks (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
340 Timeout analysis of scalability tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
341 Timeout analysis of scalability tasks (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
342 Timeout analysis of scalability tasks (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
343 Number of solutions per task (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
344 Number of solutions per task (C vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
345 Number of solutions per task (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
346 Number of solutions per task (C# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
347 Number of solutions per task (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
348 Number of solutions per task (F# vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
349 Number of solutions per task (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
350 Number of solutions per task (Go vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
351 Number of solutions per task (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
352 Number of solutions per task (Haskell vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
353 Number of solutions per task (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
354 Number of solutions per task (Java vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
355 Number of solutions per task (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
356 Number of solutions per task (Python vs. other languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
357 Number of solutions per task (all languages) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293

L IST OF TABLES
1 Classification and selection of Rosetta Code tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Rosetta Code ranking: top 20. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 TIOBE index ranking: top 20. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4 Combined ranking: the top-2 languages in each category are selected for the study. . . . . . . . . . . . . . . . . . 3
5 Comparison of lines of code (by minimum). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7 Comparison of size of executables (by minimum). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
9 Computing-intensive tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
10 Comparison of running time (by minimum) for computing-intensive tasks. . . . . . . . . . . . . . . . . . . . . . . 7
12 Comparison of maximum RAM used (by minimum). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
14 Number of solutions that ran without timeout, and their percentage that ran without errors. . . . . . . . . . . . . . 9
15 Comparisons of runtime failure proneness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
17 Number of solutions considered for compilation, and their percentage that compiled without errors. . . . . . . . . 9
18 Names and input size of scalability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
19 Comparison of conciseness (by min) for tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . . 30
22 Comparison of conciseness (by mean) for tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . 32

22
25 Comparison of conciseness (by min) for all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

28 Comparison of conciseness (by mean) for all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

31 Comparison of comments per line of code (by min) for all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

34 Comparison of comments per line of code (by mean) for all tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

37 Comparison of binary size (by min) for tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . . 42

40 Comparison of binary size (by mean) for tasks compiling successfully . . . . . . . . . . . . . . . . . . . . . . . . 44

43 Comparison of performance (by min) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . 46

46 Comparison of performance (by mean) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . 48

49 Comparison of scalability (by min) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

53 Comparison of scalability (by mean) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . 52

57 Comparison of RAM used (by min) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . 54

61 Comparison of RAM used (by mean) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . 56

65 Comparison of page faults (by min) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . 58

68 Comparison of page faults (by mean) for tasks running successfully . . . . . . . . . . . . . . . . . . . . . . . . . . 60

71 Comparison of time-outs (3 minutes) on scalability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

74 Comparison of number of solutions per task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

77 Unpaired comparisons of compilation status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

79 Unpaired comparisons of running status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

81 Unpaired comparisons of combined compilation and running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

83 Unpaired comparisons of runtime fault proneness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

85 Statistics about the compilation process: columns make ok and make ko report percentages relative to solutions
for each language; the columns in between report percentages relative to make ok for each language . . . . . . . 70

86 Statistics about the running process: all columns other than tasks and solutions report percentages relative to
solutions for each language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

87 Statistics about fault proneness: columns error and run ok report percentages relative to solutions for each language 70

23
 For all data processing we used R version 2.14.1. The Wilcoxon signed-rank test and the Mann-Whitney U-test were performed
using package coin version 1.0-23, except for the test statistics W and U that were computed using R’s standard function
wilcox.test; Cohen’s d calculations were performed using package lsr version 0.3.2.

VIII. A PPENDIX : PAIRWISE COMPARISONS

Sections VIII-A to VIII-J describe the complete measured, rendered as graphs and tables, for a number of pairwise comparisons
between programming languages; the actual graphs and table appear in the remaining parts of this appendix.
Each comparison targets a different metric M, including lines of code (conciseness), lines of comments per line of code
(comments), binary size (in kilobytes, where binaries may be native or byte code), CPU user time (in seconds, for different sets
of performance TPERF —“everyday” in the main paper—and scalability TSCAL —“computing-intensive” in the main paper—tasks),
maximum RAM usage (i.e., maximum resident set size, in kilobytes), number of page faults, time outs (with a timeout limit of 3
minutes), and number of Rosetta Code solutions for the same task. Most metrics are normalized, as we detail in the subsections.
A metric may also be such that smaller is better (such as lines of code: the fewer the more concise a program is) or larger
is better (such as comments per line of code: the more the more comments are available). Indeed, comments per line of code
and number of solutions per task are “larger is better” metrics; all other metrics are “smaller is better”. We discuss below how
this feature influences how the results should be read.
Let ` be a programming language, t a task, and M a metric. `M (t) denotes the vector of measures of M, one for each solution
to task t in language `. `M (t) may be empty if there are no solutions to task t in `.
Using this notation, the comparison of programming languages X and Y based on M works as follows. Consider a subset
T of the tasks such that, for every t ∈ T , both X and Y have at least one solution to t. T may be further restricted based on
a measure-dependent criterion, which we describe in the following subsections; for example, Section VIII-A only considers a
task t if both X and Y have at least one solution that compiles without errors (solutions that do not satisfy the criterion are
α and yα for the two languages by aggregating metric M per task using an
discarded). Based on T , we build two data vectors xM M
aggregation function α.
To this end, if M is normalized, the normalization factor νM (t, X,Y ) denotes the smallest value of M for all solutions of t in
X and in Y ; otherwise it is just one:

min (XM (t)YM (t)) if M is normalized and min(XM (t)YM (t)) > 0 ,
νM (t, X,Y ) =
1 otherwise ,
where juxtaposing vectors denotes concatenating them. Note that the normalization factor is one also if M is normalized but
the minimum is zero; this is to avoid divisions by zero when normalizing. (A minimum of zero may occur due to the limited
precision of some measures such as running time.)
We are finally ready to define vectors xMα and yα . The vectors have the same length |T | = |xα | = |yα | and are ordered by
M M M
α α α and in yα corresponding to task t, for t ∈ T :
task; thus, xM (t) and yM (t) denote the value in xM M
α
xM (t) = α(XM (t)/νM (t, X,Y )) ,
α
yM (t) = α(YM (t)/νM (t, X,Y )) .
As aggregation functions, we normally consider both minimum and mean; hence the sets of graphs and tables are often double,
one for each aggregation function. For unstandardized measures, we also define the vectors xαM and yM
α by task t as non-normalized
α α
counterparts to xM and yM :
xαM (t) = α(XM (t)) ,
α
yM (t) = α(YM (t)) .

α and yα determines two graphs and a statistical test.

The data in xM M

• One graph includes line plots of xM α and of yα , with the horizontal axis representing task number and the vertical axis
M
representing values of M (possibly normalized).
For example, Figure 88 includes a graph with normalized values of lines of code aggregated per task by minimum for C and
Python. There you can see that there are close to 350 tasks with at least one solution in both C and Python that compiles
successfully; and that there is a task whose shortest solution in C is over 50 times larger (in lines of code) than its shortest
solution in Python.
• Another graph is a scatter plot of xM α and of yα , namely of points with coordinates (xα (t), yα (t)) for all available tasks
M M M
t ∈ T . This graph also includes a linear regression line fitted using the least squares approach. Since axes have the same
◦
scales in these graphs, a linear regression line that bisects the graph diagonally at 45 would mean that there is no visible
difference in metric M between the two languages. Otherwise, if M is such that “smaller is better”, the flatter or lower the
regression line, the better language Y tends to be compared against language X on metric M. In fact, a flatter or lower

24
 line denotes more points (vX , vY ) with vY < vX than the other way round, or more tasks where Y is better (smaller metric).
Conversely, if M is such that “larger is better”, the steeper or higher the regression line, the better language Y tends to be
compared against language X on metric M.
For example, Figure 89 includes a graph with normalized values of lines of code aggregated per task by minimum for C
and Python. There you can see that most tasks are such that the shortest solution in C is larger than the shortest solution
in Python; the regression line is almost horizontal at ordinate 1.
• The statistical test is a Wilcoxon signed-rank test, a paired non-parametric difference test which assesses whether the mean
ranks of xM α and of yα differ. The test results appear in a table, under column labeled with language X at a row labeled
M
with language Y , and includes various statistics:
1) The p-value estimates the probability that the differences between xM α and yα are due to chance; thus, if p is small
M
(typically at least p < 0.1, but preferably p  0.01) it means that there is a high chance that X and Y exhibit a genuinely
different behavior with respect to metric M. Significant p-values are colored: highly significant (p < 0.01) and significant
but not highly so (“tends to be significant”: 0.01 ≤ p < 0.05).
2) The total sample size N is |xM α | + |xα |, that is twice the number of tasks considered for metric M.
M
3) The test statistics W is the absolute value of the sum of the signed ranks (see a description of the test for details).
4) The related test statistics Z is derivable from W .
5) The effect size, computed as Cohen’s d, which, for statistically significant differences, gives an idea of how large the
difference is. As a rule of thumb, d < 0.3 denotes a small effect size, 0.3 ≤ d < 0.7 denotes a medium effect size, and
d ≥ 0.7 denotes a large effect size. Non-negligible effect sizes are colored: large effect size, medium effect size, and
small (but non vanishing, that is > 0.05) effect size.
6) The difference ∆ = xf α −y α α α
M of the medians of non-normalized vectors xM and xM , which gives an unstandardized measure
f
M
and sign of the size of the overall difference. Namely, if M is such that “smaller is better” and the difference between
X and Y is significant, a positive ∆ indicates that language Y is on average better (smaller) on M than language X.
Conversely, if M is such that “larger is better”, a negative ∆ indicates that language Y is on average better (larger) on M
than language X.
7) The ratio
α ,y
max(xf α
M M)
f
R = sgn(∆)
α ,y
min(xf f α)
M M
of the largest median to the smallest median, with the same sign as ∆. This is another unstandardized measure and sign
of the size of the difference with a more direct interpretation in terms of ratio of average measures. Note that sgn(v) = 1
if v ≥ 0; otherwise sgn(v) = −1.
For example, Table 19 includes a cell comparing C (column header) against Python (row header) for normalized values
of lines of code aggregated per task by minimum. The p-value is practically zero, and hence the differences are highly
significant. The effect size is large (d > 0.9), and hence the magnitude of the differences is considerable. Since the metric
for conciseness is “smaller is better”, a positive ∆ indicates that Python is the more concise language on average; the value
of R further indicates that the average C solution is over 2.9 times longer in lines of code than the average Python solution.
These figures quantitatively confirm what we observed in the line and scatter plots.
We also include a cumulative line plot with all languages at once, which is only meant as a qualitative visualization.

A. Conciseness
The metric for conciseness is non-blank non-comment lines of code, counted using cloc version 1.6.2. The metric is
normalized and smaller is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion only selects
solutions that compile successfully (compilation returns with exit status 0), and only include tasks TLOC which we manually
marked for lines of code count.

B. Conciseness (all tasks)

The metric for conciseness on all tasks is non-blank non-comment lines of code, counted using cloc version 1.6.2. The
metric is normalized and smaller is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion only
include tasks TLOC which we manually marked for lines of code count (but otherwise includes all solutions, including those that
do not compile correctly).

C. Comments
The metric for comments is comment lines of code per non-blank non-comment lines of code, counted using cloc version
1.6.2. The metric is normalized and larger is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion
only selects tasks TLOC which we manually marked for lines of code count (but otherwise includes all solutions, including those
that do not compile correctly).

25
D. Binary size
The metric for binary size is size of binary (in kilobytes), measured using GNU du version 8.13. The metric is normalized
and smaller is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion only selects solutions that
compile successfully (compilation returns with exit status 0 and creates a non-empty binary), and only include tasks TCOMP which
we manually marked for compilation.
The “binary” is either native code or byte code, according to the language. Ruby does not feature in this comparison since
it does not generate byte code, and hence the graphs and tables for this metric do not include Ruby.

E. Performance
The metric for performance is CPU user time (in seconds), measured using GNU time version 1.7. The metric is normalized
and smaller is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion only selects solutions
that execute successfully (execution returns with exit status 0), and only include tasks TPERF which we manually marked for
performance comparison.
We selected the performance tasks based on whether they represent well-defined comparable tasks were measuring performance
makes sense, and we ascertained that all solutions used in the analysis indeed implement the task correctly (and the solutions
are comparable, that is interpret the task consistently and run on comparable inputs).

F. Scalability
The metric for scalability is CPU user time (in seconds), measured using GNU time version 1.7. The metric is normalized
and smaller is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion only selects solutions that
execute successfully (execution returns with exit status 0), and only include tasks TSCAL which we manually marked for scalability
comparison. Table 18 lists the scalability tasks and describes the size n of their inputs in the experiments.
We selected the scalability tasks based on whether they represent well-defined comparable tasks were measuring scalability
makes sense. We ascertained that all solutions used in the analysis indeed implement the task correctly (and the solutions are
comparable, that is interpret the task consistently); and we modified the input to all solutions so that they are uniform across
languages and represent challenging (or at least non-trivial) input sizes.

G. Memory usage
The metric for memory (RAM) usage is maximum resident set size (in kilobytes), measured using GNU time version 1.7.
The metric is normalized and smaller is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion
only selects solutions that execute successfully (execution returns with exit status 0), and only include tasks TSCAL which we
manually marked for scalability comparison.

H. Page faults
The metric for page faults is number of page faults in an execution, measured using GNU time version 1.7. The metric is
normalized and smaller is better. As aggregation functions we consider minimum ‘min’ and mean. The criterion only selects
solutions that execute successfully (execution returns with exit status 0), and only include tasks TSCAL which we manually marked
for scalability comparison.
A number of tests could not be performed due to languages not generating any page faults (all pairs are ties). In those cases,
the metric is immaterial.

I. Timeouts
The metric for timeouts is ordinal and two-valued: a solution receives a value of one if it times out within the allotted time;
otherwise it receives a value of zero. Time outs were detected using GNU timeout version 8.13. The metric is not normalized and
smaller is better. As aggregation function we consider maximum ‘max’, corresponding to letting `(t) = 1 iff all selected solutions
to task t in language ` time out. The criterion only selects solutions that either execute successfully (execution terminates and
returns with exit status 0) or are still running at the timeout limit, and only include tasks TSCAL which we manually marked for
scalability comparison.
The line plots for this metric are actually point plots for better readability. Also for readability, the majority of tasks with
the same value in the languages under comparison correspond to a different color (marked “all” in the legends).

26
 TASK NAME INPUT SIZE n
1 9 billion names of God the integer n = 105
2 Anagrams 100 × unixdict.txt (20.6 MB)
3 Anagrams/Deranged anagrams 100 × unixdict.txt (20.6 MB)
32
4 Arbitrary-precision integers (included) 54

25
5 Combinations 10
6 Count in factors n = 106
7 Cut a rectangle 10 × 10
8 Extensible prime generator 107 th prime
9 Find largest left truncatable prime in a given base 107 th prime
10 Hamming numbers 107 th Hamming number
11 Happy numbers 106 th Happy number
12 Hofstadter Q sequence # flips up to 105 th term
13 Knapsack problem/0-1 input from Rosetta Code task description
14 Knapsack problem/Bounded input from Rosetta Code task description
15 Knapsack problem/Continuous input from Rosetta Code task description
16 Knapsack problem/Unbounded input from Rosetta Code task description
17 Ludic numbers input from Rosetta Code task description
18 LZW compression 100 × unixdict.txt (20.6 MB)
19 Man or boy test n = 16
20 N-queens problem n = 13
21 Perfect numbers first 5 perfect numbers
22 Pythagorean triples perimeter < 108
23 Self-referential sequence n = 106
24 Semordnilap 100 × unixdict.txt
25 Sequence of non-squares non-squares < 106
26 Sorting algorithms/Bead sort n = 104 , nonnegative values < 104
27 Sorting algorithms/Bubble sort n = 3 · 104
28 Sorting algorithms/Cocktail sort n = 3 · 104
29 Sorting algorithms/Comb sort n = 106
30 Sorting algorithms/Counting sort n = 2 · 106 , nonnegative values < 2 · 106
31 Sorting algorithms/Gnome sort n = 3 · 104
32 Sorting algorithms/Heapsort n = 106
33 Sorting algorithms/Insertion sort n = 3 · 104
34 Sorting algorithms/Merge sort n = 106
35 Sorting algorithms/Pancake sort n = 3 · 104
36 Sorting algorithms/Quicksort n = 2 · 106
37 Sorting algorithms/Radix sort n = 2 · 106 , nonnegative values < 2 · 106
38 Sorting algorithms/Selection sort n = 3 · 104
39 Sorting algorithms/Shell sort n = 2 · 106
40 Sorting algorithms/Stooge sort n = 3 · 103
41 Sorting algorithms/Strand sort n = 3 · 104
42 Text processing/1 input from Rosetta Code task description (1.2 MB)
43 Text processing/2 input from Rosetta Code task description (1.2 MB)
44 Topswops n = 12
45 Towers of Hanoi n = 25
46 Vampire number input from Rosetta Code task description

Table 18: Names and input size of scalability tasks

J. Solutions per task

The metric for timeouts is a counting metric: each solution receives a value of one. The metric is not normalized and larger is
better. As aggregation function we consider the sum; hence each task receives a value corresponding to the number of solutions
in Rosetta Code for that task in each language. The criterion only selects tasks TCOMP which we manually marked for compilation
(but otherwise includes all solutions, including those that do not compile correctly).
The line plots for this metric are actually point plots for better readability. Also for readability, the majority of tasks with
the same value in the languages under comparison correspond to a different color (marked “all” in the legends).

K. Other comparisons
Table 77, Table 79, Table 85, and Table 86 display the results of additional statistics comparing programming languages.

L. Compilation
Table 77 and Table 85 give more details about the compilation process.
Table 77 is similar to the previous tables, but it is based on unpaired tests, namely the Mann-Whitney U test—a non-
parametric ordinal test that can be applied to two samples of different size. We first consider all solutions to tasks TCOMP marked
for compilation (regardless of compilation outcome). We then assign an ordinal value to each solution:
0: if the solution compiles without errors (the compiler returns with exit status 0 and, if applicable, creates a non-empty binary)
with the default compilation options;
1: if the solution compiles without errors (the compiler returns with exit status 0 and, if applicable, creates a non-empty binary),
but it requires to set a compilation flag to specify where to find libraries;

27
2: if the solution compiles without errors (the compiler returns with exit status 0 and, if applicable, creates a non-empty binary),
but it requires to specify how to merge or otherwise process multiple input files;
3: if the solution compiles without errors (the compiler returns with exit status 0 and, if applicable, creates a non-empty binary),
but only after applying a patch, which deploys some settings (such as include directives);
4: if the solution compiles without errors (the compiler returns with exit status 0 and, if applicable, creates a non-empty binary),
but only after fixing some simple error (such as a type error, or a missing variable declaration);
5: if the solution does not compile or compiles with errors (the compiler returns with exit status other than 1 or, if applicable,
creates no non-empty binary), even after applying possible patches or fixing.

To make the categories disjoint, we assign the highest possible value in each case, reflecting the fact that the lower the ordinal
value the better. For example, if a solution requires a patch and a merge, we classify it as a patch, which characterizes the most
effort involved in making it compile.
The distinction between patch and fixing is somewhat subjective; it tries to reflect whether the error that had to be rectified
was a trivial omission (patch) or a genuine error (fixing). However, we stopped fixing at simple errors, dropping all programs that
misinterpreted a task description, referenced obviously missing pieces of code, or required substantial structural modifications
to work. All solutions suffering from these problems these received an ordinal value of 5.
For each pair of languages X and Y , a Mann-Whitney U test assessed whether the two samples (ordinal values for language
X vs. ordinal values for language Y ) come from the same population. The test results appear in Table 77, under column labeled
with language X at a row labeled with language Y , and includes various statistics:

1) The p-value is the probability that the two samples come from the same population.
2) The total sample size N is the total number of solutions in language X and Y that received an ordinal value.
3) The test statistics U (see a description of the test for details).
4) The related test statistics Z, derivable from U.
5) The effect size—Cohen’s d.
6) The difference ∆ of the means, which gives a sign to the difference between the samples. Namely, if p is small, a positive
∆ indicates that language Y is on average “better” (fewer compilation problems) than language X.

Table 85 reports, for each language, the number of tasks and solutions considered for compilation; in column make ok, the
percentage of solutions that eventually compiled correctly (ordinal values in the range 0–4); in column make ko, the percentage
of solutions that did not compile correctly (ordinal value 5); in columns none through fix, the percentage of solutions that
eventually compiled correctly for each category corresponding to ordinal values in the range 0–4.

M. Execution
Table 79 and Table 86 give more details about the running process; they are the counterparts to Table 77 and Table 85.
We first consider all solutions to tasks TEXEC marked for execution that we could run. We then assign an ordinal value to
each solution:

0: if the solution runs without errors (it runs and terminates within the timeout, returns with exit status 0, and does not write
anything to standard error);
1: if the solution runs with possible errors but terminates successfully (it runs and terminates within the timeout, returns with
exit status 0, but writes some messages to standard error);
2: if the solution times out (it runs without errors, and it is still running when the time out elapses);
3: if the solution runs with visible error (it runs and terminates within the timeout, returns with exit status other than 0, and
writes some messages to standard error);
4: if the solution crashes (it runs and terminates within the timeout, returns with exit status other than 0, and writes nothing to
standard error, such as a Segfault).

The categories are disjoint and try to reflect increasing levels of problems. We consider terminating without printing error
(a crash) worse than printing some information. Similarly, we consider nontermination without manifest error better than abrupt
termination with error. (In fact, many solutions in this categories were either from correct solutions working on very large
inputs, typically in the scalability tasks, or to correct solutions to interactive tasks were termination is not to be expected.) The
distinctions are somewhat subjective; they try to reflect the difficulty of understanding and possibly debugging an error.
For each pair of languages X and Y , a Mann-Whitney U test assessed whether the two samples (ordinal values for language
X vs. ordinal values for language Y ) come from the same population. The test results appear in Table 79, under column labeled
with language X at a row labeled with language Y , and includes the same statistics as Table 77.
Table 86 reports, for each language, the number of tasks and solutions considered for execution; in columns run ok through
crash, the percentage of solutions for each category corresponding to ordinal values in the range 0–4.

28
N. Overall code quality (compilation + execution)
Table 81 compares the sum of ordinal values assigned to each solution as described in Section VIII-L and Section VIII-M,
for all tasks TEXEC marked for execution (and compilation). The overall score gives an idea of the code quality of Rosetta Code
solutions based on how much we had to do for compilation and for execution.

O. Fault proneness
Table 83 and Table 87 give an idea of the number of defects manifesting themselves as runtime failures; they draw on data
similar to those presented in Section VIII-L and Section VIII-M.
We first consider all solutions to tasks TEXEC marked for execution that we could compile without errors and that ran without
timing out. We then assign an ordinal value to each solution:
0: if the solution runs without errors (it runs and terminates within the timeout with exit status 0);
1: if the solution runs with errors (it runs and terminates within the timeout with exit status other than 0).
The categories are disjoint and do not include solutions that timed out.
For each pair of languages X and Y , a Mann-Whitney U test assessed whether the two samples (ordinal values for language
X vs. ordinal values for language Y ) come from the same population. The test results appear in Table 83, under column labeled
with language X at a row labeled with language Y , and includes the same statistics as Table 77.
Table 87 reports, for each language, the number of tasks and solutions that compiled correctly and ran without timing out;
in columns error and run ok, the percentage of solutions for each category corresponding to ordinal values 1 (error) and 0
(run ok).

Visualizations of language comparisons

To help visualize the results, a graph accompanies each table with the results of statistical significance tests between pairs of
languages. In such graphs, nodes correspond to programming languages. Two nodes `1 and `2 are arranged so that their horizontal
distance is roughly proportional to how `1 and `2 compare according to the statistical test. In contrast, vertical distances are
chosen only to improve readability and carry not meaning.
Precisely, there are two graphs for each measure: unnormalized and normalized. In the unnormalized graph, let pM (`1 , `2 ),
eM (`1 , `2 ), and ∆M (`1 , `2 ) be the p-value, effect size (d or r according to the metric), and difference of medians for the test that
compares `1 to `2 on metric M. If pM (`1 , `2 ) > 0.05 or eM (`1 , `2 ) < 0.05, then the horizontal distance between node `1 and node
`2 is zero or very small (it may still be non-zero to improve the visual layout while satisfying the p constraints as well as possible),
and there is no edge between them. Otherwise, their horizontal distance is proportional to eM (`1 , `2 ) · (1 − pM (`1 , `2 )), and
there is a directed edge to the node corresponding to the “better” language according to M from the other node. To improve
the visual layout, edges that express an ordered pair that is subsumed by others are omitted, that is if `a → `b → `c the edge
from `a to `c is omitted. Paths on the graph following edges in the direction of the arrow list languages in approximate order
from “worse” to “better”. Edges are dotted if they correspond to large significant p-values (0.01 ≤ p < 0.05); they have a color
corresponding to the color assigned to the effect size in the table: large effect size, medium effect size, and small but non
vanishing effect size.
In the normalized companion graphs (called with “normalized horizontal distances” in the captions), arrows and vertical
distances are assigned as in the unnormalized graphs, whereas the horizontal distance are proportional to the mean `αM for each
language ` over all common tasks (that is where each language has at least one solution). Namely, the language with the “worst”
average measure (consistent with whether M is such that “smaller” or “larger” is better) will be farthest on the left, the language
with the “best” average measure will be farthest on the right, with other languages in between proportionally to their rank. Since,
to have a unique baseline, normalized average measures only refer to tasks common to all languages, the normalized horizontal
distances may be inconsistent with the pairwise tests because they refer to a much smaller set of values (sensitive to noise). This
is only visible in the case of performance and scalability tasks, which are often sufficiently many for pairwise comparisons, but
become too few (for example, less than 10) when we only look at the tasks that have implementations in all languages. In these
cases, the unnormalized graphs may be more indicative (and, in any case, the data in the tables is the hard one).
For comparisons based on ordinal values, there is only one kind of graph whose horizontal distances do not have a significant
quantitative meaning but mainly represent an ordering.
Remind that all graphs use approximations and heuristics to build their layout; hence they are mainly meant as qualitative
visualization aids that cannot substitute a detailed analytical reading of the data.

29
 IX. A PPENDIX : TABLES AND GRAPHS
A. Lines of code (tasks compiling successfully)
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 5.433E-01
N 4.640E+02
W 1.245E+04
Z -6.078E-01
d 3.918E-03
∆ -1.500E+00
R -1.054E+00
F# p 0.000E+00 0.000E+00
N 3.640E+02 3.460E+02
W 1.542E+04 1.469E+04
Z 1.083E+01 1.109E+01
d 7.353E-01 9.453E-01
∆ 1.600E+01 1.600E+01
R 2.455E+00 2.600E+00
Go p 3.774E-01 8.210E-02 3.693E-30
N 6.340E+02 4.740E+02 3.800E+02
W 2.501E+04 1.416E+04 3.320E+02
Z 8.827E-01 1.739E+00 -1.141E+01
d 1.554E-01 8.305E-02 6.396E-01
∆ 0.000E+00 0.000E+00 -1.700E+01
R 1.000E+00 1.000E+00 -2.545E+00
Haskell p 0.000E+00 0.000E+00 1.675E-01 0.000E+00
N 5.540E+02 4.320E+02 3.540E+02 5.600E+02
W 3.695E+04 2.275E+04 7.878E+03 3.835E+04
Z 1.398E+01 1.241E+01 1.380E+00 1.431E+01
d 1.071E+00 1.286E+00 8.453E-02 1.255E+00
∆ 2.100E+01 1.900E+01 3.000E+00 2.100E+01
R 2.909E+00 2.727E+00 1.333E+00 2.909E+00
Java p 2.645E-02 4.328E-05 6.839E-26 2.644E-02 1.433E-33
N 4.940E+02 3.900E+02 3.140E+02 4.960E+02 4.400E+02
W 1.618E+04 1.194E+04 1.345E+02 1.631E+04 5.830E+02
Z 2.219E+00 4.089E+00 -1.052E+01 2.220E+00 -1.207E+01
d 2.615E-01 3.186E-01 7.530E-01 1.475E-01 1.052E+00
∆ 1.000E+00 4.000E+00 -1.500E+01 1.500E+00 -2.050E+01
R 1.032E+00 1.143E+00 -2.250E+00 1.048E+00 -2.864E+00
Python p 0.000E+00 0.000E+00 1.132E-05 0.000E+00 2.123E-02 0.000E+00
N 6.860E+02 5.080E+02 4.060E+02 7.000E+02 6.060E+02 5.440E+02
W 5.650E+04 3.172E+04 1.216E+04 5.826E+04 2.264E+04 3.559E+04
Z 1.533E+01 1.362E+01 4.390E+00 1.549E+01 2.304E+00 1.365E+01
d 9.511E-01 1.114E+00 3.588E-01 8.157E-01 2.085E-01 9.381E-01
∆ 2.100E+01 2.100E+01 4.000E+00 2.200E+01 2.000E+00 2.000E+01
R 2.909E+00 3.625E+00 1.571E+00 3.000E+00 1.200E+00 2.818E+00
Ruby p 0.000E+00 0.000E+00 1.342E-02 0.000E+00 7.643E-01 0.000E+00 1.522E-02
N 6.560E+02 5.040E+02 4.000E+02 6.760E+02 5.900E+02 5.400E+02 7.500E+02
W 5.109E+04 3.045E+04 1.041E+04 5.574E+04 1.913E+04 3.421E+04 2.504E+04
Z 1.484E+01 1.309E+01 2.472E+00 1.539E+01 -2.999E-01 1.347E+01 -2.427E+00
d 5.577E-01 8.824E-01 1.026E-01 7.419E-01 1.069E-01 7.625E-01 2.028E-02
∆ 1.900E+01 1.900E+01 2.000E+00 1.900E+01 -2.000E+00 1.700E+01 -3.000E+00
R 2.462E+00 2.727E+00 1.222E+00 2.462E+00 -1.182E+00 2.214E+00 -1.300E+00

Table 19: Comparison of conciseness (by min) for tasks compiling successfully

30
 Ruby

Go

C# Python

C Java F# Haskell

Figure 20: Lines of code (min) of tasks compiling successfully

Ruby

Go Python

C# Haskell

C Java F#
Figure 21: Lines of code (min) of tasks compiling successfully (normalized horizontal distances)

31
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 5.898E-01
N 4.640E+02
W 1.359E+04
Z 5.391E-01
d 1.146E-01
∆ -1.000E+00
R -1.032E+00
F# p 0.000E+00 0.000E+00
N 3.640E+02 3.460E+02
W 1.617E+04 1.470E+04
Z 1.124E+01 1.111E+01
d 7.245E-01 9.136E-01
∆ 1.800E+01 1.500E+01
R 2.500E+00 2.364E+00
Go p 9.736E-02 2.678E-01 1.140E-30
N 6.340E+02 4.740E+02 3.800E+02
W 2.689E+04 1.416E+04 2.565E+02
Z 1.658E+00 1.108E+00 -1.151E+01
d 1.008E-01 4.938E-02 6.507E-01
∆ 0.000E+00 1.000E+00 -1.775E+01
R 1.000E+00 1.032E+00 -2.543E+00
Haskell p 0.000E+00 0.000E+00 3.940E-01 0.000E+00
N 5.540E+02 4.320E+02 3.540E+02 5.600E+02
W 3.728E+04 2.277E+04 7.726E+03 3.892E+04
Z 1.405E+01 1.243E+01 8.524E-01 1.438E+01
d 9.920E-01 1.266E+00 2.400E-02 1.252E+00
∆ 2.200E+01 2.050E+01 2.000E+00 2.200E+01
R 2.692E+00 2.708E+00 1.200E+00 2.692E+00
Java p 3.469E-03 2.188E-04 2.836E-26 3.143E-02 3.333E-34
N 4.940E+02 3.900E+02 3.140E+02 4.960E+02 4.400E+02
W 1.775E+04 1.188E+04 1.090E+02 1.732E+04 5.350E+02
Z 2.923E+00 3.696E+00 -1.060E+01 2.152E+00 -1.219E+01
d 2.972E-01 1.988E-01 5.150E-01 1.594E-01 1.020E+00
∆ 0.000E+00 4.000E+00 -1.500E+01 2.750E+00 -2.000E+01
R 1.000E+00 1.138E+00 -2.250E+00 1.085E+00 -2.538E+00
Python p 0.000E+00 0.000E+00 1.067E-01 0.000E+00 6.908E-01 0.000E+00
N 6.860E+02 5.080E+02 4.060E+02 7.000E+02 6.060E+02 5.440E+02
W 5.647E+04 3.128E+04 1.083E+04 5.837E+04 2.081E+04 3.593E+04
Z 1.515E+01 1.325E+01 1.613E+00 1.539E+01 -3.978E-01 1.337E+01
d 8.660E-01 9.653E-01 2.510E-02 7.540E-01 7.594E-02 6.005E-01
∆ 2.167E+01 2.133E+01 1.667E+00 2.275E+01 0.000E+00 1.950E+01
R 2.625E+00 3.000E+00 1.179E+00 2.750E+00 1.000E+00 2.500E+00
Ruby p 0.000E+00 0.000E+00 6.440E-01 0.000E+00 1.344E-02 0.000E+00 2.928E-01
N 6.560E+02 5.040E+02 4.000E+02 6.760E+02 5.900E+02 5.400E+02 7.500E+02
W 5.063E+04 3.038E+04 8.814E+03 5.612E+04 1.607E+04 3.404E+04 3.007E+04
Z 1.457E+01 1.302E+01 -4.621E-01 1.545E+01 -2.472E+00 1.297E+01 -1.052E+00
d 5.421E-01 7.861E-01 9.270E-02 6.992E-01 3.851E-02 6.834E-01 3.108E-02
∆ 1.850E+01 1.811E+01 0.000E+00 2.000E+01 -2.000E+00 1.783E+01 -2.333E+00
R 2.194E+00 2.304E+00 1.000E+00 2.333E+00 -1.154E+00 2.176E+00 -1.184E+00

Table 22: Comparison of conciseness (by mean) for tasks compiling successfully

32
 Ruby

Go Python

C# C Java F# Haskell

Figure 23: Lines of code (mean) of tasks compiling successfully

Ruby

Python

C# Java Haskell

C Go F#
Figure 24: Lines of code (mean) of tasks compiling successfully (normalized horizontal distances)

33
B. Lines of code (all tasks)
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 4.401E-01
N 5.460E+02
W 1.704E+04
Z -7.720E-01
d 1.075E-01
∆ -3.223E-01
R -1.009E+00
F# p 0.000E+00 0.000E+00
N 4.160E+02 3.760E+02
W 1.948E+04 1.669E+04
Z 1.110E+01 1.131E+01
d 6.994E-01 9.145E-01
∆ 1.940E+01 2.017E+01
R 2.331E+00 2.453E+00
Go p 9.913E-01 2.754E-01 1.161E-31
N 7.360E+02 5.440E+02 4.220E+02
W 3.144E+04 1.774E+04 6.315E+02
Z 1.090E-02 1.091E+00 -1.171E+01
d 3.998E-02 1.160E-01 6.003E-01
∆ 2.133E+00 1.798E+00 -1.905E+01
R 1.054E+00 1.051E+00 -2.295E+00
Haskell p 0.000E+00 0.000E+00 2.036E-01 0.000E+00
N 6.840E+02 5.260E+02 4.120E+02 6.780E+02
W 5.371E+04 3.146E+04 1.024E+04 5.348E+04
Z 1.473E+01 1.266E+01 1.271E+00 1.460E+01
d 7.998E-01 8.051E-01 1.611E-01 7.639E-01
∆ 2.531E+01 2.289E+01 1.534E+00 2.248E+01
R 2.613E+00 2.552E+00 1.117E+00 2.443E+00
Java p 1.298E-03 9.403E-06 1.247E-26 3.332E-04 4.786E-41
N 6.500E+02 5.220E+02 4.040E+02 6.440E+02 6.200E+02
W 2.936E+04 2.107E+04 1.160E+03 2.868E+04 2.399E+03
Z 3.216E+00 4.430E+00 -1.068E+01 3.588E+00 -1.342E+01
d 1.419E-01 6.627E-02 5.330E-01 1.513E-01 9.305E-01
∆ 5.428E+00 3.716E+00 -1.684E+01 2.062E+00 -1.999E+01
R 1.157E+00 1.111E+00 -2.156E+00 1.058E+00 -2.324E+00
Python p 0.000E+00 0.000E+00 2.631E-05 0.000E+00 6.806E-02 0.000E+00
N 7.560E+02 5.520E+02 4.300E+02 7.540E+02 6.960E+02 6.700E+02
W 6.570E+04 3.603E+04 1.324E+04 6.559E+04 2.870E+04 5.211E+04
Z 1.555E+01 1.345E+01 4.203E+00 1.519E+01 1.825E+00 1.462E+01
d 8.906E-01 9.780E-01 1.913E-01 7.315E-01 5.456E-02 9.323E-01
∆ 2.426E+01 2.336E+01 1.656E+00 2.270E+01 -1.287E+00 2.039E+01
R 2.401E+00 2.584E+00 1.126E+00 2.299E+00 -1.081E+00 2.301E+00
Ruby p 0.000E+00 0.000E+00 2.048E-02 0.000E+00 5.191E-01 0.000E+00 1.706E-02
N 7.220E+02 5.500E+02 4.200E+02 7.180E+02 6.700E+02 6.620E+02 7.580E+02
W 5.924E+04 3.479E+04 1.116E+04 6.057E+04 2.400E+04 4.890E+04 2.557E+04
Z 1.491E+01 1.284E+01 2.317E+00 1.513E+01 -6.448E-01 1.402E+01 -2.385E+00
d 5.411E-01 7.852E-01 3.689E-02 6.791E-01 2.995E-02 8.619E-01 1.925E-02
∆ 2.343E+01 2.114E+01 2.667E-01 2.113E+01 -2.334E+00 1.839E+01 -1.306E+00
R 2.315E+00 2.229E+00 1.018E+00 2.198E+00 -1.153E+00 2.013E+00 -1.080E+00

Table 25: Comparison of conciseness (by min) for all tasks

34
 Go Ruby

C# Python

C Java F# Haskell

Figure 26: Lines of code (min) of all tasks

Ruby

Go Python

C C# Java F# Haskell

Figure 27: Lines of code (min) of all tasks (normalized horizontal distances)

35
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 2.617E-01
N 5.460E+02
W 1.959E+04
Z 1.122E+00
d 6.772E-02
∆ 1.839E+00
R 1.048E+00
F# p 0.000E+00 0.000E+00
N 4.160E+02 3.760E+02
W 2.077E+04 1.739E+04
Z 1.180E+01 1.160E+01
d 7.332E-01 8.851E-01
∆ 2.280E+01 2.092E+01
R 2.535E+00 2.465E+00
Go p 2.808E-01 8.728E-01 6.487E-33
N 7.360E+02 5.440E+02 4.220E+02
W 3.481E+04 1.742E+04 4.530E+02
Z 1.079E+00 1.601E-01 -1.195E+01
d 2.530E-02 4.081E-02 6.160E-01
∆ 3.030E+00 9.126E-01 -2.072E+01
R 1.072E+00 1.024E+00 -2.375E+00
Haskell p 0.000E+00 0.000E+00 5.726E-01 0.000E+00
N 6.840E+02 5.260E+02 4.120E+02 6.780E+02
W 5.551E+04 3.281E+04 1.031E+04 5.484E+04
Z 1.510E+01 1.306E+01 5.643E-01 1.489E+01
d 7.255E-01 6.478E-01 6.647E-02 6.996E-01
∆ 2.738E+01 2.321E+01 9.497E-01 2.364E+01
R 2.623E+00 2.482E+00 1.067E+00 2.424E+00
Java p 3.929E-06 2.200E-06 4.297E-28 6.609E-05 3.959E-41
N 6.500E+02 5.220E+02 4.040E+02 6.440E+02 6.200E+02
W 3.254E+04 2.174E+04 1.011E+03 3.115E+04 2.629E+03
Z 4.615E+00 4.734E+00 -1.099E+01 3.990E+00 -1.343E+01
d 2.289E-01 1.069E-01 4.663E-01 1.699E-01 7.125E-01
∆ 6.609E+00 2.702E+00 -1.939E+01 2.189E+00 -2.107E+01
R 1.180E+00 1.075E+00 -2.298E+00 1.058E+00 -2.295E+00
Python p 0.000E+00 0.000E+00 2.070E-01 0.000E+00 8.144E-01 0.000E+00
N 7.560E+02 5.520E+02 4.300E+02 7.540E+02 6.960E+02 6.700E+02
W 6.703E+04 3.578E+04 1.174E+04 6.625E+04 2.822E+04 5.274E+04
Z 1.543E+01 1.326E+01 1.262E+00 1.521E+01 -2.347E-01 1.418E+01
d 8.194E-01 7.497E-01 7.523E-02 6.907E-01 6.446E-02 5.965E-01
∆ 2.426E+01 2.138E+01 -9.269E-02 2.195E+01 -3.488E+00 1.910E+01
R 2.186E+00 2.190E+00 -1.006E+00 2.073E+00 -1.205E+00 2.006E+00
Ruby p 0.000E+00 0.000E+00 4.509E-01 0.000E+00 3.112E-02 0.000E+00 3.708E-01
N 7.220E+02 5.500E+02 4.200E+02 7.180E+02 6.700E+02 6.620E+02 7.580E+02
W 6.003E+04 3.544E+04 9.528E+03 6.219E+04 2.197E+04 4.965E+04 3.107E+04
Z 1.485E+01 1.300E+01 -7.539E-01 1.549E+01 -2.156E+00 1.366E+01 -8.949E-01
d 5.314E-01 7.255E-01 1.101E-01 6.592E-01 1.077E-03 7.406E-01 3.661E-02
∆ 2.488E+01 2.068E+01 -1.327E+00 2.188E+01 -2.866E+00 1.887E+01 1.601E-01
R 2.282E+00 2.094E+00 -1.088E+00 2.145E+00 -1.175E+00 1.953E+00 1.008E+00

Table 28: Comparison of conciseness (by mean) for all tasks

36
 Ruby

Go Python

C# Haskell

C Java F#
Figure 29: Lines of code (mean) of all tasks

Ruby

Python

Haskell

C Go C# Java F#

Figure 30: Lines of code (mean) of all tasks (normalized horizontal distances)

37
C. Comments per line of code
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 1.280E-02
N 5.460E+02
W 5.485E+03
Z 2.489E+00
d 1.196E-01
∆ 7.850E-03
R 1.273E+00
F# p 2.582E-03 1.438E-03
N 4.160E+02 3.760E+02
W 1.762E+03 8.215E+02
Z -3.014E+00 -3.187E+00
d 2.267E-01 2.553E-01
∆ -1.366E-01 -1.407E-01
R -4.644E+00 -5.655E+00
Go p 2.708E-04 1.590E-04 1.787E-02
N 7.360E+02 5.440E+02 4.220E+02
W 8.032E+03 2.822E+03 3.651E+03
Z -3.642E+00 -3.777E+00 2.368E+00
d 1.483E-01 8.830E-02 2.302E-01
∆ -1.739E-02 -1.843E-02 1.214E-01
R -1.478E+00 -1.611E+00 3.336E+00
Haskell p 8.026E-01 9.087E-02 2.919E-03 2.006E-03
N 6.840E+02 5.260E+02 4.120E+02 6.780E+02
W 8.228E+03 2.120E+03 2.292E+03 9.460E+03
Z 2.499E-01 -1.691E+00 2.976E+00 3.089E+00
d 1.156E-01 1.839E-01 1.080E-01 2.662E-02
∆ -1.625E-02 -2.395E-02 1.164E-01 2.825E-03
R -1.430E+00 -1.815E+00 3.086E+00 1.054E+00
Java p 3.675E-03 5.161E-01 4.373E-05 7.217E-06 8.632E-03
N 6.500E+02 5.220E+02 4.040E+02 6.440E+02 6.200E+02
W 8.242E+03 2.432E+03 2.586E+03 9.179E+03 4.854E+03
Z 2.905E+00 -6.493E-01 4.087E+00 4.487E+00 2.626E+00
d 3.301E-02 5.305E-02 2.337E-01 1.655E-01 1.544E-01
∆ 1.460E-02 1.959E-04 1.364E-01 2.487E-02 3.169E-02
R 1.678E+00 1.007E+00 6.420E+00 1.961E+00 2.335E+00
Python p 8.629E-01 1.313E-01 2.052E-04 2.637E-03 1.439E-01 1.072E-01
N 7.560E+02 5.520E+02 4.300E+02 7.540E+02 6.960E+02 6.700E+02
W 8.917E+03 2.174E+03 2.394E+03 1.136E+04 5.716E+03 3.394E+03
Z 1.727E-01 -1.509E+00 3.713E+00 3.007E+00 1.461E+00 -1.611E+00
d 1.303E-01 9.636E-02 2.101E-01 2.011E-02 5.100E-02 1.486E-01
∆ -6.968E-03 -2.533E-02 1.312E-01 4.675E-03 1.070E-02 -2.326E-02
R -1.194E+00 -1.946E+00 4.155E+00 1.098E+00 1.250E+00 -1.943E+00
Ruby p 2.721E-02 2.457E-03 5.496E-03 8.325E-01 3.642E-01 3.449E-03 5.222E-02
N 7.220E+02 5.500E+02 4.200E+02 7.180E+02 6.700E+02 6.620E+02 7.580E+02
W 7.644E+03 2.338E+03 2.738E+03 8.540E+03 5.039E+03 3.474E+03 5.300E+03
Z -2.208E+00 -3.029E+00 2.776E+00 -2.115E-01 -9.074E-01 -2.925E+00 -1.941E+00
d 1.983E-01 1.441E-01 2.170E-01 5.172E-02 5.629E-02 1.178E-01 1.088E-01
∆ -3.413E-02 -3.713E-02 1.123E-01 -2.023E-02 -1.182E-02 -3.990E-02 -1.950E-02
R -1.969E+00 -2.345E+00 2.755E+00 -1.416E+00 -1.216E+00 -2.572E+00 -1.388E+00

Table 31: Comparison of comments per line of code (by min) for all tasks

38
 Java Haskell

C# C Python Go Ruby F#

Figure 32: Comments per line of code (min) of all tasks

Java

C# Python Ruby

C Go Haskell F#

Figure 33: Comments per line of code (min) of all tasks (normalized horizontal distances)

39
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 1.126E-04
N 5.460E+02
W 9.117E+03
Z 3.862E+00
d 1.451E-01
∆ 1.888E-02
R 1.602E+00
F# p 2.433E-02 4.213E-04
N 4.160E+02 3.760E+02
W 2.972E+03 1.110E+03
Z -2.252E+00 -3.526E+00
d 2.223E-01 2.515E-01
∆ -1.314E-01 -1.490E-01
R -3.597E+00 -5.758E+00
Go p 4.530E-03 2.090E-05 4.463E-02
N 7.360E+02 5.440E+02 4.220E+02
W 1.233E+04 3.448E+03 4.752E+03
Z -2.839E+00 -4.255E+00 2.008E+00
d 1.445E-01 1.226E-01 2.258E-01
∆ -1.362E-02 -2.324E-02 1.229E-01
R -1.287E+00 -1.713E+00 3.076E+00
Haskell p 9.046E-01 1.451E-02 1.597E-02 2.478E-03
N 6.840E+02 5.260E+02 4.120E+02 6.780E+02
W 1.294E+04 2.746E+03 3.473E+03 1.338E+04
Z 1.199E-01 -2.444E+00 2.410E+00 3.026E+00
d 1.196E-01 1.826E-01 1.095E-01 3.122E-02
∆ -1.704E-02 -3.144E-02 1.182E-01 -1.459E-03
R -1.344E+00 -1.985E+00 2.880E+00 -1.023E+00
Java p 5.115E-04 9.759E-02 6.576E-04 3.450E-05 1.930E-03
N 6.500E+02 5.220E+02 4.040E+02 6.440E+02 6.200E+02
W 1.254E+04 3.376E+03 3.647E+03 1.258E+04 8.654E+03
Z 3.475E+00 -1.657E+00 3.407E+00 4.141E+00 3.101E+00
d 5.711E-02 4.856E-02 2.280E-01 1.813E-01 1.538E-01
∆ 1.815E-02 -6.076E-03 1.365E-01 2.426E-02 3.681E-02
R 1.623E+00 -1.198E+00 5.000E+00 1.723E+00 2.162E+00
Python p 2.615E-01 5.322E-04 7.370E-03 1.538E-01 7.819E-01 1.694E-03
N 7.560E+02 5.520E+02 4.300E+02 7.540E+02 6.960E+02 6.700E+02
W 1.348E+04 3.376E+03 3.951E+03 1.497E+04 1.058E+04 6.313E+03
Z -1.123E+00 -3.464E+00 2.680E+00 1.426E+00 2.768E-01 -3.139E+00
d 1.452E-01 1.291E-01 1.822E-01 2.651E-02 2.543E-02 1.609E-01
∆ -1.472E-02 -4.460E-02 1.221E-01 -8.336E-03 3.028E-03 -3.570E-02
R -1.315E+00 -2.529E+00 3.067E+00 -1.141E+00 1.048E+00 -2.138E+00
Ruby p 1.955E-02 5.776E-05 2.830E-02 6.545E-01 2.017E-01 1.513E-03 1.701E-01
N 7.220E+02 5.500E+02 4.200E+02 7.180E+02 6.700E+02 6.620E+02 7.580E+02
W 1.205E+04 3.519E+03 3.986E+03 1.217E+04 8.104E+03 5.571E+03 1.096E+04
Z -2.335E+00 -4.022E+00 2.193E+00 -4.475E-01 -1.277E+00 -3.172E+00 -1.372E+00
d 1.663E-01 1.566E-01 2.103E-01 3.913E-02 5.035E-02 1.196E-01 8.777E-02
∆ -3.943E-02 -5.352E-02 1.003E-01 -2.934E-02 -1.537E-02 -4.841E-02 -1.504E-02
R -1.862E+00 -2.786E+00 2.186E+00 -1.538E+00 -1.231E+00 -2.473E+00 -1.216E+00

Table 34: Comparison of comments per line of code (by mean) for all tasks

40
 Python Ruby

Java Haskell

C# C Go F#
Figure 35: Comments per line of code (mean) of all tasks

Python

Java Go

C# C Haskell Ruby F#

Figure 36: Comments per line of code (mean) of all tasks (normalized horizontal distances)

41
D. Size of binaries
LANGUAGE MEASURE C C# F# Go Haskell Java
C# p 0.000E+00
N 4.660E+02
W 2.611E+04
Z 1.349E+01
d 2.669E+00
∆ 6.575E+00
R 2.140E+00
F# p 0.000E+00 3.829E-16
N 3.620E+02 3.460E+02
W 1.239E+04 5.265E+02
Z 1.017E+01 -8.144E+00
d 1.395E+00 1.267E+00
∆ 3.669E+00 -2.751E+00
R 1.429E+00 -1.496E+00
Go p 1.802E-53 1.705E-40 1.270E-32
N 6.300E+02 4.720E+02 3.760E+02
W 0.000E+00 0.000E+00 0.000E+00
Z -1.539E+01 -1.332E+01 -1.189E+01
d 3.639E+00 2.312E+00 2.403E+00
∆ -1.903E+03 -1.946E+03 -1.882E+03
R -1.533E+02 -3.407E+02 -2.178E+02
Haskell p 6.818E-46 3.241E-36 5.508E-30 0.000E+00
N 5.380E+02 4.200E+02 3.440E+02 5.380E+02
W 0.000E+00 0.000E+00 0.000E+00 3.037E+04
Z -1.422E+01 -1.257E+01 -1.138E+01 9.562E+00
d 2.469E+00 2.224E+00 2.544E+00 1.071E+00
∆ -1.377E+03 -1.386E+03 -1.296E+03 4.597E+02
R -1.112E+02 -2.403E+02 -1.508E+02 1.324E+00
Java p 0.000E+00 3.330E-05 0.000E+00 0.000E+00 0.000E+00
N 4.920E+02 3.900E+02 3.140E+02 4.920E+02 4.280E+02
W 2.788E+04 2.484E+03 7.784E+03 3.038E+04 2.300E+04
Z 1.377E+01 4.150E+00 9.691E+00 1.360E+01 1.269E+01
d 3.148E+00 3.638E-01 1.680E+00 3.121E+00 1.591E+00
∆ 7.057E+00 4.103E-01 3.541E+00 1.876E+03 1.420E+03
R 2.284E+00 1.076E+00 1.656E+00 3.414E+02 2.638E+02
Python p 0.000E+00 2.220E-16 0.000E+00 0.000E+00 0.000E+00 2.067E-06
N 6.860E+02 5.100E+02 4.060E+02 6.960E+02 5.880E+02 5.460E+02
W 5.900E+04 5.686E+03 1.240E+04 6.073E+04 4.336E+04 2.076E+03
Z 1.722E+01 8.233E+00 1.151E+01 1.618E+01 1.487E+01 4.747E+00
d 5.686E+00 8.992E-01 1.517E+00 3.430E+00 1.676E+00 3.945E-01
∆ 8.292E+00 1.475E+00 4.394E+00 1.942E+03 1.455E+03 1.231E+00
R 2.937E+00 1.347E+00 2.037E+00 4.527E+02 3.383E+02 1.285E+00

Table 37: Comparison of binary size (by min) for tasks compiling successfully

42
 Go Haskell C F# C# Java Python
Figure 38: Size of binaries (min) of tasks compiling successfully

Python

Java

C#

F#

Go Haskell C
Figure 39: Size of binaries (min) of tasks compiling successfully (normalized horizontal distances)

43
LANGUAGE MEASURE C C# F# Go Haskell Java
C# p 0.000E+00
N 4.660E+02
W 2.634E+04
Z 1.343E+01
d 2.633E+00
∆ 6.595E+00
R 2.130E+00
F# p 0.000E+00 1.234E-16
N 3.620E+02 3.460E+02
W 1.270E+04 5.290E+02
Z 1.014E+01 -8.280E+00
d 1.366E+00 1.267E+00
∆ 3.663E+00 -2.769E+00
R 1.423E+00 -1.491E+00
Go p 1.948E-53 1.765E-40 1.301E-32
N 6.300E+02 4.720E+02 3.760E+02
W 0.000E+00 0.000E+00 0.000E+00
Z -1.539E+01 -1.332E+01 -1.189E+01
d 3.662E+00 2.327E+00 2.413E+00
∆ -1.912E+03 -1.959E+03 -1.896E+03
R -1.529E+02 -3.390E+02 -2.167E+02
Haskell p 7.039E-46 3.276E-36 5.572E-30 0.000E+00
N 5.380E+02 4.200E+02 3.440E+02 5.380E+02
W 0.000E+00 0.000E+00 0.000E+00 3.054E+04
Z -1.422E+01 -1.257E+01 -1.137E+01 9.697E+00
d 2.511E+00 2.256E+00 2.549E+00 1.041E+00
∆ -1.397E+03 -1.406E+03 -1.315E+03 4.488E+02
R -1.120E+02 -2.415E+02 -1.511E+02 1.311E+00
Java p 0.000E+00 7.406E-04 0.000E+00 0.000E+00 0.000E+00
N 4.920E+02 3.900E+02 3.140E+02 4.920E+02 4.280E+02
W 2.727E+04 2.972E+03 8.158E+03 3.038E+04 2.300E+04
Z 1.336E+01 3.374E+00 9.341E+00 1.360E+01 1.268E+01
d 2.863E+00 2.489E-01 1.478E+00 3.141E+00 1.617E+00
∆ 6.920E+00 2.048E-01 3.382E+00 1.887E+03 1.444E+03
R 2.205E+00 1.036E+00 1.598E+00 3.297E+02 2.612E+02
Python p 0.000E+00 1.554E-15 0.000E+00 0.000E+00 0.000E+00 7.684E-07
N 6.860E+02 5.100E+02 4.060E+02 6.960E+02 5.880E+02 5.460E+02
W 5.897E+04 6.516E+03 1.353E+04 6.073E+04 4.336E+04 2.998E+03
Z 1.699E+01 7.981E+00 1.165E+01 1.617E+01 1.486E+01 4.943E+00
d 5.140E+00 8.327E-01 1.571E+00 3.453E+00 1.701E+00 4.218E-01
∆ 8.328E+00 1.474E+00 4.494E+00 1.951E+03 1.474E+03 1.388E+00
R 2.919E+00 1.342E+00 2.061E+00 4.500E+02 3.374E+02 1.317E+00

Table 40: Comparison of binary size (by mean) for tasks compiling successfully

44
 Go Haskell C F# C# Java Python
Figure 41: Size of binaries (mean) of tasks compiling successfully

Python

Java

C#

F#

Go Haskell C
Figure 42: Size of binaries (mean) of tasks compiling successfully (normalized horizontal distances)

45
E. Performance
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 9.150E-04
N 2.800E+01
W 0.000E+00
Z -3.315E+00
d 2.009E+00
∆ -3.000E-02
R -Inf
F# p 1.449E-03 1.507E-02
N 3.000E+01 2.000E+01
W 0.000E+00 2.000E+00
Z -3.185E+00 -2.431E+00
d 1.997E+00 1.679E+00
∆ -1.400E-01 -1.100E-01
R -Inf -4.667E+00
Go p 3.173E-01 2.450E-02 6.451E-04
N 6.600E+01 3.600E+01 3.400E+01
W 1.000E+00 1.370E+02 1.200E+02
Z 1.000E+00 2.249E+00 3.412E+00
d 2.462E-01 3.290E-01 2.127E+00
∆ 0.000E+00 3.500E-02 1.400E-01
R NaN Inf Inf
Haskell p 4.652E-01 4.037E-03 8.791E-04 6.789E-02
N 5.400E+01 3.400E+01 3.200E+01 5.600E+01
W 7.000E+00 1.370E+02 1.185E+02 0.000E+00
Z 7.303E-01 2.875E+00 3.327E+00 -1.826E+00
d 3.284E-01 9.493E-02 2.019E+00 1.124E-01
∆ 0.000E+00 3.000E-02 1.350E-01 0.000E+00
R NaN Inf Inf NaN
Java p 4.089E-04 3.113E-03 4.602E-01 1.330E-03 5.756E-04
N 4.600E+01 3.000E+01 3.000E+01 5.200E+01 4.400E+01
W 2.300E+01 8.000E+00 7.300E+01 5.000E+01 2.100E+01
Z -3.534E+00 -2.956E+00 7.385E-01 -3.210E+00 -3.443E+00
d 6.839E-02 1.320E+00 1.570E-01 2.504E-01 2.652E-01
∆ -1.000E-01 -6.000E-02 5.000E-02 -1.000E-01 -1.000E-01
R -Inf -2.500E+00 1.556E+00 -Inf -Inf
Python p 4.414E-06 2.039E-03 1.336E-03 1.753E-04 7.767E-07 3.183E-05
N 6.800E+01 3.400E+01 3.600E+01 7.400E+01 6.200E+01 5.200E+01
W 3.400E+01 1.140E+02 1.300E+02 1.070E+02 0.000E+00 3.390E+02
Z -4.591E+00 3.084E+00 3.208E+00 -3.752E+00 -4.941E+00 4.160E+00
d 1.845E-01 1.247E+00 1.600E+00 2.334E-01 4.057E-01 2.077E+00
∆ -2.000E-02 2.000E-02 1.300E-01 -2.000E-02 -2.000E-02 8.500E-02
R -Inf 3.000E+00 1.400E+01 -Inf -Inf 6.667E+00
Ruby p 1.499E-05 8.060E-01 3.766E-03 2.879E-04 3.509E-05 2.538E-04 6.991E-07
N 6.200E+01 3.600E+01 3.400E+01 7.200E+01 5.800E+01 5.400E+01 7.800E+01
W 3.100E+01 4.200E+01 1.240E+02 1.050E+02 2.900E+01 3.410E+02 2.400E+01
Z -4.329E+00 -2.456E-01 2.897E+00 -3.626E+00 -4.138E+00 3.658E+00 -4.962E+00
d 3.103E-02 3.044E-01 1.438E+00 3.408E-01 2.520E-01 7.152E-01 1.068E+00
∆ -4.000E-02 -1.000E-02 1.100E-01 -4.000E-02 -4.000E-02 6.000E-02 -2.000E-02
R -Inf -1.333E+00 4.667E+00 -Inf -Inf 2.500E+00 -2.000E+00

Table 43: Comparison of performance (by min) for tasks running successfully

46
 Go

Python

Haskell

F# Java C# Ruby C

Figure 44: Performance (min) of tasks running successfully

Go

Ruby Haskell

F# Java C# Python C

Figure 45: Performance (min) of tasks running successfully (normalized horizontal distances)

47
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 9.259E-04
N 2.800E+01
W 0.000E+00
Z -3.312E+00
d 2.104E+00
∆ -3.000E-02
R -Inf
F# p 1.449E-03 2.174E-02
N 3.000E+01 2.000E+01
W 0.000E+00 5.000E+00
Z -3.185E+00 -2.295E+00
d 1.997E+00 1.597E+00
∆ -1.400E-01 -1.100E-01
R -Inf -4.667E+00
Go p 6.547E-01 8.487E-02 6.451E-04
N 6.600E+01 3.600E+01 3.400E+01
W 2.000E+00 1.250E+02 1.200E+02
Z 4.472E-01 1.723E+00 3.412E+00
d 1.441E-01 3.291E-01 2.127E+00
∆ 0.000E+00 3.500E-02 1.400E-01
R NaN Inf Inf
Haskell p 9.163E-01 3.996E-03 8.791E-04 1.730E-01
N 5.400E+01 3.400E+01 3.200E+01 5.600E+01
W 1.100E+01 1.370E+02 1.185E+02 4.000E+00
Z 1.051E-01 2.879E+00 3.327E+00 -1.363E+00
d 2.371E-01 8.247E-02 2.019E+00 1.156E-01
∆ 0.000E+00 3.000E-02 1.350E-01 0.000E+00
R NaN Inf Inf NaN
Java p 4.060E-04 3.131E-03 4.955E-01 1.337E-03 5.673E-04
N 4.600E+01 3.000E+01 3.000E+01 5.200E+01 4.400E+01
W 2.300E+01 8.000E+00 7.200E+01 5.000E+01 2.100E+01
Z -3.536E+00 -2.955E+00 6.816E-01 -3.208E+00 -3.447E+00
d 7.155E-02 1.323E+00 1.940E-01 2.507E-01 2.627E-01
∆ -1.000E-01 -6.000E-02 4.000E-02 -1.000E-01 -1.000E-01
R -Inf -2.500E+00 1.400E+00 -Inf -Inf
Python p 6.495E-05 2.037E-03 1.339E-03 1.568E-04 9.608E-07 3.192E-05
N 6.800E+01 3.400E+01 3.600E+01 7.400E+01 6.200E+01 5.200E+01
W 6.700E+01 1.415E+02 1.300E+02 1.030E+02 0.000E+00 3.390E+02
Z -3.994E+00 3.085E+00 3.208E+00 -3.780E+00 -4.899E+00 4.159E+00
d 2.588E-01 1.225E+00 1.582E+00 2.285E-01 2.931E-01 2.094E+00
∆ -2.000E-02 1.500E-02 1.263E-01 -2.000E-02 -2.000E-02 8.167E-02
R -Inf 2.000E+00 1.018E+01 -Inf -Inf 5.455E+00
Ruby p 1.665E-05 7.291E-01 5.605E-03 1.372E-03 3.551E-05 2.818E-04 2.722E-06
N 6.200E+01 3.600E+01 3.400E+01 7.200E+01 5.800E+01 5.400E+01 7.800E+01
W 3.100E+01 4.700E+01 1.350E+02 1.310E+02 2.900E+01 3.400E+02 5.450E+01
Z -4.306E+00 -3.464E-01 2.770E+00 -3.201E+00 -4.135E+00 3.631E+00 -4.691E+00
d 2.825E-02 3.041E-01 1.362E+00 3.409E-01 2.522E-01 7.172E-01 7.335E-01
∆ -4.000E-02 -1.000E-02 1.100E-01 -4.000E-02 -4.000E-02 6.000E-02 -2.000E-02
R -Inf -1.333E+00 4.667E+00 -Inf -Inf 2.500E+00 -2.000E+00

Table 46: Comparison of performance (by mean) for tasks running successfully

48
 C

Haskell

Ruby Go

F# Java C# Python

Figure 47: Performance (mean) of tasks running successfully

Ruby Haskell

F# Java C# Python Go

Figure 48: Performance (mean) of tasks running successfully (normalized horizontal distances)

49
F. Scalability
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 1.020E-03
N 4.400E+01
W 2.100E+01
Z -3.285E+00
d 3.279E-01
∆ -1.700E+00
R -7.538E+00
F# p 1.206E-02 7.537E-02
N 2.400E+01 2.200E+01
W 7.000E+00 1.300E+01
Z -2.510E+00 -1.778E+00
d 4.530E-01 6.500E-01
∆ -1.140E+00 -1.130E+00
R -8.600E+00 -2.614E+00
Go p 3.783E-05 2.027E-02 1.565E-02
N 7.400E+01 4.600E+01 2.800E+01
W 5.650E+01 1.980E+02 9.100E+01
Z -4.120E+00 2.321E+00 2.417E+00
d 4.532E-01 3.380E-01 5.775E-01
∆ -1.600E-01 2.010E+00 2.825E+00
R -1.571E+00 6.289E+00 5.593E+00
Haskell p 1.605E-05 8.356E-02 9.292E-01 6.156E-04
N 6.400E+01 3.800E+01 2.200E+01 6.400E+01
W 1.800E+01 5.200E+01 3.400E+01 6.600E+01
Z -4.314E+00 -1.730E+00 8.891E-02 -3.425E+00
d 8.950E-01 2.084E-01 4.241E-01 7.052E-01
∆ -4.580E+00 -2.910E+00 -1.170E+00 -4.410E+00
R -2.449E+01 -2.902E+00 -1.639E+00 -1.308E+01
Java p 2.506E-05 6.612E-01 1.578E-01 1.349E-02 9.809E-02
N 6.800E+01 4.400E+01 2.800E+01 6.800E+01 5.800E+01
W 5.100E+01 1.400E+02 7.500E+01 1.530E+02 2.940E+02
Z -4.214E+00 4.383E-01 1.412E+00 -2.470E+00 1.654E+00
d 3.740E-01 3.643E-01 4.690E-01 5.626E-01 4.237E-01
∆ -5.750E-01 1.105E+00 2.740E+00 -6.450E-01 4.340E+00
R -3.212E+00 1.752E+00 4.914E+00 -2.449E+00 6.636E+00
Python p 6.637E-06 3.332E-02 9.375E-01 5.483E-04 8.936E-01 8.203E-02
N 7.000E+01 4.000E+01 2.400E+01 7.200E+01 6.000E+01 6.400E+01
W 4.000E+01 4.800E+01 3.800E+01 1.040E+02 2.390E+02 1.710E+02
Z -4.505E+00 -2.128E+00 -7.845E-02 -3.456E+00 1.337E-01 -1.739E+00
d 7.044E-01 3.360E-01 3.184E-01 7.026E-01 3.858E-01 1.824E-01
∆ -6.920E+00 -6.375E+00 -2.000E-01 -6.440E+00 6.750E-01 -6.870E+00
R -2.983E+01 -4.253E+00 -1.058E+00 -1.547E+01 1.133E+00 -9.228E+00
Ruby p 1.984E-04 4.274E-03 7.537E-01 2.473E-04 3.600E-01 1.251E-02 5.479E-02
N 6.600E+01 3.800E+01 2.400E+01 6.400E+01 6.000E+01 5.800E+01 6.200E+01
W 6.500E+01 2.400E+01 3.500E+01 6.800E+01 1.880E+02 1.020E+02 1.500E+02
Z -3.721E+00 -2.857E+00 -3.138E-01 -3.665E+00 -9.153E-01 -2.497E+00 -1.921E+00
d 9.986E-01 3.582E-01 1.129E-01 9.836E-01 2.495E-01 2.039E-01 1.983E-02
∆ -1.025E+01 -1.231E+01 -2.900E+00 -8.940E+00 -2.115E+00 -9.230E+00 -3.950E+00
R -3.406E+01 -6.151E+00 -1.398E+00 -1.853E+01 -1.368E+00 -7.940E+00 -1.598E+00

Table 49: Comparison of scalability (by min) for tasks running successfully

50
 Java

F#

Haskell

Python

Ruby C# Go C

Figure 50: Scalability (min) of tasks running successfully

Java C

C# Python Go

F# Ruby Haskell
Figure 51: Scalability (min) of tasks running successfully (normalized horizontal distances)

Python Java C

C# Ruby Haskell Go

Figure 52: Scalability (min) of tasks running successfully (normalized horizontal distances)

51
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 4.061E-03
N 4.400E+01
W 3.800E+01
Z -2.873E+00
d 3.274E-01
∆ -1.820E+00
R -6.600E+00
F# p 2.218E-03 7.537E-02
N 2.400E+01 2.200E+01
W 0.000E+00 1.300E+01
Z -3.059E+00 -1.778E+00
d 4.573E-01 6.528E-01
∆ -1.502E+00 -1.130E+00
R -1.102E+01 -2.614E+00
Go p 5.218E-05 1.779E-01 1.315E-02
N 7.400E+01 4.600E+01 2.800E+01
W 6.800E+01 1.680E+02 9.200E+01
Z -4.046E+00 1.347E+00 2.480E+00
d 4.805E-01 2.749E-01 5.920E-01
∆ -4.000E-01 1.990E+00 2.228E+00
R -2.081E+00 3.584E+00 2.837E+00
Haskell p 3.902E-06 1.959E-02 3.739E-01 1.150E-04
N 6.400E+01 3.800E+01 2.200E+01 6.400E+01
W 4.000E+00 3.700E+01 2.300E+01 4.500E+01
Z -4.617E+00 -2.334E+00 -8.891E-01 -3.857E+00
d 8.685E-01 1.171E-02 1.991E-01 5.156E-01
∆ -7.832E+00 -7.075E+00 -6.775E+00 -7.567E+00
R -3.112E+01 -5.624E+00 -4.702E+00 -1.541E+01
Java p 2.702E-05 8.329E-01 6.404E-02 3.939E-02 3.692E-02
N 6.800E+01 4.400E+01 2.800E+01 6.800E+01 5.800E+01
W 5.200E+01 1.330E+02 8.200E+01 1.770E+02 3.140E+02
Z -4.197E+00 2.110E-01 1.852E+00 -2.060E+00 2.087E+00
d 4.561E-01 3.680E-01 5.048E-01 5.498E-01 4.642E-01
∆ -9.725E-01 1.295E+00 2.740E+00 -4.950E-01 7.755E+00
R -3.992E+00 1.881E+00 4.914E+00 -1.508E+00 1.012E+01
Python p 1.353E-06 1.713E-03 4.802E-01 1.795E-05 6.435E-01 2.366E-02
N 7.000E+01 4.000E+01 2.400E+01 7.200E+01 6.000E+01 6.400E+01
W 2.000E+01 2.100E+01 3.000E+01 6.000E+01 2.100E+02 1.430E+02
Z -4.832E+00 -3.136E+00 -7.060E-01 -4.289E+00 -4.628E-01 -2.263E+00
d 4.288E-01 4.852E-01 3.468E-01 8.657E-01 5.964E-02 1.312E-01
∆ -1.263E+01 -1.793E+01 -7.880E+00 -1.122E+01 -5.107E+00 -1.190E+01
R -3.514E+01 -9.359E+00 -3.291E+00 -1.251E+01 -1.631E+00 -1.017E+01
Ruby p 5.597E-05 2.902E-03 1.823E-01 1.211E-05 8.130E-01 1.591E-02 1.081E-01
N 6.600E+01 3.800E+01 2.400E+01 6.400E+01 6.000E+01 5.800E+01 6.200E+01
W 5.500E+01 2.100E+01 2.200E+01 3.000E+01 2.210E+02 1.060E+02 1.660E+02
Z -4.029E+00 -2.978E+00 -1.334E+00 -4.376E+00 -2.365E-01 -2.411E+00 -1.607E+00
d 1.035E+00 4.273E-01 3.170E-01 8.534E-01 3.905E-01 1.097E-01 1.685E-01
∆ -1.155E+01 -1.445E+01 -6.105E+00 -1.014E+01 -2.950E-01 -1.047E+01 1.320E+00
R -2.279E+01 -6.236E+00 -1.838E+00 -9.361E+00 -1.031E+00 -7.503E+00 1.109E+00

Table 53: Comparison of scalability (by mean) for tasks running successfully

52
 F#

Haskell

Python C#

Ruby Java Go C

Figure 54: Scalability (mean) of tasks running successfully

Java

Haskell

C# Python C

F# Ruby Go

Figure 55: Scalability (mean) of tasks running successfully (normalized horizontal distances)

Python Java C

C# Ruby Haskell Go

Figure 56: Scalability (mean) of tasks running successfully (normalized horizontal distances)

53
G. Maximum RAM
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 7.994E-05
N 4.400E+01
W 5.000E+00
Z -3.945E+00
d 2.022E+00
∆ -7.837E+04
R -2.479E+00
F# p 6.040E-03 9.925E-03
N 2.400E+01 2.200E+01
W 4.000E+00 4.000E+00
Z -2.746E+00 -2.578E+00
d 7.613E-01 1.045E+00
∆ -3.255E+05 -2.931E+05
R -4.453E+00 -5.158E+00
Go p 1.573E-04 7.687E-05 6.319E-03
N 7.400E+01 4.600E+01 2.800E+01
W 1.010E+02 2.680E+02 9.600E+01
Z -3.779E+00 3.954E+00 2.731E+00
d 6.437E-02 3.914E-01 7.883E-01
∆ -5.919E+04 9.602E+04 3.776E+05
R -1.790E+00 4.070E+00 3.458E+00
Haskell p 1.172E-04 8.405E-01 6.188E-02 3.812E-04
N 6.400E+01 3.800E+01 2.200E+01 6.400E+01
W 5.800E+01 1.000E+02 5.400E+01 7.400E+01
Z -3.852E+00 2.012E-01 1.867E+00 -3.553E+00
d 2.874E-01 1.234E-01 6.139E-01 3.144E-01
∆ -5.885E+05 -1.714E+05 3.409E+05 -5.310E+05
R -1.421E+01 -3.496E+00 2.536E+00 -5.614E+00
Java p 1.151E-06 7.994E-05 3.305E-01 3.181E-06 7.101E-03
N 6.800E+01 4.400E+01 2.800E+01 6.800E+01 5.800E+01
W 1.300E+01 5.000E+00 3.700E+01 2.500E+01 9.300E+01
Z -4.864E+00 -3.945E+00 -9.730E-01 -4.659E+00 -2.692E+00
d 8.903E-01 1.427E+00 2.782E-01 5.273E-01 6.167E-01
∆ -3.144E+05 -1.874E+05 1.536E+05 -2.471E+05 3.735E+05
R -5.445E+00 -2.422E+00 1.407E+00 -2.592E+00 2.142E+00
Python p 1.056E-06 3.507E-01 3.417E-02 1.795E-05 9.918E-01 5.982E-03
N 7.000E+01 4.000E+01 2.400E+01 7.200E+01 6.000E+01 6.400E+01
W 1.700E+01 8.000E+01 6.600E+01 6.000E+01 2.330E+02 4.110E+02
Z -4.881E+00 -9.333E-01 2.118E+00 -4.289E+00 1.028E-02 2.749E+00
d 3.292E-01 4.449E-01 9.637E-02 4.168E-01 9.954E-03 2.060E-01
∆ -3.210E+05 -1.616E+05 1.727E+04 -2.439E+05 4.523E+05 -2.405E+04
R -4.999E+00 -2.175E+00 1.032E+00 -2.673E+00 2.914E+00 -1.059E+00
Ruby p 3.245E-06 2.225E-03 5.303E-01 1.437E-05 4.950E-02 2.218E-01 3.601E-02
N 6.600E+01 3.800E+01 2.400E+01 6.400E+01 6.000E+01 5.800E+01 6.200E+01
W 2.000E+01 1.900E+01 4.700E+01 3.200E+01 1.370E+02 2.740E+02 1.410E+02
Z -4.655E+00 -3.058E+00 6.276E-01 -4.338E+00 -1.964E+00 1.222E+00 -2.097E+00
d 4.027E-01 5.246E-01 2.424E-01 5.307E-01 3.014E-01 3.009E-01 6.119E-02
∆ -3.117E+05 -2.579E+05 2.070E+05 -2.392E+05 3.056E+05 -3.913E+04 5.780E+03
R -4.951E+00 -4.991E+00 1.515E+00 -2.468E+00 1.816E+00 -1.107E+00 1.014E+00

Table 57: Comparison of RAM used (by min) for tasks running successfully

54
 Java

Haskell Ruby

F# Python C# Go C

Figure 58: Maximum RAM usage (min) of scalability tasks

Ruby C# C

Java F# Haskell Python Go

Figure 59: Maximum RAM usage (min) of scalability tasks (normalized horizontal distances)

C# C

Ruby Python Java Haskell Go

Figure 60: Maximum RAM usage (min) of scalability tasks (normalized horizontal distances)

55
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 4.010E-05
N 4.400E+01
W 0.000E+00
Z -4.107E+00
d 2.239E+00
∆ -1.850E+05
R -4.076E+00
F# p 4.742E-03 9.925E-03
N 2.400E+01 2.200E+01
W 3.000E+00 4.000E+00
Z -2.824E+00 -2.578E+00
d 7.702E-01 1.046E+00
∆ -3.390E+05 -2.932E+05
R -4.590E+00 -5.159E+00
Go p 2.833E-05 1.623E-04 4.286E-03
N 7.400E+01 4.600E+01 2.800E+01
W 7.400E+01 2.620E+02 9.800E+01
Z -4.187E+00 3.771E+00 2.856E+00
d 3.563E-02 3.879E-01 8.040E-01
∆ -1.437E+05 6.200E+04 3.892E+05
R -2.813E+00 1.356E+00 3.532E+00
Haskell p 6.292E-05 4.688E-01 7.537E-02 3.304E-04
N 6.400E+01 3.800E+01 2.200E+01 6.400E+01
W 5.000E+01 7.700E+01 5.300E+01 7.200E+01
Z -4.002E+00 -7.244E-01 1.778E+00 -3.590E+00
d 4.214E-01 1.444E-02 6.087E-01 3.505E-01
∆ -7.899E+05 -3.803E+05 3.064E+05 -6.345E+05
R -1.697E+01 -2.898E+00 2.194E+00 -3.914E+00
Java p 6.816E-07 6.922E-04 7.776E-01 9.141E-06 1.072E-01
N 6.800E+01 4.400E+01 2.800E+01 6.800E+01 5.800E+01
W 7.000E+00 2.200E+01 4.800E+01 3.800E+01 1.430E+02
Z -4.967E+00 -3.393E+00 -2.825E-01 -4.437E+00 -1.611E+00
d 9.115E-01 4.139E-01 7.520E-02 5.250E-01 2.382E-02
∆ -3.378E+05 -1.094E+05 1.618E+05 -1.786E+05 5.698E+05
R -5.478E+00 -1.446E+00 1.424E+00 -1.709E+00 2.589E+00
Python p 3.217E-07 1.913E-01 7.119E-02 6.050E-06 7.655E-01 5.181E-02
N 7.000E+01 4.000E+01 2.400E+01 7.200E+01 6.000E+01 6.400E+01
W 3.000E+00 7.000E+01 6.200E+01 4.500E+01 2.180E+02 3.680E+02
Z -5.110E+00 -1.307E+00 1.804E+00 -4.525E+00 -2.982E-01 1.945E+00
d 3.711E-01 3.828E-01 1.228E-01 4.469E-01 1.025E-01 6.814E-02
∆ -4.089E+05 -1.420E+05 -8.725E+04 -2.496E+05 5.896E+05 -1.034E+05
R -5.821E+00 -1.541E+00 -1.151E+00 -2.053E+00 2.849E+00 -1.236E+00
Ruby p 5.912E-07 1.124E-02 8.753E-01 1.111E-05 1.846E-01 2.386E-01 2.725E-01
N 6.600E+01 3.800E+01 2.400E+01 6.400E+01 6.000E+01 5.800E+01 6.200E+01
W 1.000E+00 3.200E+01 3.700E+01 2.900E+01 1.680E+02 2.720E+02 1.920E+02
Z -4.994E+00 -2.535E+00 -1.569E-01 -4.394E+00 -1.327E+00 1.178E+00 -1.097E+00
d 4.898E-01 5.879E-01 3.501E-01 6.147E-01 1.666E-01 3.979E-02 5.065E-02
∆ -4.139E+05 -2.077E+05 4.221E+04 -2.470E+05 4.740E+05 -8.003E+04 4.341E+03
R -5.945E+00 -2.058E+00 1.073E+00 -1.930E+00 2.113E+00 -1.201E+00 1.008E+00

Table 61: Comparison of RAM used (by mean) for tasks running successfully

56
 Ruby

Java Python C

F# Haskell C# Go

Figure 62: Maximum RAM usage (mean) of scalability tasks

Ruby C# C

Java F# Haskell Python Go

Figure 63: Maximum RAM usage (mean) of scalability tasks (normalized horizontal distances)

Go

Haskell Ruby Python Java C# C

Figure 64: Maximum RAM usage (mean) of scalability tasks (normalized horizontal distances)

57
H. Page faults
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p NaN
N 4.400E+01
W NaN
Z NaN
d NaN
∆ NaN
R NaN
F# p 4.509E-03 4.509E-03
N 2.400E+01 2.200E+01
W 0.000E+00 0.000E+00
Z -2.840E+00 -2.840E+00
d 6.346E-01 6.668E-01
∆ -5.583E+00 -6.091E+00
R -Inf -Inf
Go p NaN NaN 1.847E-03
N 7.400E+01 4.600E+01 2.800E+01
W NaN NaN 7.800E+01
Z NaN NaN 3.114E+00
d NaN NaN 6.027E-01
∆ NaN NaN 4.929E+00
R NaN NaN Inf
Haskell p 6.051E-07 1.121E-04 5.508E-02 6.568E-07
N 6.400E+01 3.800E+01 2.200E+01 6.400E+01
W 0.000E+00 0.000E+00 1.150E+01 0.000E+00
Z -4.990E+00 -3.863E+00 -1.918E+00 -4.974E+00
d 4.959E+00 5.024E+00 8.042E-01 4.309E+00
∆ -5.719E+00 -5.474E+00 0.000E+00 -5.969E+00
R -Inf -Inf 1.000E+00 -Inf
Java p NaN NaN 1.847E-03 NaN 2.176E-06
N 6.800E+01 4.400E+01 2.800E+01 6.800E+01 5.800E+01
W NaN NaN 7.800E+01 NaN 4.350E+02
Z NaN NaN 3.114E+00 NaN 4.736E+00
d NaN NaN 6.027E-01 NaN 4.501E+00
∆ NaN NaN 4.929E+00 NaN 5.793E+00
R NaN NaN Inf NaN Inf
Python p NaN NaN 4.402E-03 NaN 1.459E-06 NaN
N 7.000E+01 4.000E+01 2.400E+01 7.200E+01 6.000E+01 6.400E+01
W NaN NaN 5.500E+01 NaN 4.650E+02 NaN
Z NaN NaN 2.848E+00 NaN 4.817E+00 NaN
d NaN NaN 6.237E-01 NaN 4.207E+00 NaN
∆ NaN NaN 5.500E+00 NaN 6.000E+00 NaN
R NaN NaN Inf NaN Inf NaN
Ruby p NaN NaN 4.402E-03 NaN 1.282E-06 NaN NaN
N 6.600E+01 3.800E+01 2.400E+01 6.400E+01 6.000E+01 5.800E+01 6.200E+01
W NaN NaN 5.500E+01 NaN 4.650E+02 NaN NaN
Z NaN NaN 2.848E+00 NaN 4.843E+00 NaN NaN
d NaN NaN 6.237E-01 NaN 4.232E+00 NaN NaN
∆ NaN NaN 5.500E+00 NaN 5.867E+00 NaN NaN
R NaN NaN Inf NaN Inf NaN NaN

Table 65: Comparison of page faults (by min) for tasks running successfully

58
 Ruby

Python

Java

Go

C#

Haskell F# C

Figure 66: Page faults (min) of scalability tasks

Ruby

Python

Java

Go

C#

F# Haskell C

Figure 67: Page faults (min) of scalability tasks (normalized horizontal distances)

59
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 3.173E-01
N 4.400E+01
W 1.000E+00
Z 1.000E+00
d 2.194E-01
∆ 3.409E-02
R 4.000E+00
F# p 4.509E-03 4.509E-03
N 2.400E+01 2.200E+01
W 0.000E+00 0.000E+00
Z -2.840E+00 -2.840E+00
d 6.346E-01 6.668E-01
∆ -5.583E+00 -6.091E+00
R -Inf -Inf
Go p 3.173E-01 3.173E-01 1.847E-03
N 7.400E+01 4.600E+01 2.800E+01
W 1.000E+00 1.000E+00 7.800E+01
Z 1.000E+00 1.000E+00 3.114E+00
d 2.325E-01 2.949E-01 6.027E-01
∆ 2.703E-02 1.087E-02 4.929E+00
R Inf Inf Inf
Haskell p 7.291E-07 1.253E-04 5.557E-02 7.490E-07
N 6.400E+01 3.800E+01 2.200E+01 6.400E+01
W 0.000E+00 0.000E+00 1.150E+01 0.000E+00
Z -4.953E+00 -3.835E+00 -1.914E+00 -4.948E+00
d 2.834E-01 3.494E-01 8.268E-01 2.851E-01
∆ -4.976E+01 -7.961E+01 -1.136E-01 -5.004E+01
R -1.593E+03 -6.051E+03 -1.019E+00 -Inf
Java p 3.173E-01 3.173E-01 1.847E-03 NaN 2.408E-06
N 6.800E+01 4.400E+01 2.800E+01 6.800E+01 5.800E+01
W 1.000E+00 1.000E+00 7.800E+01 NaN 4.350E+02
Z 1.000E+00 1.000E+00 3.114E+00 NaN 4.716E+00
d 2.425E-01 3.015E-01 6.027E-01 NaN 2.950E-01
∆ 2.941E-02 1.136E-02 4.929E+00 NaN 5.441E+01
R Inf Inf Inf NaN Inf
Python p 5.930E-01 3.173E-01 3.983E-03 3.173E-01 1.635E-06 3.173E-01
N 7.000E+01 4.000E+01 2.400E+01 7.200E+01 6.000E+01 6.400E+01
W 2.000E+00 1.000E+00 6.500E+01 0.000E+00 4.650E+02 0.000E+00
Z -5.345E-01 1.000E+00 2.879E+00 -1.000E+00 4.794E+00 -1.000E+00
d 2.064E-01 3.162E-01 6.213E-01 2.357E-01 2.922E-01 2.500E-01
∆ -1.500E-01 1.250E-02 5.479E+00 -6.944E-03 5.297E+01 -7.812E-03
R -6.250E+00 Inf 2.640E+02 -Inf Inf -Inf
Ruby p 3.173E-01 3.173E-01 4.402E-03 NaN 1.586E-06 NaN 3.173E-01
N 6.600E+01 3.800E+01 2.400E+01 6.400E+01 6.000E+01 5.800E+01 6.200E+01
W 1.000E+00 1.000E+00 5.500E+01 NaN 4.650E+02 NaN 1.000E+00
Z 1.000E+00 1.000E+00 2.848E+00 NaN 4.800E+00 NaN 1.000E+00
d 2.462E-01 3.244E-01 6.237E-01 NaN 2.914E-01 NaN 2.540E-01
∆ 3.030E-02 1.316E-02 5.500E+00 NaN 5.284E+01 NaN 8.065E-03
R Inf Inf Inf NaN Inf NaN Inf

Table 68: Comparison of page faults (by mean) for tasks running successfully

60
 Ruby

Python

Java

Go

Haskell C#

F# C
Figure 69: Page faults (mean) of scalability tasks

Ruby

Python

Java

Go

C#

F# Haskell C

Figure 70: Page faults (mean) of scalability tasks (normalized horizontal distances)

61
I. Timeout analysis
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 4.550E-02
N 5.400E+01
W 0.000E+00
Z -2.000E+00
d 4.760E-01
∆ -1.481E-01
R -5.000E+00
F# p 4.550E-02 3.173E-01
N 3.400E+01 3.400E+01
W 0.000E+00 2.500E+00
Z -2.000E+00 -1.000E+00
d 6.295E-01 2.717E-01
∆ -2.353E-01 -1.176E-01
R -5.000E+00 -1.667E+00
Go p 1.573E-01 8.326E-02 4.550E-02
N 8.000E+01 5.600E+01 3.800E+01
W 0.000E+00 6.000E+00 1.000E+01
Z -1.414E+00 1.732E+00 2.000E+00
d 2.280E-01 3.224E-01 5.869E-01
∆ -5.000E-02 1.071E-01 2.105E-01
R -3.000E+00 2.500E+00 5.000E+00
Haskell p 5.878E-02 1.000E+00 2.568E-01 5.878E-02
N 7.800E+01 5.000E+01 3.600E+01 7.800E+01
W 4.000E+00 1.050E+01 2.000E+01 4.000E+00
Z -1.890E+00 0.000E+00 1.134E+00 -1.890E+00
d 4.543E-01 0.000E+00 4.186E-01 4.543E-01
∆ -1.282E-01 0.000E+00 1.667E-01 -1.282E-01
R -6.000E+00 1.000E+00 2.500E+00 -6.000E+00
Java p 3.173E-01 1.573E-01 8.326E-02 3.173E-01 2.568E-01
N 6.800E+01 5.200E+01 3.800E+01 6.800E+01 6.800E+01
W 0.000E+00 3.000E+00 6.000E+00 0.000E+00 2.000E+01
Z -1.000E+00 1.414E+00 1.732E+00 -1.000E+00 1.134E+00
d 1.415E-01 2.378E-01 4.049E-01 1.415E-01 2.891E-01
∆ -2.941E-02 7.692E-02 1.579E-01 -2.941E-02 8.824E-02
R -2.000E+00 2.000E+00 2.500E+00 -2.000E+00 2.500E+00
Python p 4.550E-02 6.547E-01 1.797E-01 1.573E-01 3.173E-01 3.173E-01
N 8.000E+01 5.400E+01 3.600E+01 8.000E+01 8.000E+01 7.000E+01
W 0.000E+00 9.000E+00 1.200E+01 0.000E+00 3.000E+01 2.500E+00
Z -2.000E+00 4.472E-01 1.342E+00 -1.414E+00 1.000E+00 -1.000E+00
d 3.818E-01 9.764E-02 4.186E-01 1.883E-01 2.163E-01 2.022E-01
∆ -1.000E-01 3.704E-02 1.667E-01 -5.000E-02 7.500E-02 -5.714E-02
R -5.000E+00 1.250E+00 2.500E+00 -2.000E+00 1.750E+00 -2.000E+00
Ruby p 1.025E-01 4.142E-01 4.550E-02 7.055E-01 1.797E-01 3.173E-01 1.000E+00
N 7.800E+01 5.200E+01 3.400E+01 7.800E+01 7.600E+01 6.800E+01 8.000E+01
W 3.500E+00 1.400E+01 1.000E+01 1.200E+01 1.200E+01 2.500E+00 1.800E+01
Z -1.633E+00 8.165E-01 2.000E+00 -3.780E-01 1.342E+00 -1.000E+00 0.000E+00
d 3.872E-01 2.103E-01 6.295E-01 8.864E-02 2.228E-01 2.054E-01 0.000E+00
∆ -1.026E-01 7.692E-02 2.353E-01 -2.564E-02 7.895E-02 -5.882E-02 0.000E+00
R -5.000E+00 1.667E+00 5.000E+00 -1.333E+00 1.750E+00 -2.000E+00 1.000E+00

Table 71: Comparison of time-outs (3 minutes) on scalability tasks

62
 Java

Python Ruby

F# C# Haskell Go C

Figure 72: Timeout analysis of scalability tasks

Java C

Python Ruby

F# C# Go Haskell

Figure 73: Timeout analysis of scalability tasks (normalized horizontal distances)

63
J. Number of solutions
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 0.000E+00
N 1.488E+03
W 2.168E+04
Z 9.277E+00
d 2.975E-01
∆ 2.285E-01
R 1.480E+00
F# p 0.000E+00 1.791E-09
N 1.488E+03 1.488E+03
W 3.365E+04 1.177E+04
Z 1.245E+01 6.016E+00
d 5.015E-01 1.992E-01
∆ 3.629E-01 1.344E-01
R 2.063E+00 1.394E+00
Go p 1.348E-01 4.389E-13 1.069E-30
N 1.488E+03 1.488E+03 1.488E+03
W 9.416E+03 6.702E+03 2.973E+03
Z 1.495E+00 -7.243E+00 -1.152E+01
d 4.663E-02 2.622E-01 4.758E-01
∆ 3.629E-02 -1.922E-01 -3.266E-01
R 1.054E+00 -1.404E+00 -1.957E+00
Haskell p 4.101E-01 5.660E-13 1.560E-29 6.990E-01
N 1.488E+03 1.488E+03 1.488E+03 1.488E+03
W 1.111E+04 4.853E+03 2.518E+03 1.086E+04
Z 8.236E-01 -7.208E+00 -1.128E+01 -3.866E-01
d 7.454E-03 2.570E-01 4.324E-01 3.391E-02
∆ 6.720E-03 -2.218E-01 -3.562E-01 -2.957E-02
R 1.010E+00 -1.466E+00 -2.043E+00 -1.044E+00
Java p 8.732E-06 1.621E-07 2.044E-23 1.997E-03 8.913E-04
N 1.488E+03 1.488E+03 1.488E+03 1.488E+03 1.488E+03
W 1.440E+04 6.256E+03 3.726E+03 1.349E+04 1.215E+04
Z 4.446E+00 -5.238E+00 -9.971E+00 3.091E+00 3.323E+00
d 1.294E-01 1.612E-01 3.571E-01 8.820E-02 1.089E-01
∆ 1.048E-01 -1.237E-01 -2.581E-01 6.855E-02 9.812E-02
R 1.175E+00 -1.260E+00 -1.756E+00 1.114E+00 1.164E+00
Python p 3.717E-18 4.154E-41 8.135E-53 3.619E-20 1.041E-17 4.076E-29
N 1.488E+03 1.488E+03 1.488E+03 1.488E+03 1.488E+03 1.488E+03
W 6.380E+03 3.192E+03 1.192E+03 5.853E+03 6.483E+03 3.950E+03
Z -8.687E+00 -1.343E+01 -1.530E+01 -9.199E+00 -8.569E+00 -1.120E+01
d 2.978E-01 5.134E-01 6.534E-01 3.368E-01 2.870E-01 3.906E-01
∆ -3.374E-01 -5.659E-01 -7.003E-01 -3.737E-01 -3.441E-01 -4.422E-01
R -1.479E+00 -2.189E+00 -3.051E+00 -1.559E+00 -1.493E+00 -1.738E+00
Ruby p 2.206E-02 6.116E-26 4.102E-40 8.831E-04 4.289E-03 8.063E-11 3.735E-13
N 1.488E+03 1.488E+03 1.488E+03 1.488E+03 1.488E+03 1.488E+03 1.488E+03
W 1.022E+04 3.386E+03 2.742E+03 8.373E+03 9.946E+03 5.436E+03 2.327E+04
Z -2.289E+00 -1.053E+01 -1.326E+01 -3.325E+00 -2.856E+00 -6.499E+00 7.265E+00
d 8.581E-02 3.472E-01 5.213E-01 1.291E-01 8.564E-02 1.994E-01 2.122E-01
∆ -7.930E-02 -3.078E-01 -4.422E-01 -1.156E-01 -8.602E-02 -1.841E-01 2.581E-01
R -1.113E+00 -1.647E+00 -2.295E+00 -1.173E+00 -1.123E+00 -1.307E+00 1.329E+00

Table 74: Comparison of number of solutions per task

64
 Haskell

Go

F# C# Java C Ruby Python

Figure 75: Number of solutions per task

Haskell

F# C# Java Go C Ruby Python

Figure 76: Number of solutions per task (normalized horizontal distances)

65
K. Compilation and execution statistics
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 2.210E-02
N 8.780E+02
U 8.511E+04
Z -2.289E+00
d 1.714E-01
∆ -3.091E-01
F# p 3.302E-06 3.089E-09
N 7.780E+02 6.080E+02
U 7.822E+04 5.583E+04
Z 4.651E+00 5.927E+00
d 2.440E-01 4.392E-01
∆ 4.210E-01 7.301E-01
Go p 3.285E-04 6.224E-07 8.050E-02
N 1.021E+03 8.510E+02 7.510E+02
U 1.446E+05 1.030E+05 5.936E+04
Z 3.592E+00 4.984E+00 -1.748E+00
d 8.553E-02 2.607E-01 1.576E-01
∆ 1.543E-01 4.634E-01 -2.667E-01
Haskell p 5.511E-13 2.488E-05 6.446E-21 1.741E-20
N 1.043E+03 8.730E+02 7.730E+02 1.016E+03
U 1.033E+05 7.753E+04 4.107E+04 9.008E+04
Z -7.212E+00 -4.216E+00 -9.382E+00 -9.277E+00
d 4.832E-01 3.183E-01 7.508E-01 5.734E-01
∆ -8.882E-01 -5.791E-01 -1.309E+00 -1.043E+00
Java p 9.520E-01 2.687E-01 6.655E-06 5.570E-04 2.318E-06
N 9.700E+02 8.000E+02 7.000E+02 9.430E+02 9.650E+02
U 1.166E+05 8.213E+04 4.719E+04 9.902E+04 1.345E+05
Z -6.024E-02 1.106E+00 -4.504E+00 -3.452E+00 4.724E+00
d 1.628E-01 6.199E-03 3.848E-01 2.430E-01 2.851E-01
∆ -3.214E-01 -1.229E-02 -7.423E-01 -4.757E-01 5.668E-01
Python p 4.132E-03 4.639E-08 1.521E-02 5.835E-01 0.000E+00 1.935E-04
N 1.299E+03 1.129E+03 1.029E+03 1.272E+03 1.294E+03 1.221E+03
U 2.194E+05 1.610E+05 9.040E+04 1.897E+05 2.675E+05 1.914E+05
Z 2.868E+00 5.465E+00 -2.427E+00 -5.482E-01 1.133E+01 3.727E+00
d 1.572E-01 3.685E-01 1.239E-01 5.988E-02 7.173E-01 3.359E-01
∆ 2.469E-01 5.560E-01 -1.741E-01 9.258E-02 1.135E+00 5.683E-01
Ruby p 2.887E-14 0.000E+00 1.889E-01 1.254E-04 0.000E+00 2.491E-13 8.535E-08
N 1.105E+03 9.350E+02 8.350E+02 1.078E+03 1.100E+03 1.027E+03 1.356E+03
U 1.850E+05 1.333E+05 7.676E+04 1.588E+05 2.172E+05 1.564E+05 2.549E+05
Z 7.603E+00 9.398E+00 1.314E+00 3.835E+00 1.456E+01 7.319E+00 5.355E+00
d 4.150E-01 6.588E-01 1.620E-01 3.196E-01 9.937E-01 5.731E-01 2.957E-01
∆ 6.269E-01 9.359E-01 2.059E-01 4.725E-01 1.515E+00 9.482E-01 3.800E-01

Table 77: Unpaired comparisons of compilation status

Python

Haskell C# Java C Go F# Ruby

Figure 78: Comparisons of compilation status

66
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 7.612E-01
N 7.490E+02
U 6.718E+04
Z -3.040E-01
d 4.167E-02
∆ 4.716E-02
F# p 1.522E-01 1.134E-01
N 6.780E+02 5.430E+02
U 5.469E+04 3.835E+04
Z 1.432E+00 1.583E+00
d 1.351E-01 1.067E-01
∆ 1.547E-01 1.075E-01
Go p 3.141E-07 8.096E-08 2.259E-03
N 8.690E+02 7.340E+02 6.630E+02
U 1.070E+05 7.566E+04 5.478E+04
Z 5.115E+00 5.365E+00 3.054E+00
d 3.887E-01 4.017E-01 2.810E-01
∆ 3.876E-01 3.404E-01 2.329E-01
Haskell p 1.594E-02 9.047E-03 5.384E-01 4.348E-03
N 8.680E+02 7.330E+02 6.620E+02 8.530E+02
U 1.005E+05 7.070E+04 5.125E+04 8.463E+04
Z 2.410E+00 2.610E+00 6.152E-01 -2.852E+00
d 2.001E-01 1.760E-01 6.403E-02 2.092E-01
∆ 2.156E-01 1.685E-01 6.097E-02 -1.719E-01
Java p 7.743E-01 9.169E-01 1.015E-01 1.165E-07 7.911E-03
N 7.810E+02 6.460E+02 5.750E+02 7.660E+02 7.650E+02
U 7.424E+04 5.185E+04 3.764E+04 6.169E+04 6.644E+04
Z -2.868E-01 -1.044E-01 -1.637E+00 -5.299E+00 -2.656E+00
d 9.449E-03 3.386E-02 1.337E-01 4.124E-01 2.022E-01
∆ 1.107E-02 -3.610E-02 -1.436E-01 -3.765E-01 -2.046E-01
Python p 9.656E-01 8.382E-01 1.263E-01 2.027E-08 6.433E-03 7.336E-01
N 1.193E+03 1.058E+03 9.870E+02 1.178E+03 1.177E+03 1.090E+03
U 1.662E+05 1.160E+05 8.433E+04 1.386E+05 1.489E+05 1.285E+05
Z 4.308E-02 2.042E-01 -1.529E+00 -5.610E+00 -2.725E+00 3.403E-01
d 1.807E-02 2.392E-02 1.214E-01 3.673E-01 1.840E-01 8.799E-03
∆ 2.097E-02 -2.620E-02 -1.337E-01 -3.666E-01 -1.947E-01 9.902E-03
Ruby p 3.597E-01 3.048E-01 4.397E-01 5.695E-06 8.662E-02 2.544E-01 3.264E-01
N 1.005E+03 8.700E+02 7.990E+02 9.900E+02 9.890E+02 9.020E+02 1.314E+03
U 1.275E+05 8.911E+04 6.480E+04 1.069E+05 1.146E+05 9.860E+04 2.163E+05
Z 9.159E-01 1.026E+00 -7.727E-01 -4.537E+00 -1.714E+00 1.140E+00 9.813E-01
d 6.754E-02 2.850E-02 7.204E-02 3.266E-01 1.349E-01 6.021E-02 5.073E-02
∆ 7.753E-02 3.037E-02 -7.714E-02 -3.100E-01 -1.381E-01 6.647E-02 5.657E-02

Table 79: Unpaired comparisons of running status

Ruby

Python

Java

F#

C#

C Haskell Go
Figure 80: Comparisons of running status

67
LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 5.525E-05
N 7.490E+02
U 5.690E+04
Z -4.032E+00
d 3.231E-01
∆ -5.362E-01
F# p 2.097E-04 1.676E-11
N 6.780E+02 5.430E+02
U 6.023E+04 4.738E+04
Z 3.707E+00 6.732E+00
d 3.497E-01 6.970E-01
∆ 5.022E-01 1.038E+00
Go p 3.584E-09 0.000E+00 1.479E-01
N 8.690E+02 7.340E+02 6.630E+02
U 1.132E+05 8.916E+04 5.315E+04
Z 5.902E+00 9.507E+00 1.447E+00
d 4.174E-01 7.971E-01 6.011E-02
∆ 5.655E-01 1.102E+00 6.338E-02
Haskell p 1.604E-11 5.731E-02 3.464E-20 7.569E-38
N 8.680E+02 7.330E+02 6.620E+02 8.530E+02
U 7.077E+04 6.034E+04 3.018E+04 4.915E+04
Z -6.738E+00 -1.901E+00 -9.204E+00 -1.286E+01
d 4.712E-01 1.379E-01 8.549E-01 9.566E-01
∆ -7.715E-01 -2.354E-01 -1.274E+00 -1.337E+00
Java p 8.076E-01 2.447E-04 2.032E-03 8.684E-08 5.558E-09
N 7.810E+02 6.460E+02 5.750E+02 7.660E+02 7.650E+02
U 7.561E+04 6.001E+04 3.474E+04 5.878E+04 8.859E+04
Z 2.435E-01 3.668E+00 -3.086E+00 -5.352E+00 5.830E+00
d 3.523E-02 2.845E-01 3.919E-01 4.659E-01 4.315E-01
∆ -5.677E-02 4.794E-01 -5.589E-01 -6.223E-01 7.148E-01
Python p 6.160E-02 7.164E-05 1.478E-08 6.179E-18 3.997E-15 2.882E-01
N 1.193E+03 1.058E+03 9.870E+02 1.178E+03 1.177E+03 1.090E+03
U 1.560E+05 1.321E+05 6.874E+04 1.171E+05 2.015E+05 1.226E+05
Z -1.869E+00 3.971E+00 -5.664E+00 -8.629E+00 7.855E+00 -1.062E+00
d 5.622E-02 3.071E-01 4.509E-01 5.161E-01 4.657E-01 1.714E-02
∆ -8.158E-02 4.546E-01 -5.837E-01 -6.471E-01 6.900E-01 -2.481E-02
Ruby p 2.424E-02 4.044E-11 4.377E-02 2.010E-05 0.000E+00 3.572E-02 1.187E-05
N 1.005E+03 8.700E+02 7.990E+02 9.900E+02 9.890E+02 9.020E+02 1.314E+03
U 1.336E+05 1.076E+05 6.123E+04 1.043E+05 1.611E+05 1.024E+05 2.385E+05
Z 2.253E+00 6.602E+00 -2.016E+00 -4.264E+00 1.006E+01 2.100E+00 4.380E+00
d 1.640E-01 5.263E-01 2.178E-01 2.780E-01 6.855E-01 2.046E-01 2.383E-01
∆ 2.349E-01 7.711E-01 -2.673E-01 -3.306E-01 1.006E+00 2.917E-01 3.165E-01

Table 81: Unpaired comparisons of combined compilation and running

Python

Haskell Java

C# C Ruby F# Go

Figure 82: Comparisons of combined compilation and running status

68
 LANGUAGE MEASURE C C# F# Go Haskell Java Python
C# p 3.687E-02
N 6.370E+02
U 5.060E+04
Z 2.087E+00
d 1.703E-01
∆ 5.215E-02
F# p 5.047E-01 2.038E-01
N 6.060E+02 4.610E+02
U 4.280E+04 2.555E+04
Z 6.671E-01 -1.271E+00
d 5.662E-02 1.187E-01
∆ 1.834E-02 -3.381E-02
Go p 1.944E-08 1.129E-03 4.138E-06
N 7.800E+02 6.350E+02 6.040E+02
U 8.402E+04 5.036E+04 4.543E+04
Z 5.617E+00 3.256E+00 4.604E+00
d 4.104E-01 2.673E-01 3.980E-01
∆ 1.048E-01 5.261E-02 8.641E-02
Haskell p 5.836E-03 7.483E-01 8.328E-02 1.787E-03
N 7.670E+02 6.220E+02 5.910E+02 7.650E+02
U 7.783E+04 4.656E+04 4.206E+04 6.977E+04
Z 2.757E+00 3.209E-01 1.732E+00 -3.123E+00
d 2.000E-01 2.630E-02 1.483E-01 2.272E-01
∆ 5.883E-02 6.681E-03 4.049E-02 -4.592E-02
Java p 3.860E-01 6.292E-03 1.732E-01 4.081E-10 5.300E-04
N 6.880E+02 5.430E+02 5.120E+02 6.860E+02 6.730E+02
U 5.674E+04 3.379E+04 3.061E+04 5.040E+04 5.128E+04
Z -8.668E-01 -2.732E+00 -1.362E+00 -6.251E+00 -3.465E+00
d 6.671E-02 2.370E-01 1.221E-01 4.957E-01 2.712E-01
∆ -2.283E-02 -7.498E-02 -4.117E-02 -1.276E-01 -8.166E-02
Python p 3.316E-01 3.058E-03 1.408E-01 4.591E-11 1.132E-04 9.521E-01
N 1.066E+03 9.210E+02 8.900E+02 1.064E+03 1.051E+03 9.720E+02
U 1.291E+05 7.692E+04 6.968E+04 1.147E+05 1.167E+05 1.004E+05
Z -9.710E-01 -2.962E+00 -1.473E+00 -6.584E+00 -3.860E+00 6.003E-02
d 6.171E-02 2.215E-01 1.154E-01 4.277E-01 2.501E-01 4.178E-03
∆ -2.135E-02 -7.350E-02 -3.969E-02 -1.261E-01 -8.018E-02 1.481E-03
Ruby p 5.891E-01 9.674E-03 2.601E-01 6.770E-10 7.194E-04 6.778E-01 6.576E-01
N 9.070E+02 7.620E+02 7.310E+02 9.050E+02 8.920E+02 8.130E+02 1.191E+03
U 9.964E+04 5.938E+04 5.377E+04 8.862E+04 9.011E+04 7.743E+04 1.757E+05
Z -5.401E-01 -2.587E+00 -1.126E+00 -6.171E+00 -3.382E+00 4.154E-01 4.432E-01
d 3.620E-02 2.012E-01 9.143E-02 4.232E-01 2.307E-01 3.024E-02 2.591E-02
∆ -1.228E-02 -6.443E-02 -3.062E-02 -1.170E-01 -7.111E-02 1.055E-02 9.070E-03

Table 83: Unpaired comparisons of runtime fault proneness

Ruby

Python

Java F#

C C# Haskell Go

Figure 84: Comparisons of fault proneness (based on exit status) of solutions that compile correctly and do not timeout

69
 tasks solutions make ok none libs merge patch fix make ko
C 398 524 0.85 0.68 0.21 0.03 0.06 0.02 0.15
C# 288 354 0.90 0.58 0.08 0.00 0.34 0.00 0.10
F# 216 254 0.95 0.82 0.00 0.00 0.18 0.00 0.05
Go 392 497 0.89 0.82 0.00 0.01 0.17 0.00 0.11
Haskell 358 519 0.84 0.47 0.00 0.06 0.47 0.00 0.16
Java 344 446 0.78 0.82 0.01 0.02 0.10 0.04 0.22
Python 415 775 1.00 0.67 0.00 0.05 0.28 0.00 0.00
Ruby 391 581 1.00 0.80 0.00 0.03 0.17 0.00 0.00
TOTAL 442 3950 0.91 0.70 0.03 0.03 0.22 0.01 0.09

Table 85: Statistics about the compilation process: columns make ok and make ko report percentages relative to solutions for
each language; the columns in between report percentages relative to make ok for each language

tasks solutions run ok stderr timeout error crash

C 351 442 0.76 0.01 0.11 0.06 0.05
C# 253 307 0.73 0.01 0.20 0.06 0.00
F# 201 236 0.81 0.01 0.09 0.10 0.00
Go 349 427 0.89 0.01 0.09 0.01 0.00
Haskell 298 426 0.82 0.00 0.12 0.06 0.00
Java 275 339 0.75 0.02 0.11 0.13 0.00
Python 399 751 0.76 0.01 0.10 0.13 0.01
Ruby 378 563 0.79 0.01 0.08 0.12 0.00
TOTAL 422 3491 0.79 0.01 0.11 0.09 0.01

Table 86: Statistics about the running process: all columns other than tasks and solutions report percentages relative to solutions
for each language

tasks solutions error run ok

C 313 391 0.13 0.87
C# 203 246 0.07 0.93
F# 182 215 0.11 0.89
Go 317 389 0.02 0.98
Haskell 264 376 0.07 0.93
Java 247 297 0.15 0.85
Python 354 675 0.15 0.85
Ruby 349 516 0.14 0.86
TOTAL 422 3105 0.11 0.89

Table 87: Statistics about fault proneness: columns error and run ok report percentages relative to solutions for each language

C: incomplete (27 solutions), snippet (21 solutions), nomain (21 solutions), nonstandard libs (17 solutions), other (12
solutions), unsupported dialect (2 solutions) C#: nomain (24 solutions), nonstandard libs (17 solutions), snippet (14 solutions),
bug (3 solutions), filename (1 solutions), other (1 solutions) F#: nonstandard libs (10 solutions), bug (2 solutions), other (1
solutions) Go: nonstandard libs (41 solutions), snippet (9 solutions), other (5 solutions), bug (2 solutions), nomain (0 solutions)
Haskell: nomain (45 solutions), nonstandard libs (30 solutions), snippet (20 solutions), bug (18 solutions), other (16 solutions)
Java: nomain (60 solutions), snippet (56 solutions), other (24 solutions), nonstandard libs (13 solutions), incomplete (6 solutions)
Python: other (2 solutions) Ruby: other (519 solutions), nonstandard libs (35 solutions), merge (9 solutions), package (7 solutions),
bug (6 solutions), snippet (2 solutions), abort (1 solutions), incomplete (1 solutions), require (1 solutions)

70
X. A PPENDIX : P LOTS

71
A. Lines of code (tasks compiling successfully)

72
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

62
C C

7
C# F#

5.3

34
19
4
lines of code (min)

lines of code (min)

10
3
2.2

5
1.5

3
1

1
0 50 100 150 200 0 50 100 150

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

5.3

34
C C
Go Haskell
4.2

20
3.3

12
lines of code (min)

lines of code (min)

2.6

7
1.9

4
1.4

2
1

0 50 100 150 200 250 300 0 50 100 150 200 250

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

7.8

55

C C
Java Python
5.9

31
4.4

17
lines of code (min)

lines of code (min)

3.2

10
2.3

5
1.6

2
1

0 50 100 150 200 250 0 50 100 150 200 250 300 350

task task

Conciseness: tasks compiling successfully

148

C
Ruby
72
34
lines of code (min)

16
7
3

73
1

0 50 100 150 200 250 300

task

Figure 88: Lines of code (min) of tasks compiling successfully (C vs. other languages)
 Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)

60
6

50
●
●
●

40
●

C#

F#
4

30
●

●
●
●

3
●

20
●
●
●
●
●
●
●
●
●
●
●
2
●

10
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●●
●●
●●
●
●●
●●
●
●●●
●●
●
●●
●●
●
●●
●
●
●●
●
●●
●●●●
●●● ●
●● ●
●●
●●●● ● ●●
●● ●
● ●● ●● ● ● ● ● ● ●
●●
●
●
●●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●●●
●●●●●●
●●● ●●● ●●●
● ●●● ● ● ● ● ● ●● ● ●
1

0
1 2 3 4 5 6 7 0 10 20 30 40 50 60

C C

Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)

●
5

30
25
4

20
Haskell
Go

●
3

●
●

15
●
●
●
●
●
●

10
●
●
●
●
●
●
2

●
●
●
●
●
●
●
●
●
●
●
●
5
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
● ●
●
●
●●
●
●
●●
●●●
●
●●
●●●
●
●●
●
●●
●●
●●
●●
●●
●●
●
●
●●●
●●●
●
●●●
●
●●●
●●
●●
●●
●
●●
●
●●
●
●●●
●●●●●
●●
●●●
●● ● ●●● ●● ● ●● ● ●
● ●●
●●● ●
● ● ● ●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●
●
●
●●
●●
●●
●●
●
●●●
●●●
●
●●
●●●
●
●●●●● ● ● ● ● ● ● ●
1

1 2 3 4 5 0 5 10 15 20 25 30

C C

Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)
8

50
7

40
6
5

30
Python
Java

●
●
20

●
3

●
●
●
●
●
10

●
●
2

●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●●
●●●
●
●●●
●● ●●●●
●●●● ●
●● ● ● ●● ● ● ● ●
●●
●●
●
●
●●
●
●●●
●
●
●●
●
●●
●
●●
●●
●
●
●●●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●
●●
●●
●●
●●
●●●●
●●
●●●●
●●
●● ● ●
●●●
●●●
●● ●●●● ●● ● ● ●
1

1 2 3 4 5 6 7 8 0 10 20 30 40 50

C C

Conciseness: tasks compiling successfully (min)

150
100
Ruby

50

●
●
●●●
●●
●●●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●●●
●
●●●
●●●●●
●● ● ●● ●●
● ● ●
74
0

0 50 100 150

Figure 89: Lines of code (min) of tasks compiling successfully (C vs. other languages)
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

33
C# C#

8
F# Go

20

6
4.5
12
lines of code (min)

lines of code (min)

3.2
7

2.3
4

1.6
2
1

1
0 50 100 150 0 50 100 150 200

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

5.8
21

C# C#
Haskell Java

4.5
14

3.5
9
lines of code (min)

lines of code (min)

2.7
6
3

2
1.5
2
1

0 50 100 150 200 0 50 100 150 200

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

41

46

C# C#
Python Ruby
24

27
14

15
lines of code (min)

lines of code (min)

8

9
5

5
2

2
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 90: Lines of code (min) of tasks compiling successfully (C# vs. other languages)

75
 Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)

8
30

7
●

25
●

6
●

●
20

5
●

Go
F#

15

4
●
●

●
10

3
●
●
●
●
●
●
●
●
●

2
●
5

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●●
●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●
●●●
●●●
●
●●●
●●●
●●
●
●●●●●
●
●●●●
●●●
●●●●
●●
●●● ●●●●●● ● ●● ● ● ●
●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
●●
●●●
●
●
●●●●
●●●
●
●●●
●●●
●●
●●●
●●
●●●●
●● ●● ●
●● ● ●●
●● ●
● ● ●

1
0

0 5 10 15 20 25 30 1 2 3 4 5 6 7 8

C# C#

Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)
20

●
●

5
●
15

4
Haskell

Java
10

●
●
●
●
●
●
●
2
5

●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●●
●●●
●●
●●
●●
●●
●
●●●
●●
●●
●
●●
●●
●●
●
●●
●●●
●●●●
●●●●●
●
●●● ●●●●● ●
● ●● ● ● ● ● ● ●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●●●
●●
●●
●●●
●
●
●●●●
●●●●● ●
● ●●●● ● ● ● ●● ●● ● ●● ●
1

5 10 15 20 1 2 3 4 5

C# C#

Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)
40

40
30

30
Python

Ruby
20

20
10

10

●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●●
●
●●
●●●
●
●
●●
●●
●
●
●●
●●●
●●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●●
●
●●
●
●●●
●●●
●●●
●●
●●
●●●● ●●●●● ●● ● ● ● ● ● ● ● ●
●
●●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●●
●
●●
●●
●
●
●●
●
●
●●
●
●●●
●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●●● ●●
●●● ●●●●●●●●● ●●●●● ● ● ● ● ● ● ●● ●
0
0

0 10 20 30 40 0 10 20 30 40

C# C#

Figure 91: Lines of code (min) of tasks compiling successfully (C# vs. other languages)

76
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

65

10
F# F#
Go Haskell

36

7
20

5
lines of code (min)

lines of code (min)

10

4
5

3
3

2
1

1
0 50 100 150 0 50 100 150

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

48

F# F#

9
Java Python

6.6
28

4.8
16
lines of code (min)

lines of code (min)

3.5
9

2.4
5

1.6
2
1

0 50 100 150 0 50 100 150 200

task task

Conciseness: tasks compiling successfully

30

F#
Ruby
19
11
lines of code (min)

7
4
2
1

0 50 100 150 200

task

Figure 92: Lines of code (min) of tasks compiling successfully (F# vs. other languages)

77
 Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)

10
● ●

60
● ●

8
50
●

40 ●

6
Haskell
Go

●
30

4
●
20

● ●
●
●
● ●
● ●
●
● ●
●
●
10

●
●
● ●
● ●

2
●
●
●
●
● ●
●
●
●
● ●
●
●
● ●
●
● ●
● ●
●
●
● ●
●
●
●
●
●
●●● ●
●
●
●●●
●
●●
●
●●
●
●●
●●●●●
●●
●●
●●●●●●
●●●●●●● ● ● ● ●●
● ● ● ● ● ●●
0

0 10 20 30 40 50 60 2 4 6 8 10

F# F#

Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)

8
40

●
●

●
●
●
30

● 6
Python
Java

●
20

●
4

●
●
●

● ●
●
● ●
10

●
● ●
● ●
●
●
● ●
2

● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
● ●
●
●
● ●
●
●
● ●
●
●
●
●
●● ●
●
●●
●
●●
●●●
●
●●
●●
●●
●●
●●
●
●●
●●
●
●● ●●●●
●●●●●●●
● ● ●●
●● ● ● ● ● ●● ●● ● ● ● ●● ●
0

0 10 20 30 40 2 4 6 8

F# F#

Conciseness: tasks compiling successfully (min)

30

●
25
20
Ruby

15
10

●
●
5

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●●● ●● ●●● ●● ●● ●●
0

0 5 10 15 20 25 30

F#

Figure 93: Lines of code (min) of tasks compiling successfully (F# vs. other languages)

78
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

7.2
26

Go Go
Haskell Java

5.5
16

4.1
10
lines of code (min)

lines of code (min)

3.1
6

2.2
4

1.5
2
1

1
0 50 100 150 200 250 0 50 100 150 200 250

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

68

68

Go Go
Python Ruby
37

37
20

20
lines of code (min)

lines of code (min)

11

11
6

6
3

3
1

0 50 100 150 200 250 300 350 0 50 100 150 200 250 300 350

task task

Figure 94: Lines of code (min) of tasks compiling successfully (Go vs. other languages)

79
 Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)
25

7
6
20

●
●

5
15
Haskell

Java
●

4
10

3
●
●
●
●
●
●
●
●
●
●
5

2
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●
●●●●
●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●
●●●
●
●●
●●●
●
●●
●●●
●
●●
●
●●
●●●
●
●●●●●
●● ●●●●
●●● ●●● ● ● ● ● ● ● ● ●
●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●●
●
●●
●●●
●●●
●
●●●
●●●
●
●●●
●
●●●
●●●●●● ● ● ●● ● ●

1
5 10 15 20 25 1 2 3 4 5 6 7

Go Go

Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)
70

70
60

60
50

50
40

40
Python

Ruby
30

30
20

20
10

10

●
●
● ●
●
●●
●●
●
●
●●
●●
●
●●
●●
●●
●●
●
●
●●
●●
●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●
●●●
●
●
●●
●●
●●●●●●
●
●●●
●●
●● ● ● ● ● ● ● ●●
●
●
●●
●●
●●
●
●
●●●
●
●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●●
●
●●
●
●●
●●●●
●●●
●●●●●
●
●●●●● ●● ●
● ● ● ●● ● ● ● ●
0

0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70

Go Go

Figure 95: Lines of code (min) of tasks compiling successfully (Go vs. other languages)

80
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully
29

10
Haskell Haskell
Java Python
18

7
11

5
lines of code (min)

lines of code (min)

7

4
4

3
2

2
1

1
0 50 100 150 200 0 50 100 150 200 250 300

task task

Conciseness: tasks compiling successfully

27

Haskell
Ruby
17
11
lines of code (min)

6
4
2
1

0 50 100 150 200 250 300

task

Figure 96: Lines of code (min) of tasks compiling successfully (Haskell vs. other languages)

81
 Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)

30

10
●
●

25
●

8
●

20
●

6
Python
Java

15
● ●
●
●
●
●
● ●
●
●
● ●

4
10
●
●
●
●
● ●
●
●
● ●
●
● ●
●
● ●
●
● ●
● ●
●
●
● ●
●
●
● ●
● ●
●
5

●
● ●
●
●
● ●
●
● ●

2
●
● ●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●●
●●● ●● ● ●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●
●●●
●●
●●
●●
●
●●
●●●●
●
●●
●●
● ●●
●●●
●●●●●●
●●●●●●
●● ● ● ● ● ● ● ● ●
0

0 5 10 15 20 25 30 2 4 6 8 10

Haskell Haskell

Conciseness: tasks compiling successfully (min)

25
20
15
Ruby

10

●
●

●
●
●
●
5

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●●
●●●● ●●
●
●●●
●● ● ● ● ● ●
0

0 5 10 15 20 25

Haskell

Figure 97: Lines of code (min) of tasks compiling successfully (Haskell vs. other languages)

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

48

58

Java Java
Python Ruby
28

33
16

18
lines of code (min)

lines of code (min)

10
9
5

5
2

3
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 98: Lines of code (min) of tasks compiling successfully (Java vs. other languages)

82
 Conciseness: tasks compiling successfully (min) Conciseness: tasks compiling successfully (min)

60
50
40

40
30
Python

Ruby

30
20

20
10

10
●
●
●
●
●
●
● ●
●
●
●
●●
●
●●
●
●●
●●●
●
●
●●
●
●●
●
●●
●●
●
●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●●●
●●
●●●
●● ●
●
●●●● ●●● ● ● ●● ● ● ● ●
●●
●
●●
●●●
●
●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●
●
●
●●
●●
●
●
●●
●●
●
●●●
●
●●
●●●●●●
●●
●● ● ●
●●● ●● ● ● ● ● ● ●

0
0

0 10 20 30 40 0 10 20 30 40 50 60

Java Java

Figure 99: Lines of code (min) of tasks compiling successfully (Java vs. other languages)

Conciseness: tasks compiling successfully

92

Python
Ruby
48
25
lines of code (min)

13
6
3
1

0 100 200 300

task

Figure 100: Lines of code (min) of tasks compiling successfully (Python vs. other languages)

Conciseness: tasks compiling successfully (min)

80
60
Ruby

40

●
20

●
●

●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●●
●●●
●●●
●● ●
● ●● ●
0

0 20 40 60 80

Python

Figure 101: Lines of code (min) of tasks compiling successfully (Python vs. other languages)

83
 Conciseness: tasks compiling successfully
67

C
C#
F#
Go
37

Haskell
Java
Python
Ruby
20
lines of code (min)

11
5
3
1

0 20 40 60 80 100

task

Figure 102: Lines of code (min) of tasks compiling successfully (all languages)

84
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

64
C C

7
C# F#

5.3

36
19
4
lines of code (mean)

lines of code (mean)

10
3
2.2

5
1.5

3
1

1
0 50 100 150 200 0 50 100 150

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

17

39
C C
Go Haskell
11

23
14
8
lines of code (mean)

lines of code (mean)

5

8
3

4
2

2
1

0 50 100 150 200 250 300 0 50 100 150 200 250

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

7.8

55

C C
Java Python
5.9

31
4.4

17
lines of code (mean)

lines of code (mean)

3.2

10
2.3

5
1.6

2
1

0 50 100 150 200 250 0 50 100 150 200 250 300 350

task task

Conciseness: tasks compiling successfully

148

C
Ruby
72
34
lines of code (mean)

16
7
3

85
1

0 50 100 150 200 250 300

task

Figure 103: Lines of code (mean) of tasks compiling successfully (C vs. other languages)
 Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)

60
6
●

50
●

40
●

C#

F#
4

30
●

●
●
●

3
●●

20
● ●
●
●
● ●
●
●
● ● ●
●●
●
● ● ● ● ●
2
●
● ●

10
●
● ●● ●
● ● ●
●
●
●● ●
● ● ●
●●
● ●
●
●● ●● ● ●
●●
● ● ●
● ●●● ●
●
●
●
● ● ● ● ●●● ●
●●●●●
●
● ● ● ●
●
●● ●● ● ●
●●
●
●●
●●
●●
●●●
●●
●●●
●
●●
●●
●
●●
●
●●
●
●●
●●●●
●●●● ●●●●
●●
● ●●●● ●●
●● ● ● ● ● ● ● ● ●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
● ●
●
●
●●
●●
●●●
●●
●
●●●●●● ●
●●● ●●●
●●●● ●● ● ● ● ● ● ●
1

0
1 2 3 4 5 6 7 0 10 20 30 40 50 60

C C

Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)

40
●
15

30
10

Haskell
Go

20
●
5

10
●
●
●● ●
●●
● ●●●● ● ●
●● ●
●●
●
●
● ●● ● ●●
● ●●● ●
●
●
●●● ●●●● ● ●●● ● ●
● ● ●● ● ●
●
●●●
●
● ●●●●●●● ● ●●
●● ● ●● ● ●
● ●●● ●●● ●●
● ● ●
●
●●●● ● ● ●
●●
●●
●● ●
●● ●
●●
●●
● ●● ●
● ●
●● ●●● ● ● ● ●
●
●●
●
●●
●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●●
●
●●
●●
●●●
●
●●●●●
●●●
●●● ●
●
●
●●
●●●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
● ●
●
●●●●●●●● ● ● ●
0

5 10 15 0 10 20 30 40

C C

Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)
8

50
7

40
6
5

30
Python
Java

●
4

●
20

●
● ●
●
●
3

● ●
● ●
● ●
●
● ● ●
● ●●
●
10

●
●
● ●
2

● ●
●
● ●
● ●
● ● ●
●
●
● ● ● ●
● ● ● ● ●
●●●●● ●●●● ● ●
●
●
● ● ● ● ● ● ●
●●
● ●● ● ●● ●
●●
●●
●● ●●●
●● ●● ● ● ● ● ●●● ● ●
●
●
●●
●●● ●
● ●●●● ● ●
●
● ●●
●●
●
●
● ●●
●●●
●●● ● ●●●
● ● ● ●● ●● ●
● ●
●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●●
●
●●
●●
●●
●●
●●●
●
●
●●●
●●
●●● ●
●●●●●●●●
● ● ●●
●● ● ●● ● ● ● ●
●
●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●●
●●
●●
●●
●
● ●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●
●●
●●
●
●●
●●
●●● ●●●
●●●● ●●●
●●● ●● ●
● ● ● ● ●
1

1 2 3 4 5 6 7 8 0 10 20 30 40 50

C C

Conciseness: tasks compiling successfully (mean)

150
100
Ruby

50

● ●

●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●●
●
●● ●
●
●
●●●
●●
●●
●
●
●●
●●
●●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●●
●
● ●
●
●●● ●●
●●● ●●
●●●● ●
●
●● ●
●
● ● ●
86
0

0 50 100 150

Figure 104: Lines of code (mean) of tasks compiling successfully (C vs. other languages)
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

33
C# C#

8
F# Go

20

6
4.5
12
lines of code (mean)

lines of code (mean)

3.2
7

2.3
4

1.6
2
1

1
0 50 100 150 0 50 100 150 200

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

21

11
C# C#
Haskell Java
14

8
9

6
lines of code (mean)

lines of code (mean)

6

4
3

3
2

2
1

0 50 100 150 200 0 50 100 150 200

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

41

46

C# C#
Python Ruby
24

27
14

15
lines of code (mean)

lines of code (mean)

8

9
5

5
2

2
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 105: Lines of code (mean) of tasks compiling successfully (C# vs. other languages)

87
 Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)

8
30

7
●

25
●

6
●

●
20

5
●

Go
F#

15

4
● ●

● ●
10

3
● ●
●
●
● ● ●
● ●
●
●
●
● ●
●

2
● ●●
● ●
5

●
● ● ●
●
● ●
●
● ●
● ● ● ●
●●
●
● ●● ● ● ● ●
● ●
●● ●●● ● ● ●
●● ● ● ● ●
● ●
●
●
● ●● ●
●● ● ● ●
●
●
●
●●
●
●
●●
●
●●
●●
●●
●
●
●●
●●
●
●●●
●
●●
●
●
●●●
●
●
●●
●●
●●●
●●
●●
●●●
●●
●●●
●
●●●●●●
● ●●●
●●● ●● ● ●●●● ● ●● ● ● ● ● ● ●
●●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●
●
●
●●
●●●
●●
●
●●
●
●
●● ●
●●●
●●●
●●●
●●
●●●
●● ●● ●● ●
● ●● ●● ●

1
0

0 5 10 15 20 25 30 1 2 3 4 5 6 7 8

C# C#

Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)

●
20

10
15

8
Haskell

Java

6
10

●
●
4
5

●
●
●●
● ● ● ●
●
● ●
2

● ● ●
● ● ● ●●
● ● ● ● ● ● ●
● ● ●● ● ● ● ●●
● ● ● ●
● ● ● ● ●● ● ● ● ● ● ●
●
●
● ●● ●● ● ● ●
● ● ● ● ●●●●● ● ● ●● ● ●● ●
●●●●●●●
● ● ● ● ●●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●●
●
●
●●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●●●●
●●●
●
●●●
●●
●●
●●
●●
●●
●●
●●● ●●●
●
●●● ●
●●● ●●● ● ● ● ●●● ●
●
●●
●
●●
●
●●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●● ●●●●
●●●● ●● ● ●● ● ● ●

5 10 15 20 2 4 6 8 10

C# C#

Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)
40

40
30

30
Python

Ruby
20

20

● ●
●
10

10

● ●
●
●
● ●● ● ●
● ●
● ● ● ● ●●
●●● ● ● ● ●
●● ●
● ●
● ●● ●● ●● ● ● ●● ● ●
● ● ● ●● ●● ●● ●●● ● ●●●●● ●● ●● ●● ●●● ● ●● ●
●
●
● ●
●●●
●
●●●
● ●
●●●●
●●
●● ●●●●
● ● ● ● ●
●● ●●
●
●●●●
●● ●
●● ● ●
●
●
●
●
●●
●
●●
●●
●●
●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●
●●
●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●●●●●● ●
●
●
● ●●●●
●● ● ● ● ● ● ● ●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
●●
●●
●●●
●●
●●●
●
●●
●
●●
●●●●
●● ● ●●●●● ●● ● ● ● ● ● ●
0
0

0 10 20 30 40 0 10 20 30 40

C# C#

Figure 106: Lines of code (mean) of tasks compiling successfully (C# vs. other languages)

88
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

65

11
F# F#
Go Haskell

36

8
20

5
lines of code (mean)

lines of code (mean)

10

4
5

3
3

2
1

1
0 50 100 150 0 50 100 150

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

100

16
F# F#
Java Python
51

11
26

7
lines of code (mean)

lines of code (mean)

13

5
6

3
3

2
1

0 50 100 150 0 50 100 150 200

task task

Conciseness: tasks compiling successfully

30

F#
Ruby
19
11
lines of code (mean)

7
4
2
1

0 50 100 150 200

task

Figure 107: Lines of code (mean) of tasks compiling successfully (F# vs. other languages)

89
 Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)

10
●

60
●

●
●

50
●

8
●
40 ●

Haskell
●

6
Go

●
30

● ●

●
●
● ●

4
20

●
● ● ●
●
●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
10

●
● ●●
●● ● ● ●
● ●
●● ● ● ●

2
●
●
● ●
●●
●
● ● ● ●
●
● ●
●
●
●
● ● ●●● ●● ●
●
●● ●
●
●
●
●● ● ●
● ● ●
●
●●
● ● ●
● ●● ● ●
●
●
●
●●
●
●
●● ●
●
●
●
●●●
●
●●
●●
●● ●
●●●●●
●●
●●
●●●●●● ●●●●● ● ● ●
● ● ● ● ●
0

0 10 20 30 40 50 60 2 4 6 8 10

F# F#

Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)
100

● ●

15
●
80

10
60

●
Python
Java

●
●
●
40

●
●
●
●
●

●
● ● ●
5

●
20

●
● ●
●
● ● ●
● ●
● ●● ● ●
● ●● ●
● ●
● ●●
●
● ● ● ●●
●
● ● ●
●
●
● ●
● ●●●●
●● ● ● ●
●
●
●● ●
●● ●●● ●● ●
●
●● ● ●● ● ● ●
●●
●
●● ●●●
●
●
● ● ●● ●●●●
●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●
●●
●
●●
●
●
●●● ●●
●
●●●●
●●●●● ● ● ● ●● ● ● ●
0

0 20 40 60 80 100 5 10 15

F# F#

Conciseness: tasks compiling successfully (mean)

30

●
25
20
Ruby

15

●●
10

●
● ●
●
●
● ●
●●
● ●
5

● ● ●
● ●●
●●
●
●
●
●
●● ●
● ●
●
● ●● ● ●
● ●
● ● ●● ●
●
●●
●●●
● ●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●●●
● ● ●● ●● ● ●● ●● ●
0

0 5 10 15 20 25 30

F#

Figure 108: Lines of code (mean) of tasks compiling successfully (F# vs. other languages)

90
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

7.2
26

Go Go
Haskell Java

5.5
16

4.1
10
lines of code (mean)

lines of code (mean)

3.1
6

2.2
4

1.5
2
1

1
0 50 100 150 200 250 0 50 100 150 200 250

task task

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

68

68

Go Go
Python Ruby
37

37
20

20
lines of code (mean)

lines of code (mean)

11

11
6

6
3

3
1

0 50 100 150 200 250 300 350 0 50 100 150 200 250 300 350

task task

Figure 109: Lines of code (mean) of tasks compiling successfully (Go vs. other languages)

91
 Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)
25

7
6
20

● ●

5
15
Haskell

Java
●

4
10

3
●
● ●
● ●
●
●● ●●
●
● ●
●
● ●●
5

2
●
●● ●
● ●
●
●● ●
●
● ●
●
●●● ● ● ● ● ●●● ● ●●●
●
●
● ● ● ●● ●
●● ● ● ●
● ● ● ● ● ●
●
●
●●●●●●● ●● ●
●●●● ● ● ● ●
●
●●● ●● ●
●●●●● ●
● ●● ●● ●● ● ●
●● ● ●●● ● ● ●●● ● ●
●●●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●●●
●●●
●●●
●●
●●
●
●
●●●●●●●●
●● ●●●●●
●●●●●●● ●● ●●
● ● ● ●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●●
●●
●●
●
● ●
●●●
●●●
●●
●●
●●●●
●●●
●●
●
●●
●●
●
● ● ●●● ● ●● ● ● ●

1
5 10 15 20 25 1 2 3 4 5 6 7

Go Go

Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)
70

70
60

60
50

50
40

40
Python

Ruby
30

30
20

20

●
● ●
●
10

10

● ●
● ●
●● ●
● ● ● ●● ●
● ● ● ● ●
● ●●● ●● ●●● ● ●
● ●●●●● ●
●
●
●
●●
●●●●
●●●●●
● ●●
●● ●●
●
● ●
●●
● ●● ● ● ● ● ●
● ●●
●●
●●
●
●● ●●●● ● ● ● ●● ● ● ●
●
●
●
●●●
●
●●
●●
●
●●●
●●
●●
●●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●●●
●
●●
●●●●●● ●● ● ● ● ● ●
●
●●
●●
●
●
●●
●
●●
●●
●●
●
●●
●
●●
● ●
●
●
●●
●
●
●
●●
●●
●
●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●●●●● ●●●
● ●● ●● ● ● ● ● ●
0

0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70

Go Go

Figure 110: Lines of code (mean) of tasks compiling successfully (Go vs. other languages)

92
 Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully
29

26
Haskell Haskell
Java Python
18

16
11

10
lines of code (mean)

lines of code (mean)

7

6
4

4
2

2
1

1
0 50 100 150 200 0 50 100 150 200 250 300

task task

Conciseness: tasks compiling successfully

27

Haskell
Ruby
17
11
lines of code (mean)

6
4
2
1

0 50 100 150 200 250 300

task

Figure 111: Lines of code (mean) of tasks compiling successfully (Haskell vs. other languages)

93
 Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)

30
● ●

25
●

25

20
20
●

15
Python
Java
●

15
●
●

●
●

10
●
● ●
●
10
●●
● ●
●● ● ●
●●
● ●
●● ●
●
●
●● ●
● ●
●
●●● ● ●●
● ●●

5
●●● ● ● ●
● ● ●
●
5

●
●●
●● ●●
●
●
●● ● ● ●
●
●
● ●
●●●● ●
●
●
●● ●●●
● ●
●● ● ● ●●
●
●
● ●
●
●
● ● ● ●
●●● ● ●●● ●
●
●
●●● ●
●
●●
●● ●●●●●●
● ●● ● ● ●
●
●● ● ●● ●●
●
●●
● ●●
●
●
●●●
● ●●●
●●●●
●● ● ● ●●
● ●● ● ●
●
●● ● ● ●● ●
● ●
●
●
●
●
●●
●
●
●
●●
●● ● ● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●
●●
●●
●●
●
●
●●
●
●●
●●
●●●
●●
●
●●
●
●●●●
● ●
● ● ● ● ●● ●
0

0 5 10 15 20 25 30 5 10 15 20 25

Haskell Haskell

Conciseness: tasks compiling successfully (mean)

25
20
15
Ruby

● ●

●
10

●●
●
●
● ● ●
●
●●
●
● ● ●
5

●●● ● ●
●
● ●●● ●
●
●●●
●●
●●
● ●
●● ●
●
●
● ● ●●
● ● ●● ●●● ●
●
●●● ● ●
●
●
●● ●●●●●● ● ● ●
●
●
●
●
●
●●●●●
●
● ●
●●●
●●●
●
●●
●●
●
● ●● ● ● ●
●
●●
●
●●
●
●●
●
●●●
●
●●●● ●●
●●●● ●
●●●
● ● ● ● ●
0

0 5 10 15 20 25

Haskell

Figure 112: Lines of code (mean) of tasks compiling successfully (Haskell vs. other languages)

Conciseness: tasks compiling successfully Conciseness: tasks compiling successfully

100

58

Java Java
Python Ruby
51

33
26

18
lines of code (mean)

lines of code (mean)

13

10
6

5
3

3
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 113: Lines of code (mean) of tasks compiling successfully (Java vs. other languages)

94
 Conciseness: tasks compiling successfully (mean) Conciseness: tasks compiling successfully (mean)

60
100

50
80

40
60
Python

Ruby

30
40

20
● ●
20

10
●
● ●
●
●
●
● ● ● ●
● ● ●●
● ● ● ●● ● ●
● ● ● ●●
●
● ●
●●
●●●●
●●
●●● ●● ●● ●
●●●●●
●●
●
● ●
●● ●●●●● ● ● ●
●
●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●
●●
●
●●
●
●●
●
●●
● ●
●●
●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
● ●●
●
●● ●
●● ● ●● ● ● ●●
●●
●
●●
● ●
●
●●
●
●
●●
●
●
●●●
●●
●●
●●
●
●●
●●
●
●●
●
●
●
●●
●
●●
●
●●
●
●●●
●
●● ●
●●●●
● ●● ● ●
●● ●● ● ● ●
0

0
0 20 40 60 80 100 0 10 20 30 40 50 60

Java Java

Figure 114: Lines of code (mean) of tasks compiling successfully (Java vs. other languages)

Conciseness: tasks compiling successfully

92

Python
Ruby
48
25
lines of code (mean)

13
6
3
1

0 100 200 300

task

Figure 115: Lines of code (mean) of tasks compiling successfully (Python vs. other languages)

Conciseness: tasks compiling successfully (mean)

80
60
Ruby

40

●
20

● ●
●
● ●
●
●
● ●
●
●●●● ● ● ●
●
●● ● ●●
●●●
●● ●
● ●
●
●●
●●
●
●●
●
●● ●●
●●
●
●●
●
●●
●
●●
●
●
●
●●
●
●
●●●●
● ●
●●
● ●
●
●
●●
●
●●
●
●●
●
●●●
●
●●
● ●
● ● ●● ●
0

0 20 40 60 80

Python

Figure 116: Lines of code (mean) of tasks compiling successfully (Python vs. other languages)

95
 Conciseness: tasks compiling successfully
100

C
C#
F#
Go
51

Haskell
Java
Python
Ruby
26
lines of code (mean)

13
6
3
1

0 20 40 60 80 100

task

Figure 117: Lines of code (mean) of tasks compiling successfully (all languages)

96
B. Lines of code (all tasks)

97
 Conciseness: all tasks Conciseness: all tasks

30

62
C C
C# F#

19

34
11

19
lines of code (min)

lines of code (min)

10
7
4

5
2

3
1

1
0 50 100 150 200 250 0 50 100 150 200

task task

Conciseness: all tasks Conciseness: all tasks

68

56
C C
Go Haskell
37

32
20

18
lines of code (min)

lines of code (min)

11

10
6

5
3

2
1

0 100 200 300 0 50 100 150 200 250 300 350

task task

Conciseness: all tasks Conciseness: all tasks

31

55

C C
Java Python
19

31
12

17
lines of code (min)

lines of code (min)

10
7
4

5
2

2
1

0 50 100 150 200 250 300 0 100 200 300

task task

Conciseness: all tasks

148

C
Ruby
72
34
lines of code (min)

16
7
3

98
1

0 50 100 150 200 250 300 350

task

Figure 118: Lines of code (min) of all tasks (C vs. other languages)
 Conciseness: all tasks (min) Conciseness: all tasks (min)

30

60
25

50
20

40
C#

F#
15

30
●

20
10
●
●

10
●
●
5

●
●
●
●
●
●
● ●
●
●
●
● ●
●
● ●
●
●
●
● ●
●●
●
●
●●
●●
●●
●
●
●●
●
●●
●
●
●●●
●
●●
●●
●●●
●
●●● ●●●●●● ● ● ● ● ● ●
●●
●
●
●●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●●●●●●
●●● ● ●●●●●
● ●●● ● ● ● ●● ● ●

0
0

0 5 10 15 20 25 30 0 10 20 30 40 50 60

C C

Conciseness: all tasks (min) Conciseness: all tasks (min)

70

50
60
50

40
40

Haskell

30
Go

30

20
20

10
10

●
●
● ●
●
● ●
●
● ●
●
● ●
●
●
●●
● ●
●
●●
●
●●
●
●●
●
●●●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●●
●●
●●● ● ● ●
●●
●
●●●
●
●
●
●●
●
●
●●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●●●
●
●●
●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●
●●●
●● ●●● ●●● ● ● ● ● ●● ●● ●
0

0 10 20 30 40 50 60 70 0 10 20 30 40 50

C C

Conciseness: all tasks (min) Conciseness: all tasks (min)

●
30

50
25

40
20

30
Python
Java

15

20
10

●
●
●

●
●
10
5

●
●
●
●
●
● ●
●
● ●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●●
●●
●
●●
●●
●
●
●●
●●
●
●●
●●
●●
●
●
●●●
●
●●
●
●●●
●
●
●●● ●●●● ●● ● ● ● ●
●●
●●
●
●
●●
●
●●●
●
●
●●
●
●●
●
●●
●
●
●●
●●●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●●
●●
●●
●●
●●
●
●●●●
●●●
●●●
●● ● ●
●●●
●●●
●● ●●●● ●● ● ● ●
0
0

0 5 10 15 20 25 30 0 10 20 30 40 50

C C

Conciseness: all tasks (min)

150
100
Ruby

50

●
●
●
●●
●●
●●●
●
●
●●
●●
●
●●
●
●
●●
●
●●
●●
●
●●
●●
●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●●
●
●●●
●●●●●
●●● ● ●● ●●
● ● ●
99
0

0 50 100 150

Figure 119: Lines of code (min) of all tasks (C vs. other languages)
 Conciseness: all tasks Conciseness: all tasks

33

63
C# C#
F# Go

20

35
12

19
lines of code (min)

lines of code (min)

10
7
4

5
2

3
1

1
0 50 100 150 0 50 100 150 200 250

task task

Conciseness: all tasks Conciseness: all tasks

44

41
C# C#
Haskell Java
26

24
15

14
lines of code (min)

lines of code (min)

9

8
5

5
2

2
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Conciseness: all tasks Conciseness: all tasks

41

48

C# C#
Python Ruby
24

28
14

16
lines of code (min)

lines of code (min)

8

9
5

5
2

2
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 120: Lines of code (min) of all tasks (C# vs. other languages)

100
 Conciseness: all tasks (min) Conciseness: all tasks (min)

60
30

50
25

40
20

Go
F#

30
15

20
10

10
5

●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●● ●
●●
●
●
●
●
●●
●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●
●●●
●●●
●
●●●
●●●
●●
●
●●●●●
●
●●●●
●●●●●
●●
●●● ●●●●●● ● ●● ● ● ● ●
●●
●●
●●
●●
●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●●
●

0
0

0 5 10 15 20 25 30 0 10 20 30 40 50 60

C# C#

Conciseness: all tasks (min) Conciseness: all tasks (min)

40
40

30
30

●
Haskell

Java

20
20

10
10

●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
● ●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●
●
●●
●●
●
●●●●
●
●
●●
●
●
●●
●●●
●
●●
●●
●●
●●
●
●●●
●●●●
●●●
● ●●●● ● ● ● ●● ●●● ●● ● ● ●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●
●●
●●●●●
●● ●
●
● ●● ●●
0

0 10 20 30 40 0 10 20 30 40

C# C#

Conciseness: all tasks (min) Conciseness: all tasks (min)

40

40
30

30

●
Python

Ruby
20

20
10

10

●
●
●
● ●
●
● ●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●
●●●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●●
●
●●
●●●
●
●●
●
●●●●●
●●●
●●
●●
●●●● ●●●●● ●● ● ● ● ● ● ● ● ●
●
●
●
●●
●
●●
●
●
●●
●●
●●●
●
●
●●●
●
●●
●
●
●●
●●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●● ●●
●●● ●●●●●●●●● ●●●●● ● ● ● ● ● ● ●● ● ●
0
0

0 10 20 30 40 0 10 20 30 40

C# C#

Figure 121: Lines of code (min) of all tasks (C# vs. other languages)

101
 Conciseness: all tasks Conciseness: all tasks

65

22
F# F#
Go Haskell

36

14
20

9
lines of code (min)

lines of code (min)

10

6
5

4
3

2
1

1
0 50 100 150 200 0 50 100 150 200

task task

Conciseness: all tasks Conciseness: all tasks

82

26
F# F#
Java Python
44

17
23

10
lines of code (min)

lines of code (min)

12

6
6

4
3

2
1

0 50 100 150 200 0 50 100 150 200

task task

Conciseness: all tasks

30

F#
Ruby
19
12
lines of code (min)

7
4
2
1

0 50 100 150 200

task

Figure 122: Lines of code (min) of all tasks (F# vs. other languages)

102
 Conciseness: all tasks (min) Conciseness: all tasks (min)

●
●

60

20
●

50
●

15
40 ●

Haskell
Go

30

10
●

● ●
20

●
●
● ●
●
●
●

5
●
●
●
●
10

● ●
●
●
● ●
●
● ●
●
● ●
●
●
●
● ●
● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
● ●
●
●
●
●
●
●●●
●● ●
●●
●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●●● ●●● ●● ● ● ●
● ● ● ● ●
0

0 10 20 30 40 50 60 5 10 15 20

F# F#

Conciseness: all tasks (min) Conciseness: all tasks (min)

● ●
80

25
20
60

●
15
Python
Java

40

10

●
20

●
●
● ●
●
●
●
5

●
●
● ●
●
●
● ●
●
●
●
● ●
●
● ●
● ●
●
●
● ●
●
● ●
●
●
●
●
●●●● ● ● ●
●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●
●●●
●●
●●● ●
●●●● ● ● ●● ●
0

0 20 40 60 80 0 5 10 15 20 25

F# F#

Conciseness: all tasks (min)

●
30

●
25
20
Ruby

15
10

●
●
5

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●●● ●●
● ●● ●● ●●
0

0 5 10 15 20 25 30

F#

Figure 123: Lines of code (min) of all tasks (F# vs. other languages)

103
 Conciseness: all tasks Conciseness: all tasks
63

21
Go Go
Haskell Java
35

14
19

9
lines of code (min)

lines of code (min)

10

6
5

3
3

2
1

1
0 50 100 150 200 250 300 350 0 50 100 150 200 250 300

task task

Conciseness: all tasks Conciseness: all tasks

68

68

Go Go
Python Ruby
37

37
20

20
lines of code (min)

lines of code (min)

11

11
6

6
3

3
1

0 100 200 300 0 50 100 150 200 250 300 350

task task

Figure 124: Lines of code (min) of all tasks (Go vs. other languages)

104
 Conciseness: all tasks (min) Conciseness: all tasks (min)
60

20
50

15
40
Haskell

Java
30

10
●
●
20

●
●
●
●

5
●
10

●
●
●
●
● ●
●
● ●
●
●
●
●
● ●
●
●
●
● ●
●
● ●
●
●●
●
●
●
●●
●●
●
●●
●
●●
●●
●●
●
●
●●
●●
●●
●
●
●●
●●●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●●
●●
●●●
● ●● ●
● ● ● ● ●● ● ● ● ● ●●
●
●
●●
●
●●●
●
●
●
●●●
●
●
●
●●
●●
●●
●
●●
●
●●●
●
●
●●
●●
●
●●
●
●●●
●●
●●●●
●●
● ●
●●
●●
● ● ● ●
●●● ●
0

0 10 20 30 40 50 60 5 10 15 20

Go Go

Conciseness: all tasks (min) Conciseness: all tasks (min)

70

70
60

60
50

50
40

40
Python

Ruby
30

30
20

20

●
10

10

●
●
● ●
●
●
● ●
●
●
● ●
●
●
●●
●
●●
●
●●
●●
●●
●●
●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●
●●
●●
●●●●●●
●
●●●
●●
●● ● ● ● ● ● ● ● ●●
●
●●
●●
●●
●
●
●●●
●
●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●
●●
●●
●
●
●●
●
●●
●
●●
●●●●
●●●
●●●●●
●
●●●●● ●● ●
● ● ● ●● ● ● ● ● ●
0

0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70

Go Go

Figure 125: Lines of code (min) of all tasks (Go vs. other languages)

105
 Conciseness: all tasks Conciseness: all tasks
31

17
Haskell Haskell
Java Python
19

11
12

8
lines of code (min)

lines of code (min)

7

5
4

3
2

2
1

1
0 50 100 150 200 250 300 0 50 100 150 200 250 300 350

task task

Conciseness: all tasks

27

Haskell
Ruby
17
11
lines of code (min)

6
4
2
1

0 50 100 150 200 250 300

task

Figure 126: Lines of code (min) of all tasks (Haskell vs. other languages)

106
 Conciseness: all tasks (min) Conciseness: all tasks (min)

● ●

30
● ●

15
25
●
●

20

10
Python
Java
●

15
●
●
● ●
●
● ●
● ●
●
● ●
●
● ●
10

●
● ●

5
●
● ●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
● ●
●
● ●
●
●
5

●
● ●
●
● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●●● ● ● ● ● ●
●●
●●
●
●●
●
●
●●
●●
●
●
●●
●●
●
●
●●
●●
●
●●
●●
●
●●
●
●●●
●●
●●
●●●●●
●●●
●●●
●● ● ● ● ●● ● ● ●
0

0 5 10 15 20 25 30 5 10 15

Haskell Haskell

Conciseness: all tasks (min)

25

●
20
15
Ruby

●
10

●
●

●
●
●
●
●
5

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●●●
●
●●●● ●●
●
●●●
●● ● ● ● ● ● ●
0

0 5 10 15 20 25

Haskell

Figure 127: Lines of code (min) of all tasks (Haskell vs. other languages)

Conciseness: all tasks Conciseness: all tasks

48

42

Java Java
Python Ruby
28

25
16

14
lines of code (min)

lines of code (min)

9

8
5

5
2

2
1

0 50 100 150 200 250 300 0 50 100 150 200 250 300

task task

Figure 128: Lines of code (min) of all tasks (Java vs. other languages)

107
 Conciseness: all tasks (min) Conciseness: all tasks (min)

40
40

30
30
Python

Ruby

20
20

10
10

● ●

● ●
●
●
● ●
●
●
●
● ●
●
● ●
●
●
●
●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●●
●●
●●●
●●
●●●●
●● ●●● ● ● ●● ● ● ● ●
●●
●
●
●●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●●
●●
●
●●
●
●
●●
●
●●
●
●●●●
●●
●●●
●●
●●
●●
●●
●●
●●●●
● ● ● ●●●●●●
● ●● ● ● ● ● ●
0

0
0 10 20 30 40 0 10 20 30 40

Java Java

Figure 129: Lines of code (min) of all tasks (Java vs. other languages)

Conciseness: all tasks

92

Python
Ruby
48
25
lines of code (min)

13
6
3
1

0 100 200 300

task

Figure 130: Lines of code (min) of all tasks (Python vs. other languages)

Conciseness: all tasks (min)

80
60
Ruby

40

●
20

●
●

●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●●
●●●
●●●
●● ●
● ●● ●
0

0 20 40 60 80

Python

Figure 131: Lines of code (min) of all tasks (Python vs. other languages)

108
 Conciseness: all tasks
67

C
C#
F#
Go
37

Haskell
Java
Python
Ruby
20
lines of code (min)

11
5
3
1

0 50 100 150

task

Figure 132: Lines of code (min) of all tasks (all languages)

109
 Conciseness: all tasks Conciseness: all tasks

67

64
C C
C# F#

37

36
20

19
lines of code (mean)

lines of code (mean)

11

10
5

5
3

3
1

1
0 50 100 150 200 250 0 50 100 150 200

task task

Conciseness: all tasks Conciseness: all tasks

68

58
C C
Go Haskell
37

33
20

18
lines of code (mean)

lines of code (mean)

11

10
6

5
3

3
1

0 100 200 300 0 50 100 150 200 250 300 350

task task

Conciseness: all tasks Conciseness: all tasks

31

55

C C
Java Python
19

31
12

17
lines of code (mean)

lines of code (mean)

10
7
4

5
2

2
1

0 50 100 150 200 250 300 0 100 200 300

task task

Conciseness: all tasks

148

C
Ruby
72
34
lines of code (mean)

16
7
3

110
1

0 50 100 150 200 250 300 350

task

Figure 133: Lines of code (mean) of all tasks (C vs. other languages)
 Conciseness: all tasks (mean) Conciseness: all tasks (mean)

60
60

50
50

40
40
C#

F#

30
30

20
20
● ●

10
●
10

●
● ● ●
●●
● ● ●
●
●
●
● ●● ●
●
●●
●●● ● ●● ● ●
●
●
●●
●
●
●●
●
●●
●●●●●
●
● ●
●
●●
● ●●●●● ●
●
●
●
●●
●
●●
●
●●
●●●
●
●●
●●●
●●●
●
●●● ● ●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
● ●
●
●
●●
●
●●
●
●
●●●
●●
●
●●●●●
●● ●
●●●●●●●● ●● ● ● ●● ● ● ● ●
0

0
0 10 20 30 40 50 60 0 10 20 30 40 50 60

C C

Conciseness: all tasks (mean) Conciseness: all tasks (mean)

60
70

●
60

50
50

40
40

Haskell

30
Go

30

20
20

●
● ●

10
10

● ●
● ●
●● ●● ● ● ● ● ●
●
● ●
● ● ●
●
●●
● ● ●●●●● ● ● ● ●
●
●
●
●
●●
●
●
●●
●●
●●
●●●●
●
●
●
●
●●
● ●
●
●●
●●
●
●
●
●
●
●
●●●●●
● ●
●●
● ● ●●
●
●●● ● ● ● ● ●
●
●
●
●
●
●●
●●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●●●● ●●● ● ●
●
●
●●
●
●●
●●
●●●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●
●●
●●●●
●
●
●●
●●
●
●●
●
● ●
●● ●●● ●● ● ● ● ●
0

0 10 20 30 40 50 60 70 0 10 20 30 40 50 60

C C

Conciseness: all tasks (mean) Conciseness: all tasks (mean)

●
30

50
25

40
20

30
Python
Java

15

20
10

●
● ●
●
●
●
● ●
10

●
5

● ●
● ●
● ●
●● ● ● ● ●
●
● ● ● ●● ●
●●● ● ●● ● ● ●● ● ● ● ●
●
● ●
● ● ● ●● ● ● ● ● ●
●
●●
●
●
●
●●● ●
●
●
● ● ●
●
●
●●●●
●●
●
●
●
●
●
●
●●
●●
●
●●●● ●
● ● ●●●
● ● ● ● ● ● ● ●● ● ●
●
●●
●●●
● ● ● ●● ●
● ●
●●●
●●●
●●●
● ●●
● ●
● ● ● ● ● ●
● ●
●●
●●
●●
●●
●●●
●
●●●
●●
●●●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●●
●● ● ● ●
●● ● ● ● ● ●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●●
●●
●
●
●●
●●
●
●
●●
●
●●
●●
●
●
●●
●
●●
●
●●●
●●●
●● ●●●
●● ●
●● ●
●●●●● ●
● ●●●● ● ● ●
0
0

0 5 10 15 20 25 30 0 10 20 30 40 50

C C

Conciseness: all tasks (mean)

150
100
Ruby

50

● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●●
●
●
●
●
●●
●●●
●●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●●
● ●
●
●●●●●
●●●●
●
●●●
●
●● ●
●
● ●● ●
●
● ● ●
111
0

0 50 100 150

Figure 134: Lines of code (mean) of all tasks (C vs. other languages)
 Conciseness: all tasks Conciseness: all tasks

33

67
C# C#
F# Go

20

37
12

20
lines of code (mean)

lines of code (mean)

11
7
4

5
2

3
1

1
0 50 100 150 0 50 100 150 200 250

task task

Conciseness: all tasks Conciseness: all tasks

67

67
C# C#
Haskell Java
37

37
20

20
lines of code (mean)

lines of code (mean)

11

11
5

5
3

3
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Conciseness: all tasks Conciseness: all tasks

67

48

C# C#
Python Ruby
37

28
20

16
lines of code (mean)

lines of code (mean)

11

9
5

5
3

2
1

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 135: Lines of code (mean) of all tasks (C# vs. other languages)

112
 Conciseness: all tasks (mean) Conciseness: all tasks (mean)

30

60
25

50
20

40
Go
F#

30
15

● ●

20
●
10

10
5

●● ●
●
●●
● ●
●
● ● ●
●
●
●●●
●● ●
● ●● ● ●
●
●●●● ●●
●
●
● ●
●
●
●●●
●
●
●
●
●
●●
●
●
●●
●
●●
●●●
●●
●
●
●●
●●
●
●●●
●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●●●
●●
●
●●●
●●
●●●
●
●● ●●●
●●●●●● ●● ● ●●●● ● ●● ● ● ● ● ● ●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●●
●
●● ●
●

0
0

0 5 10 15 20 25 30 0 10 20 30 40 50 60

C# C#

Conciseness: all tasks (mean) Conciseness: all tasks (mean)

60

60
50

50
●
40

40
Haskell

Java
30

30

●
20

20

●
10

10

●
● ● ●
● ● ●
● ● ●●
●●● ●
●
● ●●● ● ●
●●
●●● ●● ● ●●●● ●
● ● ●
● ●
●
●
●●
●
●●
●●●●
●● ●●
●●
●●
●●
●
●
●●
●●
●●
●
●
● ●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●●
●
● ●
●●●●
●●
●●
● ●●●●
●
● ●●
●
● ● ● ● ●
● ●
●
●
●
●●
●
●●
●●
●●
●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●●●
●●●
●●
0

0 10 20 30 40 50 60 0 10 20 30 40 50 60

C# C#

Conciseness: all tasks (mean) Conciseness: all tasks (mean)

60

40
50

30
40
Python

Ruby
30

20

●
20

● ●

●
10

●
●
●
10

●
●
● ●
● ● ●
● ● ●● ● ● ● ● ●●●
● ●●
● ● ●●● ● ● ● ● ● ●● ● ● ●
●●
●●
●
●●●●●● ●●● ●●●●
●●
● ● ● ●● ● ●● ●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
● ●
● ●●
●●●
● ●●●●● ●●● ● ● ●
●
●
●
●●
●
●
●
●
●
●●
● ●
●
●●
●●
●
●
●●
●●
●●
●
●●
●
●
●●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●●●
●
●●
●
●●
●●●
●
●●
●●●●●
●
● ●
● ● ●●●●●●●● ● ● ● ● ● ●
0

0 10 20 30 40 50 60 0 10 20 30 40

C# C#

Figure 136: Lines of code (mean) of all tasks (C# vs. other languages)

113
 Conciseness: all tasks Conciseness: all tasks

65

22
F# F#
Go Haskell

36

14
20

9
lines of code (mean)

lines of code (mean)

10

6
5

4
3

2
1

1
0 50 100 150 200 0 50 100 150 200

task task

Conciseness: all tasks Conciseness: all tasks

100

26
F# F#
Java Python
51

17
26

10
lines of code (mean)

lines of code (mean)

13

6
6

4
3

2
1

0 50 100 150 200 0 50 100 150 200

task task

Conciseness: all tasks

30

F#
Ruby
19
12
lines of code (mean)

7
4
2
1

0 50 100 150 200

task

Figure 137: Lines of code (mean) of all tasks (F# vs. other languages)

114
 Conciseness: all tasks (mean) Conciseness: all tasks (mean)

●
●

60

20
●

50
●

15
●

40 ●

Haskell
Go

●
30

10
●

●
● ●
20

●
● ●
●
● ●
●
●
●
●
●

5
●
●
● ● ●
●
10

● ●
●●
●
● ●
●
●
●● ● ●
●
● ● ● ● ●
●
●
● ● ● ● ●
●
● ●
● ● ● ●
●
●● ●
●● ●●● ●
●
●
●
●● ●● ●● ● ●
●
●●
● ●● ●
● ● ●●● ●
●
●● ●
●●
●● ●
●
●●
●●● ●
●
●●
●
●●
●●
●●
●
●●●
●
●
●●
●●●
●●
●●●
●●●● ● ●● ● ● ●
0

0 10 20 30 40 50 60 5 10 15 20

F# F#

Conciseness: all tasks (mean) Conciseness: all tasks (mean)

100

● ●

25
●
80

20
60

●
15
Python
Java

●
40

●
10

●
●
●
●
● ●
●
●
●
20

●
● ●
● ● ● ●
5

●
●
● ●
●
●
● ● ● ● ●
● ● ●
●
● ● ●● ●
●
●
●
● ● ●● ●● ● ●●
●
●● ● ●
●
● ● ●● ● ●
●
●
●
●●
●●
●
●
●
●●
●●●
●●●
● ●●●
●
● ●●● ● ● ●
●
●
● ● ● ● ●
●●
● ●● ●
●
●● ●
●●●
●
●
●
●● ● ●
●
●●
●
●
●●
●●
●
●● ●
●●
●●●●
● ●●●● ●
● ● ● ●
0

0 20 40 60 80 100 0 5 10 15 20 25

F# F#

Conciseness: all tasks (mean)

●
30

●
25
20
Ruby

15

●●
10

●
● ●
●
●
● ●
●●
● ●
5

● ●
● ●● ●
●●
●
●
●
●
● ●
●
●
●●●●●
●● ●
●
●●
●
●
● ●● ● ● ●
●●●
● ●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●●●
● ●●● ●● ● ●● ●● ●
0

0 5 10 15 20 25 30

F#

Figure 138: Lines of code (mean) of all tasks (F# vs. other languages)

115
 Conciseness: all tasks Conciseness: all tasks
63

21
Go Go
Haskell Java
35

14
19

9
lines of code (mean)

lines of code (mean)

10

6
5

3
3

2
1

1
0 50 100 150 200 250 300 350 0 50 100 150 200 250 300

task task

Conciseness: all tasks Conciseness: all tasks

68

68

Go Go
Python Ruby
37

37
20

20
lines of code (mean)

lines of code (mean)

11

11
6

6
3

3
1

0 100 200 300 0 50 100 150 200 250 300 350

task task

Figure 139: Lines of code (mean) of all tasks (Go vs. other languages)

116
 Conciseness: all tasks (mean) Conciseness: all tasks (mean)
60

20
50

15
40
Haskell

Java
30

10
●
●
20

● ●
● ●
●
● ●● ●

5
●
10

●
●●
●
● ●● ●
● ●
● ●
●
●
●
●
●●●●
●● ●
● ●●●
●
● ●
●●● ●● ● ●
●● ●
●●●● ● ●
●● ●
● ●● ● ● ●
● ●
●●●●
●
●
●
●●●●
●●●●
● ●●●
● ●●● ● ● ● ● ● ● ● ●●
●
●●●
●
●●●
●
●●
●
●
● ●●●●
●
●
●●
●●
●●
●●●
●
●●●
●
●●
●●
●
●●●●
●●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●●
●●● ●● ● ● ●
●
●●
●
●●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●●
●●●●●
● ●●● ●●●
● ● ● ●● ●
0

0 10 20 30 40 50 60 5 10 15 20

Go Go

Conciseness: all tasks (mean) Conciseness: all tasks (mean)

70

70
60

60
50

50
40

40
Python

Ruby
30

30
20

20

●
●
● ●
●
●
10

10

● ●
● ● ●
●● ●
●●● ● ●
● ● ●
●
● ● ● ●● ● ● ●
● ●●●●● ●● ●●● ● ●
● ●● ●● ●
●
●
●●●
●●
● ●
●●
●● ●
●● ●
●
● ●
●●
●● ●●
●
● ●
●●
●● ●● ● ● ● ● ●
●
● ●
●●
●●
●●
●
●
●● ●
●
●●●● ● ●● ● ●● ● ● ● ●
●
●
●
●●●
●●
●
●
●●
●●●
●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●●
●●●●●● ●● ●● ● ● ●
●
●
●●
●
●
●●
●●
●●
●●
●●
●
●●
●
●●
● ●
●
●
●●
●
●
●
●●
●●
●
●●
●
●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●● ●
●●● ●●●
● ●● ●● ● ● ● ● ● ●
0

0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70

Go Go

Figure 140: Lines of code (mean) of all tasks (Go vs. other languages)

117
 Conciseness: all tasks Conciseness: all tasks
56

27
Haskell Haskell
Java Python
32

17
18

11
lines of code (mean)

lines of code (mean)

10

6
5

4
2

2
1

1
0 50 100 150 200 250 300 0 50 100 150 200 250 300 350

task task

Conciseness: all tasks

52

Haskell
Ruby
30
17
lines of code (mean)

9
5
2
1

0 50 100 150 200 250 300

task

Figure 141: Lines of code (mean) of all tasks (Haskell vs. other languages)

118
 Conciseness: all tasks (mean) Conciseness: all tasks (mean)

● ●

25
50

20
40
● ●

15
Python
30
Java
●
● ●
●

10
20 ●
●

●
● ●
●
●
● ●
● ● ●
● ●
●
●●
●
10

●●● ●● ●
●● ●

5
● ● ● ● ●
●●
● ● ● ● ●
●● ●
●
●● ● ●
●
●
●●
●
● ●●
● ● ●● ●
●
●
●
●●
●●
●●
●●●● ●●●
●
● ● ● ● ●●● ●
●●
●
● ● ●●● ●
●
●
●
●
●
●●
●
●
●
●●● ●●●
● ●● ● ●●
● ●● ●
●
●
●●
●
●
●
●
●
●
●
●●●
● ●
●
●
●●
● ●● ●●
●●● ●●●
●●
● ●● ● ● ● ●
●
●
●●
●
● ●●● ●
● ●
●
●
● ●
●
●●●●
●
●
●●●●● ●
●
●●
●
●
●
●●
●
●●
●●●●● ● ●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●●
●●
●
●●
●
●●
●●
●●
●
●
●●●●
● ●
● ● ● ● ●● ●
0

0
0 10 20 30 40 50 0 5 10 15 20 25

Haskell Haskell

Conciseness: all tasks (mean)

50
40
30
Ruby

● ●
20

● ●
●
●
10

●●●
● ●●
● ●
●
●
● ● ●
●
● ●●●●● ●●
●
●
●● ●●
● ● ● ●
●●
●
● ● ●
●●●
●● ●
●
●
●●
●●●
●●●●● ● ● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●● ●● ●
●
●
●
●●
●●
●
●
●●
●
●●
●●●
●●
●
●●● ●●
●
●● ● ● ● ● ●
0

0 10 20 30 40 50

Haskell

Figure 142: Lines of code (mean) of all tasks (Haskell vs. other languages)

Conciseness: all tasks Conciseness: all tasks

100

42

Java Java
Python Ruby
51

25
26

14
lines of code (mean)

lines of code (mean)

13

8
6

5
3

2
1

0 50 100 150 200 250 300 0 50 100 150 200 250 300

task task

Figure 143: Lines of code (mean) of all tasks (Java vs. other languages)

119
 Conciseness: all tasks (mean) Conciseness: all tasks (mean)

100

40
80

30
60
Python

Ruby

20
40
● ●

10
20

●
● ●
● ● ●
● ● ●
● ● ● ● ● ●
● ● ● ●
●● ● ● ● ●● ●●● ● ● ● ●
●
●● ●● ●● ● ● ●
●●●●
● ●●
●● ●● ●● ● ● ●● ●● ● ●
●
●
●●
●
●
●●
●
●●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●● ●●●●
●
●
●
●
●●●
●
●
●
●●●●
●
●
●
●●●
●● ●
●●
●
●
●●
●●
●
●
●●
● ●
●●
●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●
●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●●
●●
●● ● ● ●● ● ●
●
●
●
●●
●
●
●●
●
●●
● ●
●●
●●
●
●
●●
●
●●
●●
●●●
●●●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●
●●
●●
●●
●●
● ●●
●●●● ●
●●●
● ● ● ● ● ●● ●
●● ● ● ● ● ●
0

0
0 20 40 60 80 100 0 10 20 30 40

Java Java

Figure 144: Lines of code (mean) of all tasks (Java vs. other languages)

Conciseness: all tasks

92

Python
Ruby
48
25
lines of code (mean)

13
6
3
1

0 100 200 300

task

Figure 145: Lines of code (mean) of all tasks (Python vs. other languages)

Conciseness: all tasks (mean)

80
60
Ruby

40

●
20

● ●
●
● ●
●
●
● ●
●
●●●● ● ● ●
●
●●
● ● ●
●●●
●● ●
●
● ●
●
●●
●●
●
●●
●
●● ●●
●●
●
●●
●
●●
●●
●
●
●●
●
●
●●●●
● ●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●●
●
●●
● ● ●●
● ● ●● ●
0

0 20 40 60 80

Python

Figure 146: Lines of code (mean) of all tasks (Python vs. other languages)

120
 Conciseness: all tasks
100

C
C#
F#
Go
51

Haskell
Java
Python
Ruby
26
lines of code (mean)

13
6
3
1

0 50 100 150

task

Figure 147: Lines of code (mean) of all tasks (all languages)

121
C. Comments per line of code

122
 Comments per line of code: all tasks Comments per line of code: all tasks

17

90
C C
C# F#

10

42
line of comment per line of code (min)

line of comment per line of code (min)

19
6
3

9
2

3
1

1
0

0
0 50 100 150 200 250 0 50 100 150 200

task task

Comments per line of code: all tasks Comments per line of code: all tasks

13.5
22

C C
Go Haskell

8.3
12
line of comment per line of code (min)

line of comment per line of code (min)

4.9
7

2.8
4

1.4
2

0.6
1
0

0 100 200 300 0 50 100 150 200 250 300 350

task task

Comments per line of code: all tasks Comments per line of code: all tasks
17

11

C C
Java Python
6.9
10
line of comment per line of code (min)

line of comment per line of code (min)

4.2
6

2.5
3

1.3
2

0.5
1
0

0 50 100 150 200 250 300 0 100 200 300

task task

Comments per line of code: all tasks

22

C
Ruby
12
line of comment per line of code (min)

7
4
2
1

123
0

0 50 100 150 200 250 300 350

task

Figure 148: Comments per line of code (min) of all tasks (C vs. other languages)
 Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

80
15

60
10
●

C#

F#

40
●

●
●

5
●

20
●

●
● ●
●
●
● ●
●
●
●
●
●●● ● ●● ●● ●
● ●● ● ● ●
●●
● ●●
●
● ●●
●
● ●
●●●●● ●
●
●
●
●
●●
●●
●
●● ●
●
●
0

0
0 5 10 15 0 20 40 60 80

C C

Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

14
● ●
20

12
10
●
15

● ●

●
●

8
Haskell
●
Go

●
10

6
●
●
●
● ●
●

4
●
●
●
5

● ●
●
● ●
●
● 2 ●
●
●
●
● ●
●
●
●
● ●
● ●
●●●
●
●●●
●● ●● ● ● ● ●● ●
●
●
●
●
●
●●●
●
●●●
●
●●● ●●● ●
● ● ●
● ●
●
●
● ●
●
●
● ●
●
●
●
●
●● ●●
●●
●
●
●●
●
●
●●
●●●
0

0 5 10 15 20 0 2 4 6 8 10 12 14

C C

Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

●
●
●
10

●
●
15

●
10

●
6
Python
Java

●
●
4

●
●
●
●
●
5

● ●
●
●
● ●
●
2

● ●
●
●
● ●
●
●
● ●
● ●
●
● ●● ● ● ● ● ● ● ●
●
●
●
●
●●●●● ●
●● ● ● ●● ● ● ●
●
●
● ●
●
●
● ●
●
●
●●
●
●
●●
●
●●
●●● ●●
●●
●
●
●●
●
●●
●
●●
●
●●
●●
●●
0

0 5 10 15 0 2 4 6 8 10

C C

Comments per line of code: all tasks (min)

●
20

●
15
Ruby

●
10

●
5

●
●

●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●●●● ●●
● ● ●●
124
0

0 5 10 15 20

Figure 149: Comments per line of code (min) of all tasks (C vs. other languages)
 Comments per line of code: all tasks Comments per line of code: all tasks

25

18
C# C#
F# Go

14

11
line of comment per line of code (min)

line of comment per line of code (min)

8

6
4

3
2

2
1

1
0

0
0 50 100 150 0 50 100 150 200 250

task task

Comments per line of code: all tasks Comments per line of code: all tasks
29

34
C# C#
Haskell Java
16

18
line of comment per line of code (min)

line of comment per line of code (min)

10
9
4

5
2

2
1

1
0

0 50 100 150 200 250 0 50 100 150 200 250

task task

Comments per line of code: all tasks Comments per line of code: all tasks
19

44

C# C#
Python Ruby
11

23
line of comment per line of code (min)

line of comment per line of code (min)

12
6
3

6
2

3
1

1
0

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 150: Comments per line of code (min) of all tasks (C# vs. other languages)

125
 Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

25
●

15
20
15
●
●

10
Go
F#

●
10

● ●
●

5
● ●
●
5

●
●
●
● ●
●
● ●
● ●
●
● ●
● ● ●
●●
● ●●● ● ● ● ●
● ● ● ● ● ●
●
● ●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●●
●
●●
●●●
0

0
0 5 10 15 20 25 0 5 10 15

C# C#

Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

35
●

30
25

25
20

20
Haskell

15

Java

●
15

● ●
10

● ●
10

●
5

●
5

● ●
● ●
●
● ●
●
● ●
●●
●●● ● ● ● ●
●
●●●● ● ● ●●● ● ●
●
● ●
●
●●
●
●
●
●●
●●
● ●
●●
●
●
●
●●
●●
0

0 5 10 15 20 25 0 5 10 15 20 25 30 35

C# C#

Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

● ●
40
15

30

●
Python

10

Ruby

20

●
●
5

10

●
●
● ●
●
●

● ●
● ●
● ● ●
● ●
●
●● ●● ●
●
● ● ● ● ● ● ●
●
● ●●
●
● ●
● ●
●● ●●
●
● ●
●
● ●
●
●
●
●
●●
●
●●●
● ●●
●
●
●●
●
●
0

0 5 10 15 0 10 20 30 40

C# C#

Figure 151: Comments per line of code (min) of all tasks (C# vs. other languages)

126
 Comments per line of code: all tasks Comments per line of code: all tasks

46

90
F# F#
Go Haskell

24

42
line of comment per line of code (min)

line of comment per line of code (min)

12

19
6

9
3

3
1

1
0

0
0 50 100 150 200 0 50 100 150 200

task task

Comments per line of code: all tasks Comments per line of code: all tasks
49

25
F# F#
Java Python
25

14
line of comment per line of code (min)

line of comment per line of code (min)

13

8
6

4
3

2
1

1
0

0 50 100 150 200 0 50 100 150 200

task task

Comments per line of code: all tasks

45

F#
Ruby
23
line of comment per line of code (min)

12
6
3
1
0

0 50 100 150 200

task

Figure 152: Comments per line of code (min) of all tasks (F# vs. other languages)

127
 Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

80
40

60
30

Haskell
Go

40
20

20
10

●
● ●
● ●
●●
●
●●●
●●●●
●● ●● ● ● ● ● ● ●
●●●
●● ● ●
●
●
●
●●
●
●●●
●● ● ●●
●
●
●
●●
●●
●● ●
0

0
0 10 20 30 40 0 20 40 60 80

F# F#

Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)
50

25
40

20
30

15 ●
Python
Java

20

10
10

● ●
● ●●
●● ● ●● ● ● ● ●
●●● ● ●
●
● ● ● ● ●● ● ● ●
●
●
●
●
●●
●
●●
●●● ● ● ●
●
●
●●
●
●●
●●
●
●●●● ● ● ●
0

0 10 20 30 40 50 0 5 10 15 20 25

F# F#

Comments per line of code: all tasks (min)

40
30
Ruby

20

●
10

●
●
●
● ●
● ●● ●●● ● ●● ● ● ●
●
●
●
●
●
●
●●
●
●●
●
●●
●● ● ●
0

0 10 20 30 40

F#

Figure 153: Comments per line of code (min) of all tasks (F# vs. other languages)

128
 Comments per line of code: all tasks Comments per line of code: all tasks
13

57
Go Go
Haskell Java

28
8
line of comment per line of code (min)

line of comment per line of code (min)

4.8

14
2.7

7
1.4

3
0.6

1
0

0
0 50 100 150 200 250 300 350 0 50 100 150 200 250 300

task task

Comments per line of code: all tasks Comments per line of code: all tasks
14.5

27

Go Go
Python Ruby
8.8

15
line of comment per line of code (min)

line of comment per line of code (min)

5.2

8
2.9

4
1.5

2
0.6

1
0

0 100 200 300 0 50 100 150 200 250 300 350

task task

Figure 154: Comments per line of code (min) of all tasks (Go vs. other languages)

129
 Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

●
12

50
10

40
8
Haskell

30
Java
6

20
●
●
●
4

●
●
●
●
●
●
●

10
2

●
●
●
●
●
●●●
●
●●●● ●
●● ●●
●●●
● ●
● ● ●
●
● ●
●
●
● ●
●
●
● ●●
●
●
●
●●●
●●●●●●
● ● ●● ● ●● ●
●●
●
●
●
●●
●●
●
●●
●
●
●●
●●
●● ●
●
●●
●
0

0
0 2 4 6 8 10 12 0 10 20 30 40 50

Go Go

Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)
15

25
20
10

●
15
Python

●
Ruby

●
●
10

●
5

●
●
● ●
●

●
5

● ●
● ●
●
●
●
● ●
●
●
● ● ● ●
●
● ●●
●●● ●●● ●●●●● ● ● ● ● ● ● ● ●
●
● ●
●
● ● ●
●
●●●
● ●● ●● ● ● ●
●
● ●
●
●
● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●● ●●
●●
●
●
●
●●
●●●
0

0 5 10 15 0 5 10 15 20 25

Go Go

Figure 155: Comments per line of code (min) of all tasks (Go vs. other languages)

130
 Comments per line of code: all tasks Comments per line of code: all tasks
60

35
Haskell Haskell
Java Python
30

19
line of comment per line of code (min)

line of comment per line of code (min)

14

10
7

5
3

2
1

1
0

0
0 50 100 150 200 250 300 0 50 100 150 200 250 300 350

task task

Comments per line of code: all tasks

22

Haskell
Ruby
13
line of comment per line of code (min)

7
4
2
1
0

0 50 100 150 200 250 300

task

Figure 156: Comments per line of code (min) of all tasks (Haskell vs. other languages)

131
 Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

35
60

30
50

25
40

20
Python
Java

30

15
20
●

10
●
●
●
●
10

5
●
●
●
●
●
●
● ● ●
● ● ●●
●
●●●● ●●● ● ● ● ●
●●
●●●●● ● ● ● ● ●
●
●
●
●
●
●● ●●
●
●
●
●●
●
● ●
0

0
0 10 20 30 40 50 60 0 5 10 15 20 25 30 35

Haskell Haskell

Comments per line of code: all tasks (min)

20
15
Ruby

10

●
5

●
●
●
●
●
●
●
● ● ●●●
●●●● ● ● ● ● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
0

0 5 10 15 20

Haskell

Figure 157: Comments per line of code (min) of all tasks (Haskell vs. other languages)

Comments per line of code: all tasks Comments per line of code: all tasks
36

99

Java Java
Python Ruby
19

46
line of comment per line of code (min)

line of comment per line of code (min)

10

21
5

9
2

4
1

1
0

0 50 100 150 200 250 300 0 50 100 150 200 250 300

task task

Figure 158: Comments per line of code (min) of all tasks (Java vs. other languages)

132
 Comments per line of code: all tasks (min) Comments per line of code: all tasks (min)

100
● ●

30

80
60
20
Python

Ruby
●
●
●

40
●

10

20
●
●
● ●
●
● ●● ●
●
● ●
●
● ●
●
●●●
●
●●
● ●●
● ●●
●
●
●
●
●
●●
●● ●●
●
●●●
●●●●
● ●
0

0
0 10 20 30 0 20 40 60 80 100

Java Java

Figure 159: Comments per line of code (min) of all tasks (Java vs. other languages)

Comments per line of code: all tasks

26

Python
Ruby
15
line of comment per line of code (min)

8
4
2
1
0

0 100 200 300

task

Figure 160: Comments per line of code (min) of all tasks (Python vs. other languages)

Comments per line of code: all tasks (min)

●
25
20
15
Ruby

●
10

●
5

●
●
●
●
● ●
●
●
●
● ●●●●
● ● ●● ●●
●● ●● ●●
●
●
●●
●
●
●
●●
●
●●●
●●
0

0 5 10 15 20 25

Python

Figure 161: Comments per line of code (min) of all tasks (Python vs. other languages)

133
 Comments per line of code: all tasks

C
5

C#
F#
Go
3.5

Haskell
Java
line of comment per line of code (min)

Python
Ruby
2.3
1.4
0.8
0.3
0

0 50 100 150

task

Figure 162: Comments per line of code (min) of all tasks (all languages)

134
 Comments per line of code: all tasks Comments per line of code: all tasks

17

90
C C
C# F#

10

42
line of comment per line of code (mean)

line of comment per line of code (mean)

19
6
3

9
2

3
1

1
0

0
0 50 100 150 200 250 0 50 100 150 200

task task

Comments per line of code: all tasks Comments per line of code: all tasks
22

18
C C
Go Haskell
12

11
line of comment per line of code (mean)

line of comment per line of code (mean)

7

6
4

3
2

2
1

1
0

0 100 200 300 0 50 100 150 200 250 300 350

task task

Comments per line of code: all tasks Comments per line of code: all tasks
17

22

C C
Java Python
10

12
line of comment per line of code (mean)

line of comment per line of code (mean)

6

7
3

4
2

2
1

1
0

0 50 100 150 200 250 300 0 100 200 300

task task

Comments per line of code: all tasks

40

C
Ruby
21
line of comment per line of code (mean)

11
5
2
1

135
0

0 50 100 150 200 250 300 350

task

Figure 163: Comments per line of code (mean) of all tasks (C vs. other languages)
 Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

80
15

60
10
●

C#

F#

40
●

●
●

5
●
●

20
●
●
●
● ●

● ●●
● ●
●
●
●
●
●
●●● ●●● ●● ● ●● ● ● ● ●●
● ●●
●
● ●●
●
●
● ●
●●
●●●● ●
●
●
●●
●
●
●●
●
●●
●
●● ●
●
●
0

0
0 5 10 15 0 20 40 60 80

C C

Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

● ●
20

15
●
15

●
●

10
Haskell
● ●
Go

●
10

●
● ●
●
●

5
● ●
5

● ●
● ●
●
●
● ● ●
●
●
● ●
●
●
● ●
●● ● ●
●
● ●
● ●
● ●●
● ●
● ● ● ●
● ● ● ●
●●●
●
●
●●●
● ●●
●●●● ●●● ●
●
● ● ● ● ●●●●●●● ● ● ●●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
● ●
●
●●
●
●
●●
●
●●●
●
0

0 5 10 15 20 0 5 10 15

C C

Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

● ●
20

●
15

15
10

Python
Java

●
●
10

● ●

● ●
●
5

●
●
5

●
●●
●
●
●
● ●
● ●
● ●●
●
● ● ● ●
●
● ● ●
●
● ●
●
●
●●● ● ●● ●● ● ●● ● ● ●
●● ●
●
●●
● ●● ● ● ●
● ●●
●
●
●
●
●●
●●
●
● ●
●
●●
●
●
●●
●●
●●
●
●●
●
●●
● ●
● ●●
●
●
●●
●
●●
●
●●
●
0

0 5 10 15 0 5 10 15 20

C C

Comments per line of code: all tasks (mean)

●
40
30

●
Ruby

20

●
10

●
●
●

●
●● ●
●
●
● ●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●● ● 136
0

0 10 20 30 40

Figure 164: Comments per line of code (mean) of all tasks (C vs. other languages)
 Comments per line of code: all tasks Comments per line of code: all tasks

25

18
C# C#
F# Go

14

11
line of comment per line of code (mean)

line of comment per line of code (mean)

8

6
4

3
2

2
1

1
0

0
0 50 100 150 0 50 100 150 200 250

task task

Comments per line of code: all tasks Comments per line of code: all tasks
29

34
C# C#
Haskell Java
16

18
line of comment per line of code (mean)

line of comment per line of code (mean)

10
9
4

5
2

2
1

1
0

0 50 100 150 200 250 0 50 100 150 200 250

task task

Comments per line of code: all tasks Comments per line of code: all tasks
19

44

C# C#
Python Ruby
11

23
line of comment per line of code (mean)

line of comment per line of code (mean)

12
6
3

6
2

3
1

1
0

0 50 100 150 200 250 0 50 100 150 200 250

task task

Figure 165: Comments per line of code (mean) of all tasks (C# vs. other languages)

137
 Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

25
●

15
20
15
●
●

10
Go
F#

●
10

● ●
●
●
●

5
● ●
● ●
5

●
● ●
●
● ●
● ●
●
● ●
●●
● ●
● ●●●● ● ● ● ●
● ● ● ● ● ●
●
● ●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●●
●
●●●●
●
0

0
0 5 10 15 20 25 0 5 10 15

C# C#

Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

35
●

30
25

25
20

20
Haskell

15

Java

●
15

● ●
10

● ●
10

●
5

●
5

● ●
● ●
●
● ●
●
●
● ● ●
●●● ● ●
● ●
● ● ● ●●●● ● ● ● ●
●
● ●
●
●●
●
●
●
●●
●●
● ●
●
●
●
●●
●●
●●
0

0 5 10 15 20 25 0 5 10 15 20 25 30 35

C# C#

Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

● ●
40
15

30

●
Python

10

Ruby

20

●
●
●

●
5

10

● ●
●
● ●
● ●

●
● ●
● ●
● ● ●
●● ●● ● ●
●
● ● ● ● ● ● ●
●
●
●
●●
●
● ● ●●
●
● ●
●● ●●●
● ●
●
●
●
● ●● ●
●
●
●●
●
●●● ●●
●
●
●●
●
●
0

0 5 10 15 0 10 20 30 40

C# C#

Figure 166: Comments per line of code (mean) of all tasks (C# vs. other languages)

138
 Comments per line of code: all tasks Comments per line of code: all tasks

46

90
F# F#
Go Haskell

24

42
line of comment per line of code (mean)

line of comment per line of code (mean)

12

19
6

9
3

3
1

1
0

0
0 50 100 150 200 0 50 100 150 200

task task

Comments per line of code: all tasks Comments per line of code: all tasks
49

25
F# F#
Java Python
25

14
line of comment per line of code (mean)

line of comment per line of code (mean)

13

8
6

4
3

2
1

1
0

0 50 100 150 200 0 50 100 150 200

task task

Comments per line of code: all tasks

45

F#
Ruby
23
line of comment per line of code (mean)

12
6
3
1
0

0 50 100 150 200

task

Figure 167: Comments per line of code (mean) of all tasks (F# vs. other languages)

139
 Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

80
40

60
30

Haskell
Go

40
20

20
10

●
●
●
● ●
●
●
●●●
●●●●
●● ●● ● ● ● ● ● ●●
●
● ●●●
● ●● ●
●
●
●●
●
●●●
●● ● ●●
●
●
●
●●
●
●●
●● ●
0

0
0 10 20 30 40 0 20 40 60 80

F# F#

Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)
50

25
40

20
30

15 ●
Python
Java

20

10

●
10

●
●
●
● ●
● ● ●● ● ●● ● ● ●
●
●● ●
●
●●● ● ● ●● ● ● ●●
●
●
●
●
●
●●
●●
●●● ● ● ●
●
●●
●●
●●
●
●●●
●
●●
●●●● ● ● ●
0

0 10 20 30 40 50 0 5 10 15 20 25

F# F#

Comments per line of code: all tasks (mean)

40
30
Ruby

20

●
10

●
●
●
● ●
● ●
● ●● ●●
●● ● ● ● ● ●
●
●●
●
●
●
●●
●
●●
●●
●●
●
●●● ● ●
●
0

0 10 20 30 40

F#

Figure 168: Comments per line of code (mean) of all tasks (F# vs. other languages)

140
 Comments per line of code: all tasks Comments per line of code: all tasks
13

57
Go Go
Haskell Java

28
8
line of comment per line of code (mean)

line of comment per line of code (mean)

4.8

14
2.7

7
1.4

3
0.6

1
0

0
0 50 100 150 200 250 300 350 0 50 100 150 200 250 300

task task

Comments per line of code: all tasks Comments per line of code: all tasks
14.5

27

Go Go
Python Ruby
8.8

15
line of comment per line of code (mean)

line of comment per line of code (mean)

5.2

8
2.9

4
1.5

2
0.6

1
0

0 100 200 300 0 50 100 150 200 250 300 350

task task

Figure 169: Comments per line of code (mean) of all tasks (Go vs. other languages)

141
 Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

●
12

50
10

40
8
Haskell

30
Java
●
6

●
●

20
●
●
4

●
●
●
●
●●

10
●
2

●
●
●
● ● ●
●
●●●
●●● ●●●● ●●
● ●● ●
●● ● ●
● ●●
● ●
●
●
●
● ● ●
●
●●
●●
● ●●●
●●●●
●● ●
● ● ●● ●●
● ●● ●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●●
●●●
●
● ●
●
●
●
●●
●
0

0
0 2 4 6 8 10 12 0 10 20 30 40 50

Go Go

Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)
15

25

●
20
10

●
●
15
Python

●
Ruby

●
●
●
10
5

●
●
● ●
●

●
5

●
● ●
● ●
●
●
●
● ●
● ●
●● ●
● ●
● ●
● ●
● ●●●● ●●● ●●●●● ● ● ● ● ● ● ●
●●
●● ●
●
● ● ●
●
●
●
●
●● ●
● ●●
●●● ●● ●● ● ● ●
●
●
●
●●
● ●
●
●
●●
●
●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●●
●● ●
●
●●
●●
●
●
●
●●
●●●
0

0 5 10 15 0 5 10 15 20 25

Go Go

Figure 170: Comments per line of code (mean) of all tasks (Go vs. other languages)

142
 Comments per line of code: all tasks Comments per line of code: all tasks
60

35
Haskell Haskell
Java Python
30

19
line of comment per line of code (mean)

line of comment per line of code (mean)

14

10
7

5
3

2
1

1
0

0
0 50 100 150 200 250 300 0 50 100 150 200 250 300 350

task task

Comments per line of code: all tasks

22

Haskell
Ruby
13
line of comment per line of code (mean)

7
4
2
1
0

0 50 100 150 200 250 300

task

Figure 171: Comments per line of code (mean) of all tasks (Haskell vs. other languages)

143
 Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

35
60

30
50

25
40

20
Python
Java

30

15
●
20
●

10
●
●
●
●
10

5
●
●
●
●
● ● ●
●
● ● ●●
●● ● ●●●●● ●● ● ● ● ●
●●●●●● ●
● ● ● ●●
●
●
●
●●
● ●
●
●
●●
●●
●●
●
●● ●
0

0
0 10 20 30 40 50 60 0 5 10 15 20 25 30 35

Haskell Haskell

Comments per line of code: all tasks (mean)

20
15
Ruby

10

●
5

●
● ●
●
●
●
●
●●
● ●●
●●●
●
●● ● ● ● ● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
0

0 5 10 15 20

Haskell

Figure 172: Comments per line of code (mean) of all tasks (Haskell vs. other languages)

Comments per line of code: all tasks Comments per line of code: all tasks
36

99

Java Java
Python Ruby
19

46
line of comment per line of code (mean)

line of comment per line of code (mean)

10

21
5

9
2

4
1

1
0

0 50 100 150 200 250 300 0 50 100 150 200 250 300

task task

Figure 173: Comments per line of code (mean) of all tasks (Java vs. other languages)

144
 Comments per line of code: all tasks (mean) Comments per line of code: all tasks (mean)

100
● ●

30

80
60
20
Python

Ruby
●
●
●

40
●
●
10

20
●
●● ●
●
● ●
● ●● ● ●
●
● ●
●
● ●
●●●
●
● ●●
● ● ●
●
●
●● ●●
●●
● ●
●●
●
●
●
●
●
●●
●● ●●
●
●
● ●● ●
0

0
0 10 20 30 0 20 40 60 80 100

Java Java

Figure 174: Comments per line of code (mean) of all tasks (Java vs. other languages)

Comments per line of code: all tasks

26

Python
Ruby
15
line of comment per line of code (mean)

8
4
2
1
0

0 100 200 300

task

Figure 175: Comments per line of code (mean) of all tasks (Python vs. other languages)

Comments per line of code: all tasks (mean)

●
25
20
15
Ruby

●
10

●
●
5

● ●
●
●●
●
●●●
●
● ●
● ●●●●● ●●●
●●● ●● ●
●
●●●
●
●
●●
●
●●
●
●
●●
●
●●
●●●
●●
0

0 5 10 15 20 25

Python

Figure 176: Comments per line of code (mean) of all tasks (Python vs. other languages)

145
 Comments per line of code: all tasks

C
5

C#
F#
Go
3.5

Haskell
line of comment per line of code (mean)

Java
Python
Ruby
2.3
1.4
0.8
0.3
0

0 50 100 150

task

Figure 177: Comments per line of code (mean) of all tasks (all languages)

146
D. Size of binaries

147
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

C C

3
C# F#

3.3

2.6
2.7

2.2
size of binaries (min)

size of binaries (min)

2.2

1.8
1.7

1.5
1.3

1.2
1

1
0 50 100 150 200 0 50 100 150

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
559

595
C C
Go Haskell
218

230
85

88
size of binaries (min)

size of binaries (min)

32

34
12

12
4

4
1

0 50 100 150 200 250 300 0 50 100 150 200 250

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

C C
4

Java Python
3.3

3.3
2.7

2.7
size of binaries (min)

size of binaries (min)

2.2

2.2
1.7

1.7
1.3

1.3
1

0 50 100 150 200 250 0 50 100 150 200 250 300 350

task task

Figure 178: Size of binaries (min) of tasks compiling successfully (C vs. other languages)

148
 Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)

4.0

3.0
3.5
●

2.5
3.0
●

2.5

2.0
C#

F#
●
2.0

1.5
●
1.5

●
1.0

1.0
● ● ● ● ● ● ● ● ● ● ●

1.0 1.5 2.0 2.5 3.0 3.5 4.0 1.0 1.5 2.0 2.5 3.0

C C

Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)

600
● ●

●
500

500
●
400

● ●

400
●
●
●
●
●
300

Haskell

●
300
Go

●
●

●
●
●
● ●
200

● ●
●
200

● ●
●
●
● ●
●
● ●
●
●
●
● ●
● ●
●
● ●
●
●
●
●
● ●
● ●
●
●
100

●
●
100

●
●
● ●
●
●
●
● ●
●
● ●
●
●
●
●
0
0

0 100 200 300 400 500 0 100 200 300 400 500 600

C C

Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)
4.0

4.0

●
3.5

3.5
3.0

3.0
Python
Java

2.5

2.5

●
2.0

2.0

●
1.5

1.5

●
1.0

1.0

● ● ● ● ● ● ● ● ● ●

1.0 1.5 2.0 2.5 3.0 3.5 4.0 1.0 1.5 2.0 2.5 3.0 3.5 4.0

C C

Figure 179: Size of binaries (min) of tasks compiling successfully (C vs. other languages)

149
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

1676
C# C#

4
F# Go

545
3.3

177
2.7
size of binaries (min)

size of binaries (min)

2.2

57
1.7

18
1.3

5
1

1
0 50 100 150 0 50 100 150 200

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
1159

5.5
C# C#
Haskell Java
401

4.3
138

3.4
size of binaries (min)

size of binaries (min)

2.6
47
16

2
1.4
5
1

0 50 100 150 200 0 50 100 150 200

task task

Size of binaries: tasks compiling successfully

C#
3

Python
2.6
2.2
size of binaries (min)

1.8
1.5
1.2
1

0 50 100 150 200 250

task

Figure 180: Size of binaries (min) of tasks compiling successfully (C# vs. other languages)

150
 Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)

4.0
● ●

1500
3.5
●

3.0
●

1000
●
2.5

Go
F#

● ●
●
● ●

●
●
●
2.0

●
●
● ●

500
●
●
●
●

● ●
●
●
1.5

● ●
●
●
●
●
●
●
●
●
●
1.0

● ●

0
1.0 1.5 2.0 2.5 3.0 3.5 4.0 0 500 1000 1500

C# C#

Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)
1200

● ●
●

●
1000

5
●

●
800

4
●
●
Haskell

600

Java

●
●
●
●
3

●
●
●
●
●
●
400

●
●
●
●
●
●
●
●
●
●
●
●
● ●
2

●
●
●
●
200

●
●
●
●
●
●
●
● ●
● ●

● ● ● ●
1
0

0 200 400 600 800 1000 1200 1 2 3 4 5

C# C#

Size of binaries: tasks compiling successfully (min)

3.0
2.5
Python

2.0

●
1.5
1.0

● ● ●

1.0 1.5 2.0 2.5 3.0

C#

Figure 181: Size of binaries (min) of tasks compiling successfully (C# vs. other languages)

151
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
1400

665
F# F#
Go Haskell
469

252
157

95
size of binaries (min)

size of binaries (min)

52

35
17

13
5

4
1

1
0 50 100 150 0 50 100 150

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

F# F#
5

Java Python
4

6
3.2

4.5
size of binaries (min)

size of binaries (min)

2.5

3.2
1.9

2.3
1.4

1.6
1

0 50 100 150 0 50 100 150 200

task task

Figure 182: Size of binaries (min) of tasks compiling successfully (F# vs. other languages)

152
 Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)
1400

● ●

600
1200

●
●

500
1000

400
800

● ●

Haskell
●
Go

300
●
600

●
●
●
●
●
●
●
● ●
●
● ●
●
● ●
●
●
●
●
●

200
●
400

●
●
●
● ●
● ●
●
●
●
●
●
●
● ●
●
●
● ●

100
● ●
200

● ●
● ●
●
●
● ●
● ●
●
● ●
●
● ●
●
● ●
●
●
●
●
0

0
0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 600

F# F#

Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)
5

8
7
4

6
5
Python
Java

4
3

●
2

●
2

●
●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
1

1 2 3 4 5 1 2 3 4 5 6 7 8

F# F#

Figure 183: Size of binaries (min) of tasks compiling successfully (F# vs. other languages)

153
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

1582
5.5

Go Go
Haskell Java

519
4.3

170
3.4
size of binaries (min)

size of binaries (min)

2.6

55
18
2
1.4

5
1

1
0 50 100 150 200 250 0 50 100 150 200 250

task task

Size of binaries: tasks compiling successfully

1676

Go
Python
545
177
size of binaries (min)

57
18
5
1

0 50 100 150 200 250 300 350

task

Figure 184: Size of binaries (min) of tasks compiling successfully (Go vs. other languages)

154
 Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)

1500
5
4

1000
●

Haskell

Java
3
●
●

500
●
●
●
●
●
2

●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●●●●●
●
●●●
●
●●
●
●●
●●
●
●●●
●
●
●●●
●
●●●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●●●● ●
●
●●
●● ● ● ● ● ●●●
●●
●●
●
●●●● ●
●●
●
●●●
● ● ●●
●●● ●
●
●
●●●
●
●
●●
●● ●
●
●●
● ●●
●●
●●● ●● ●
● ●
1

0
1 2 3 4 5 0 500 1000 1500

Go Go

Size of binaries: tasks compiling successfully (min)

1500
1000
Python

500

●
● ●
● ● ●●
●
●● ●
● ●
● ●
●
●
●●
●●●
●●
●
●●
●
●●●
●
●●
●●●●
●
●●●
●● ● ● ●● ● ●
● ● ●
● ●
0

0 500 1000 1500

Go

Figure 185: Size of binaries (min) of tasks compiling successfully (Go vs. other languages)

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
3385

3385

Haskell Haskell
Java Python
980

980
283

283
size of binaries (min)

size of binaries (min)

81

81
23

23
6

6
1

0 50 100 150 200 0 50 100 150 200 250 300

task task

Figure 186: Size of binaries (min) of tasks compiling successfully (Haskell vs. other languages)

155
 Size of binaries: tasks compiling successfully (min) Size of binaries: tasks compiling successfully (min)

3500

3500
3000

3000
2500

2500
2000

2000
Python
Java

1500

1500
1000

1000
500

500
●●
●●●●
●
●●
●●●●
●●
●●
●●
●
●●●
●●
●
●●
●●
●
●●
●●●
●●
●
●●
●●
●●● ●
●● ● ● ●● ● ● ●
●
●
●● ●
●●
●
●●●
●●
●
●●
●
●●
●
●
●●
●●
●●●
●
●
●●
●●
●●
●●● ●
●●● ●● ● ● ●● ● ● ● ●● ● ●
0

0
0 500 1000 1500 2000 2500 3000 3500 0 500 1000 1500 2000 2500 3000 3500

Haskell Haskell

Figure 187: Size of binaries (min) of tasks compiling successfully (Haskell vs. other languages)

Size of binaries: tasks compiling successfully

11

Java
Python
8
6
size of binaries (min)

4
3
2
1

0 50 100 150 200 250

task

Figure 188: Size of binaries (min) of tasks compiling successfully (Java vs. other languages)

Size of binaries: tasks compiling successfully (min)

10
8
Python

6
4

●
2

● ● ● ● ● ● ● ●

2 4 6 8 10

Java

Figure 189: Size of binaries (min) of tasks compiling successfully (Java vs. other languages)

156
Figure 190: Size of binaries (min) of tasks compiling successfully (Python vs. other languages)

Figure 191: Size of binaries (min) of tasks compiling successfully (Python vs. other languages)

Size of binaries: tasks compiling successfully

1582

C
C#
F#
Go
519

Haskell
Java
Python
170
size of binaries (min)

55
18
5
1

0 20 40 60 80 100

task

Figure 192: Size of binaries (min) of tasks compiling successfully (all languages)

157
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

C C

3
C# F#

3.3

2.6
2.7

2.2
size of binaries (mean)

size of binaries (mean)

2.2

1.8
1.7

1.5
1.3

1.2
1

1
0 50 100 150 200 0 50 100 150

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
560

595
C C
Go Haskell
218

230
85

88
size of binaries (mean)

size of binaries (mean)

32

34
12

12
4

4
1

0 50 100 150 200 250 300 0 50 100 150 200 250

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

C C
5

Java Python
4

4
3.2

3.2
size of binaries (mean)

size of binaries (mean)

2.5

2.5
1.9

1.9
1.4

1.4
1

0 50 100 150 200 250 0 50 100 150 200 250 300 350

task task

Figure 193: Size of binaries (mean) of tasks compiling successfully (C vs. other languages)

158
 Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)

4.0

3.0
3.5
●

2.5
3.0
●

2.5

2.0
C#

F#
● ●
2.0

● ● ●

1.5
● ● ● ●
1.5

●
● ●
●
● ●
1.0

1.0
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

1.0 1.5 2.0 2.5 3.0 3.5 4.0 1.0 1.5 2.0 2.5 3.0

C C

Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)

600
● ●

●
500

500
●
400

● ●

400
●
●
●
●
●
300

Haskell
●
300
Go

●
●

●
●
●
●
● ●
200

● ●
●
200

●
●
●
●
● ●
●
● ●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
● ●
● ●
●
●
100

●
●
100

●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
0
0

0 100 200 300 400 500 0 100 200 300 400 500 600

C C

Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)

● ●
5

5
4

●
Python
Java

● ●
2

● ● ● ●
● ● ● ● ●
● ●

● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●
1

1 2 3 4 5 1 2 3 4 5

C C

Figure 194: Size of binaries (mean) of tasks compiling successfully (C vs. other languages)

159
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

1679
C# C#

4
F# Go

546
3.3

177
2.7
size of binaries (mean)

size of binaries (mean)

2.2

57
1.7

18
1.3

5
1

1
0 50 100 150 0 50 100 150 200

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
1159

5.5
C# C#
Haskell Java
401

4.3
138

3.4
size of binaries (mean)

size of binaries (mean)

2.6
47
16

2
1.4
5
1

0 50 100 150 200 0 50 100 150 200

task task

Size of binaries: tasks compiling successfully

C#
5

Python
4
3.2
size of binaries (mean)

2.5
1.9
1.4
1

0 50 100 150 200 250

task

Figure 195: Size of binaries (mean) of tasks compiling successfully (C# vs. other languages)

160
 Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)

4.0
● ●

1500
3.5
●

3.0
● ● ●

1000
●
2.5

Go
F#

● ●
●
● ●
●
●
●
●
2.0

●
●
● ● ●● ● ●
●

500
●
●
●
●
●
●
●
●
● ● ●
●
●
1.5

● ●
●
●
●
●
●
●
●
●
●
●
1.0

● ●

0
1.0 1.5 2.0 2.5 3.0 3.5 4.0 0 500 1000 1500

C# C#

Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)
1200

● ●
●

●
1000

5
●

●
800

4
●
●
Haskell

600

Java

●
●
●
●
3

●
●
●
●
●
●
●
400

●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
2

●
●
●
●
200

●
●
●
●
●
●
●
● ● ● ●
●
●
● ●
● ●

● ● ●● ● ● ●
1
0

0 200 400 600 800 1000 1200 1 2 3 4 5

C# C#

Size of binaries: tasks compiling successfully (mean)

●
5
4
Python

●
2

● ● ●● ● ● ● ●
1

1 2 3 4 5

C#

Figure 196: Size of binaries (mean) of tasks compiling successfully (C# vs. other languages)

161
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
1400

665
F# F#
Go Haskell
469

252
157

95
size of binaries (mean)

size of binaries (mean)

52

35
17

13
5

4
1

1
0 50 100 150 0 50 100 150

task task

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

F# F#
5

Java Python
4

6
3.2

4.5
size of binaries (mean)

size of binaries (mean)

2.5

3.2
1.9

2.3
1.4

1.6
1

0 50 100 150 0 50 100 150 200

task task

Figure 197: Size of binaries (mean) of tasks compiling successfully (F# vs. other languages)

162
 Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)
1400

● ●

600
1200

●
●

500
●
1000

400
800

● ●

Haskell
●
Go

300
●
600

●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
● ●
●
● ●
● ●
●

200
●
400

● ●
●
● ●
●
●
●
●
●
● ●
●
● ●
●
●
● ●

100
● ●
200

● ●
● ●
●
●
● ●
● ●
●
● ●
● ●
●
●
● ●
●
●
●
●
0

0
0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 600

F# F#

Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)

●
5

8
7
4

6
5
Python
Java

4
3

● ● ●
2

●
2

● ● ●
●
● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●
1

1 2 3 4 5 1 2 3 4 5 6 7 8

F# F#

Figure 198: Size of binaries (mean) of tasks compiling successfully (F# vs. other languages)

163
 Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully

1582
5.5

Go Go
Haskell Java

519
4.3

170
3.4
size of binaries (mean)

size of binaries (mean)

2.6

55
18
2
1.4

5
1

1
0 50 100 150 200 250 0 50 100 150 200 250

task task

Size of binaries: tasks compiling successfully

1679

Go
Python
546
177
size of binaries (mean)

57
18
5
1

0 50 100 150 200 250 300 350

task

Figure 199: Size of binaries (mean) of tasks compiling successfully (Go vs. other languages)

164
 Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)

1500
5
4

1000
●

Haskell

Java
3
●
●

500
●
●●
●
●
● ●
●
●
● ●
2

● ●
●
●
●
●●●
● ● ●●
● ●
●●
● ● ●
● ●● ●
●
●
● ●●
● ●●
● ●● ●● ●
●●
●●
●●
●●
●●●●●
●
●●●
●
●●
●
●●
●●
●
●●●
●●
●
●
●●●
●
●●●
●
●●
●
●●
●
●●
●●●
●
●
●●
●●
● ●
●●●
●●● ● ● ●●●●
●●
●●
●
●●●● ●
●●
●
●●
●●
● ● ●●●●● ●
●
●● ●
●●●
●
●●● ●
● ●
●●
●
●●●●●●●●
●● ●● ●
● ●
1

0
1 2 3 4 5 0 500 1000 1500

Go Go

Size of binaries: tasks compiling successfully (mean)

1500
1000
Python

500

●
●
● ● ● ●●
●
●● ●
● ● ● ●
●
●
●●
●
●●
●
●●
●●
●
●
●●
●●●
●
●●
●
●●●
●●●
●
●●●●
●● ● ●● ● ●
● ● ●
● ●
0

0 500 1000 1500

Go

Figure 200: Size of binaries (mean) of tasks compiling successfully (Go vs. other languages)

Size of binaries: tasks compiling successfully Size of binaries: tasks compiling successfully
3385

3385

Haskell Haskell
Java Python
980

980
283

283
size of binaries (mean)

size of binaries (mean)

81

81
23

23
6

6
1

0 50 100 150 200 0 50 100 150 200 250 300

task task

Figure 201: Size of binaries (mean) of tasks compiling successfully (Haskell vs. other languages)

165
 Size of binaries: tasks compiling successfully (mean) Size of binaries: tasks compiling successfully (mean)

3500

3500
3000

3000
2500

2500
2000

2000
Python
Java

1500

1500
1000

1000
500

500
●●
●
●●●
●
●●
●●
●
●●●●
●●
●
●
●●●
●●
●●
●●●
●●
●
●
●●
●
●
●●
●●
●●●
●
●
●●
●●
●
●●● ●
●● ● ● ●● ● ● ●
●
●
●● ●
●●
●
●●●
●●
●
●●
●●
●
●●
●
●
●●
●
●●
●●●
●
●
●●
●●
●●
●●● ●
●●● ●● ● ● ●● ● ● ● ●● ● ●
0

0
0 500 1000 1500 2000 2500 3000 3500 0 500 1000 1500 2000 2500 3000 3500

Haskell Haskell

Figure 202: Size of binaries (mean) of tasks compiling successfully (Haskell vs. other languages)

Size of binaries: tasks compiling successfully

11

Java
Python
8
size of binaries (mean)

6
4
3
2
1

0 50 100 150 200 250

task

Figure 203: Size of binaries (mean) of tasks compiling successfully (Java vs. other languages)

Size of binaries: tasks compiling successfully (mean)

10
8
Python

●
4

●
2

●
● ●● ● ● ●● ● ● ● ● ● ● ●

2 4 6 8 10

Java

Figure 204: Size of binaries (mean) of tasks compiling successfully (Java vs. other languages)

166
Figure 205: Size of binaries (mean) of tasks compiling successfully (Python vs. other languages)

Figure 206: Size of binaries (mean) of tasks compiling successfully (Python vs. other languages)

Size of binaries: tasks compiling successfully

1582

C
C#
F#
Go
519

Haskell
Java
Python
170
size of binaries (mean)

55
18
5
1

0 20 40 60 80 100

task

Figure 207: Size of binaries (mean) of tasks compiling successfully (all languages)

167
E. Performance

168
 Performance: tasks running without errors Performance: tasks running without errors

0.12

0.2
C C
C# F#

0.16
0.1
0.08

0.13
running time (min)

running time (min)

0.06

0.1
0.04

0.06
0.02

0.03
0

0
2 4 6 8 10 12 14 2 4 6 8 10 12 14

task task

Performance: tasks running without errors Performance: tasks running without errors
0.24

0.9
C C
Go Haskell
0.2

0.7
0.15

0.5
running time (min)

running time (min)

0.11

0.4
0.07

0.2
0.04

0.1
0

0 5 10 15 20 25 30 0 5 10 15 20 25

task task

Performance: tasks running without errors Performance: tasks running without errors
2.7

24

C C
Java Python
14
2
1.4

8
running time (min)

running time (min)

0.9

4
0.5

2
0.2

1
0

5 10 15 20 0 5 10 15 20 25 30 35

task task

Performance: tasks running without errors

C
3

Ruby
2.2
1.5
running time (min)

1
0.6
0.3

169
0

0 5 10 15 20 25 30

task

Figure 208: Performance (min) of tasks running successfully (C vs. other languages)
 Performance: tasks running without errors (min) Performance: tasks running without errors (min)

0.12

0.20
● ●

0.10
●

0.15
●

0.08
●

0.06

0.10
C#

F#
●

0.04
●
●

0.05
●
0.02
●
●

●
●
●
0.00

0.00
●

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.00 0.05 0.10 0.15 0.20

C C

Performance: tasks running without errors (min) Performance: tasks running without errors (min)

0.8
0.20

0.6
0.15

Haskell
Go

0.4
0.10

0.2
0.05
0.00

●
0.0

●
● ● ● ● ●

0.00 0.05 0.10 0.15 0.20 0.0 0.2 0.4 0.6 0.8

C C

Performance: tasks running without errors (min) Performance: tasks running without errors (min)
2.5

20
2.0

15
1.5

Python
Java

10
1.0

●
5
0.5

●
●
● ● ●
0.0

●
0

0.0 0.5 1.0 1.5 2.0 2.5 0 5 10 15 20

C C

Performance: tasks running without errors (min)

3.0
2.5
2.0
Ruby

1.5
1.0

●
0.5

● 170
0.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

Figure 209: Performance (min) of tasks running successfully (C vs. other languages)
 Performance: tasks running without errors Performance: tasks running without errors

190
7.5
C# C#
F# Go

5.7

79
4.2

32
running time (min)

running time (min)

3.1

13
2.2

5
1.5

1
1

0
2 4 6 8 10 5 10 15

task task

Performance: tasks running without errors Performance: tasks running without errors

11
C# C#
2

Haskell Java
1.5

8
1.1

6
running time (min)

running time (min)

0.7

4
0.4

3
0.2

2
0

5 10 15 2 4 6 8 10 12 14

task task

Performance: tasks running without errors Performance: tasks running without errors
76

C# C#
8

Python Ruby
41
6
4.5

22
running time (min)

running time (min)

3.2

11
2.3

6
1.6

3
1

5 10 15 5 10 15

task task

Figure 210: Performance (min) of tasks running successfully (C# vs. other languages)

171
 Performance: tasks running without errors (min) Performance: tasks running without errors (min)

● ●

150
6
●

5
●

100
Go
F#

●
4

●
●
3

50
2

● ● ●●
●
1

0
1 2 3 4 5 6 7 0 50 100 150

C# C#

Performance: tasks running without errors (min) Performance: tasks running without errors (min)
2.0

10
●
1.5

8
Haskell

Java
1.0

●
6

●
4
0.5

●
●
●
●
●
●
2

●
0.0

●
●●●
●
●● ● ●

0.0 0.5 1.0 1.5 2.0 2 4 6 8 10

C# C#

Performance: tasks running without errors (min) Performance: tasks running without errors (min)
8
7

60
6
5
Python

40
Ruby
4
3

20

●
2

●
●
● ● ● ● ● ● ● ● ●
●
●
●●
● ●
1

1 2 3 4 5 6 7 8 0 20 40 60

C# C#

Figure 211: Performance (min) of tasks running successfully (C# vs. other languages)

172
 Performance: tasks running without errors Performance: tasks running without errors

0.2

0.2
F# F#
Go Haskell

0.16

0.16
0.13

0.13
running time (min)

running time (min)

0.1

0.1
0.06

0.06
0.03

0.03
0

0
5 10 15 5 10 15

task task

Performance: tasks running without errors Performance: tasks running without errors

20
F# F#
9

Java Python
5.8

12
3.6

7
running time (min)

running time (min)

2.2

4
1.2

2
0.5

1
0

2 4 6 8 10 12 14 5 10 15

task task

Performance: tasks running without errors

5.7

F#
Ruby
3.9
2.5
running time (min)

1.6
0.9
0.4
0

5 10 15

task

Figure 212: Performance (min) of tasks running successfully (F# vs. other languages)

173
 Performance: tasks running without errors (min) Performance: tasks running without errors (min)

0.20

0.20
0.15

0.15
Haskell
0.10

0.10
Go

0.05

0.05
●
0.00

0.00
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0.00 0.05 0.10 0.15 0.20 0.00 0.05 0.10 0.15 0.20

F# F#

Performance: tasks running without errors (min) Performance: tasks running without errors (min)

20
●
8

15
6

Python
Java

10

●
4

5
2

●
●

●
● ●
●●● ● ●
●
● ● ● ● ● ● ● ● ● ● ●
● ●
0

0 2 4 6 8 0 5 10 15 20

F# F#

Performance: tasks running without errors (min)

5
4
Ruby

●
3
2

● ● ● ● ● ● ● ● ● ● ●
1

●
0

0 1 2 3 4 5

F#

Figure 213: Performance (min) of tasks running successfully (F# vs. other languages)

174
 Performance: tasks running without errors Performance: tasks running without errors

63
Go Go
2

Haskell Java
1.5

31
1.1

15
running time (min)

running time (min)

0.7

7
0.4

3
0.2

1
0

0
0 5 10 15 20 25 0 5 10 15 20 25

task task

Performance: tasks running without errors Performance: tasks running without errors
285

124

Go Go
Python Ruby
110

55
42

24
running time (min)

running time (min)

16

10
6

4
2

1
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Figure 214: Performance (min) of tasks running successfully (Go vs. other languages)

175
 Performance: tasks running without errors (min) Performance: tasks running without errors (min)
2.0

60
50
1.5

40
Haskell

Java
1.0

30
20
0.5

10
●

● ●
0.0

●
●
● ●

0
0.0 0.5 1.0 1.5 2.0 0 10 20 30 40 50 60

Go Go

Performance: tasks running without errors (min) Performance: tasks running without errors (min)
120
250

100
200

80
150
Python

Ruby

60
100

40
50

20

●
●
●●● ● ●●
● ● ● ●
0

0 50 100 150 200 250 0 20 40 60 80 100 120

Go Go

Figure 215: Performance (min) of tasks running successfully (Go vs. other languages)

176
 Performance: tasks running without errors Performance: tasks running without errors

Haskell Haskell
9

4
Java Python
5.8

2.8
3.6

1.9
running time (min)

running time (min)

2.2

1.2
1.2

0.7
0.5

0.3
0

0
5 10 15 20 0 5 10 15 20 25 30

task task

Performance: tasks running without errors

134

Haskell
Ruby
59
25
running time (min)

11
4
1
0

0 5 10 15 20 25 30

task

Figure 216: Performance (min) of tasks running successfully (Haskell vs. other languages)

177
 Performance: tasks running without errors (min) Performance: tasks running without errors (min)

● ●

4
8
●

3
6

Python
Java
● ●

2
4

1
2

●
● ●
● ●
0

0
0 2 4 6 8 0 1 2 3 4

Haskell Haskell

Performance: tasks running without errors (min)

120
100
80
Ruby

60
40
20

●
●
● ●
0

0 20 40 60 80 100 120

Haskell

Figure 217: Performance (min) of tasks running successfully (Haskell vs. other languages)

Performance: tasks running without errors Performance: tasks running without errors
21

27

Java Java
Python Ruby
14

17
11
9
running time (min)

running time (min)

6

6
3

4
2

2
1

0 5 10 15 20 25 0 5 10 15 20 25

task task

Figure 218: Performance (min) of tasks running successfully (Java vs. other languages)

178
 Performance: tasks running without errors (min) Performance: tasks running without errors (min)

20

25
20
15

15
Python

Ruby
10

10
●
●
5

5
●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●●
●●
● ● ●● ●

0
5 10 15 20 0 5 10 15 20 25

Java Java

Figure 219: Performance (min) of tasks running successfully (Java vs. other languages)

Performance: tasks running without errors

16

Python
Ruby
11
7
running time (min)

5
3
2
1

0 10 20 30 40

task

Figure 220: Performance (min) of tasks running successfully (Python vs. other languages)

Performance: tasks running without errors (min)

●
15
10
Ruby

●
5

●
●
●
●
●
●
● ● ●

5 10 15

Python

Figure 221: Performance (min) of tasks running successfully (Python vs. other languages)

179
 Performance: tasks running without errors
0.17

C
C#
F#
Go
0.14

Haskell
Java
Python
Ruby
0.11
running time (min)

0.08
0.05
0.03
0

1 2 3 4 5 6 7

task

Figure 222: Performance (min) of tasks running successfully (all languages)

180
 Performance: tasks running without errors Performance: tasks running without errors

0.12

0.2
C C
C# F#

0.16
0.1
0.08

0.13
running time (mean)

running time (mean)

0.06

0.1
0.04

0.06
0.02

0.03
0

0
2 4 6 8 10 12 14 2 4 6 8 10 12 14

task task

Performance: tasks running without errors Performance: tasks running without errors
0.24

48
C C
Go Haskell
0.2

24
0.15

12
running time (mean)

running time (mean)

0.11

6
0.07

3
0.04

1
0

0 5 10 15 20 25 30 0 5 10 15 20 25

task task

Performance: tasks running without errors Performance: tasks running without errors
2.7

54

C C
Java Python
27
2
1.4

13
running time (mean)

running time (mean)

0.9

6
0.5

3
0.2

1
0

5 10 15 20 0 5 10 15 20 25 30 35

task task

Performance: tasks running without errors

C
3

Ruby
2.2
1.5
running time (mean)

1
0.6
0.3

181
0

0 5 10 15 20 25 30

task

Figure 223: Performance (mean) of tasks running successfully (C vs. other languages)
 Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)

0.12

0.20
● ●

0.10
●

0.15
●

0.08
●

0.06

0.10
C#

F#
●

0.04
●
●

0.05
●
●
0.02 ●
●
●

●
●
●
0.00

0.00
●

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.00 0.05 0.10 0.15 0.20

C C

Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)
0.20

40
0.15

30
Haskell
Go

0.10

20
●
0.05

10
●
0.00

● ● ●
●
0

0.00 0.05 0.10 0.15 0.20 0 10 20 30 40

C C

Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)
2.5

50
2.0

40
1.5

30
Python
Java

1.0

20

●
0.5

10

●
●
●
● ●
0.0

●
0

0.0 0.5 1.0 1.5 2.0 2.5 0 10 20 30 40 50

C C

Performance: tasks running without errors (mean)

3.0
2.5
2.0
Ruby

1.5
1.0

●
0.5

● 182
0.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

Figure 224: Performance (mean) of tasks running successfully (C vs. other languages)
 Performance: tasks running without errors Performance: tasks running without errors

190
7.5
C# C#
F# Go

5.7

79
4.2

32
running time (mean)

running time (mean)

3.1

13
2.2

5
1.5

1
1

0
2 4 6 8 10 5 10 15

task task

Performance: tasks running without errors Performance: tasks running without errors

11
C# C#
2

Haskell Java
1.5

8
1.1

6
running time (mean)

running time (mean)

0.7

4
0.4

3
0.2

2
0

5 10 15 2 4 6 8 10 12 14

task task

Performance: tasks running without errors Performance: tasks running without errors
76

C# C#
8

Python Ruby
41
6
4.5

22
running time (mean)

running time (mean)

3.2

11
2.3

6
1.6

3
1

5 10 15 5 10 15

task task

Figure 225: Performance (mean) of tasks running successfully (C# vs. other languages)

183
 Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)

● ●

150
6
●

5
●

100
Go
F#

●
4

●
●
3

50
2

● ● ●●
●
1

0
1 2 3 4 5 6 7 0 50 100 150

C# C#

Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)
2.0

10
●
1.5

8
Haskell

Java
1.0

●
6

●
●

●
4
0.5

●
●
●
●
●
●
●
2

●
●
0.0

●
●
●●
●●
●
●● ●

0.0 0.5 1.0 1.5 2.0 2 4 6 8 10

C# C#

Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)
8
7

60
6
5
Python

40
Ruby
4
3

20

●
2

● ●
●
● ●
●● ● ● ● ● ● ● ● ●
●
●
●
●
●●
● ●
1

1 2 3 4 5 6 7 8 0 20 40 60

C# C#

Figure 226: Performance (mean) of tasks running successfully (C# vs. other languages)

184
 Performance: tasks running without errors Performance: tasks running without errors

0.2

0.2
F# F#
Go Haskell

0.16

0.16
0.13

0.13
running time (mean)

running time (mean)

0.1

0.1
0.06

0.06
0.03

0.03
0

0
5 10 15 5 10 15

task task

Performance: tasks running without errors Performance: tasks running without errors

20
F# F#
9

Java Python
5.8

12
3.6

7
running time (mean)

running time (mean)

2.2

4
1.2

2
0.5

1
0

2 4 6 8 10 12 14 5 10 15

task task

Performance: tasks running without errors

5.7

F#
Ruby
3.9
2.5
running time (mean)

1.6
0.9
0.4
0

5 10 15

task

Figure 227: Performance (mean) of tasks running successfully (F# vs. other languages)

185
 Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)

0.20

0.20
0.15

0.15
Haskell
0.10

0.10
Go

0.05

0.05
●
0.00

0.00
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0.00 0.05 0.10 0.15 0.20 0.00 0.05 0.10 0.15 0.20

F# F#

Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)

20
●
8

15
6

●
Python
Java

10
4

5
2

●
●

● ●
● ●●
● ● ● ●
● ● ●
●
● ● ● ● ● ● ● ● ●
● ●
0

0 2 4 6 8 0 5 10 15 20

F# F#

Performance: tasks running without errors (mean)

5
4

●
Ruby

3
2

●
●

● ● ● ● ● ● ● ● ● ●
1

●
0

0 1 2 3 4 5

F#

Figure 228: Performance (mean) of tasks running successfully (F# vs. other languages)

186
 Performance: tasks running without errors Performance: tasks running without errors

63
Go Go
2

Haskell Java
1.5

31
1.1

15
running time (mean)

running time (mean)

0.7

7
0.4

3
0.2

1
0

0
0 5 10 15 20 25 0 5 10 15 20 25

task task

Performance: tasks running without errors Performance: tasks running without errors
285

124

Go Go
Python Ruby
110

55
42

24
running time (mean)

running time (mean)

16

10
6

4
2

1
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Figure 229: Performance (mean) of tasks running successfully (Go vs. other languages)

187
 Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)
2.0

60
50
1.5

40
Haskell

Java
1.0

30
20
0.5

10
●

●
● ●
0.0

●
●
● ● ●

0
0.0 0.5 1.0 1.5 2.0 0 10 20 30 40 50 60

Go Go

Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)
120
250

100
200

80
150
Python

Ruby

60
100

40
50

20

●
●●
●● ● ●●
● ● ● ●
0

0 50 100 150 200 250 0 20 40 60 80 100 120

Go Go

Figure 230: Performance (mean) of tasks running successfully (Go vs. other languages)

188
 Performance: tasks running without errors Performance: tasks running without errors

10.4
Haskell Haskell
9

Java Python
5.8

6.6
3.6

4.1
running time (mean)

running time (mean)

2.2

2.4
1.2

1.2
0.5

0.5
0

0
5 10 15 20 0 5 10 15 20 25 30

task task

Performance: tasks running without errors

134

Haskell
Ruby
59
25
running time (mean)

11
4
1
0

0 5 10 15 20 25 30

task

Figure 231: Performance (mean) of tasks running successfully (Haskell vs. other languages)

189
 Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)

● ●

10
8
●

8
6

6
Python
Java
●

4
●

4
2

2
●

●●
●
● ●
●
0

0
0 2 4 6 8 0 2 4 6 8 10

Haskell Haskell

Performance: tasks running without errors (mean)

120
100
80
Ruby

60
40
20

●
●
● ●
0

0 20 40 60 80 100 120

Haskell

Figure 232: Performance (mean) of tasks running successfully (Haskell vs. other languages)

Performance: tasks running without errors Performance: tasks running without errors
21

27

Java Java
Python Ruby
14

17
11
9
running time (mean)

running time (mean)

6

6
3

4
2

2
1

0 5 10 15 20 25 0 5 10 15 20 25

task task

Figure 233: Performance (mean) of tasks running successfully (Java vs. other languages)

190
 Performance: tasks running without errors (mean) Performance: tasks running without errors (mean)

20

25
20
15

15
Python

Ruby
10

10
●
●
5

5
●
● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ●●●
●●●●●●
●●
●●● ●● ●

0
5 10 15 20 0 5 10 15 20 25

Java Java

Figure 234: Performance (mean) of tasks running successfully (Java vs. other languages)

Performance: tasks running without errors

16

Python
Ruby
11
7
running time (mean)

5
3
2
1

0 10 20 30 40

task

Figure 235: Performance (mean) of tasks running successfully (Python vs. other languages)

Performance: tasks running without errors (mean)

●
15
10
Ruby

●
5

● ●● ●
●
●●●
●
●
●
●
●
● ● ●

5 10 15

Python

Figure 236: Performance (mean) of tasks running successfully (Python vs. other languages)

191
 Performance: tasks running without errors
0.17

C
C#
F#
Go
0.14

Haskell
Java
Python
Ruby
0.11
running time (mean)

0.08
0.05
0.03
0

1 2 3 4 5 6 7

task

Figure 237: Performance (mean) of tasks running successfully (all languages)

192
F. Scalability

193
 Scalability: tasks running without errors Scalability: tasks running without errors

2082

2077
C C
C# F#

582

581
162

162
running time (min)

running time (min)

45

45
12

12
3

3
0

0
5 10 15 20 2 4 6 8 10 12

task task

Scalability: tasks running without errors Scalability: tasks running without errors
244

298
C C
Go Haskell

115
97
38

44
running time (min)

running time (min)

15

16
5

6
2

2
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Scalability: tasks running without errors Scalability: tasks running without errors
859

605

C C
Java Python
278

207
89

71
running time (min)

running time (min)

28

24
9

7
2

2
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Scalability: tasks running without errors

285

C
Ruby
110
42
running time (min)

16
6
2

194
0

0 5 10 15 20 25 30

task

Figure 238: Scalability (min) of tasks running successfully (C vs. other languages)
 Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

● ●

2000
2000

1500
1500
C#

F#

1000
1000

500
500

● ●
●
●●
● ●
●●
●
0

0
0 500 1000 1500 2000 0 500 1000 1500 2000

C C

Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)
250

300
● ●

250
200

200
●
●
150

Haskell

150
Go

100

100
●
●
50

●
50
●
●
●
●
●
●
● ●
●

● ●
●
●
● ●
●
●
● ●
●
●
0

0 50 100 150 200 250 0 50 100 150 200 250 300

C C

Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)
600

● ●
800

500
600

400
Python
Java

300
400

●
200

●
200

100

●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
● ●
● ●
●
● ●
●
●●
●
● ●
●●
●
0

0 200 400 600 800 0 100 200 300 400 500 600

C C

Scalability: tasks running without errors (min)

●
250
200

●
150
Ruby

●
100

●
●
●
●
●
●
●
50

●
●
●
●
●
●
●
●
●
● ●
●
● ● ● ●
195
0

0 50 100 150 200 250

Figure 239: Scalability (min) of tasks running successfully (C vs. other languages)
 Scalability: tasks running without errors Scalability: tasks running without errors

297
31
C# C#
F# Go

114
19
12

44
running time (min)

running time (min)

16
7
4

6
2

2
1

0
2 4 6 8 10 5 10 15 20

task task

Scalability: tasks running without errors Scalability: tasks running without errors
1041

859
C# C#
Haskell Java
326

312
102

113
running time (min)

running time (min)

31

40
14
9
2

4
0

5 10 15 5 10 15 20

task task

Scalability: tasks running without errors Scalability: tasks running without errors
1767

1301

C# C#
Python Ruby
570

441
183

149
running time (min)

running time (min)

58

50
18

16
5

5
1

5 10 15 20 5 10 15

task task

Figure 240: Scalability (min) of tasks running successfully (C# vs. other languages)

196
 Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

300
●

30

250
25

200
20

150
Go
F#

15

100
10

●
●

50
5

● ●

●
● ●
●
● ● ● ●
●
●●
●●
●●
●●
●● ●
●● ●

0
0

0 5 10 15 20 25 30 0 50 100 150 200 250 300

C# C#

Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

●
1000

800
800

600
600
Haskell

Java

400
400

●
200

●
200

●
●
●
● ●
●
●
● ● ●
●
●
●● ●
0

0 200 400 600 800 1000 0 200 400 600 800

C# C#

Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

● ●
1200
1500

1000
800
1000
Python

Ruby

600
400
500

200

●
● ●
● ●
●
●
● ● ●
●
● ●
0

0 500 1000 1500 0 200 400 600 800 1000 1200

C# C#

Figure 241: Scalability (min) of tasks running successfully (C# vs. other languages)

197
 Scalability: tasks running without errors Scalability: tasks running without errors

1038
297
F# F#
Go Haskell

128

326
102
55
running time (min)

running time (min)

23

31
10

9
4

2
1

0
2 4 6 8 10 12 14 2 4 6 8 10

task task

Scalability: tasks running without errors Scalability: tasks running without errors
216

163
F# F#
Java Python
98

78
45

37
running time (min)

running time (min)

20

17
9

8
3

3
1

2 4 6 8 10 12 14 2 4 6 8 10 12

task task

Scalability: tasks running without errors

115

F#
Ruby
58
29
running time (min)

14
7
3
1

2 4 6 8 10 12

task

Figure 242: Scalability (min) of tasks running successfully (F# vs. other languages)

198
 Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

300

1000
250

800
200

600
Haskell
150
Go

400
100

200
50

●
●
●
●●●
● ● ● ● ●
●
●
●● ●

0
0

0 50 100 150 200 250 300 0 200 400 600 800 1000

F# F#

Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

●
200

150
150

100
Python
Java

100

50

●
50

●
●
●
● ●
●●●● ● ● ●
● ● ●● ●
0

0 50 100 150 200 0 50 100 150

F# F#

Scalability: tasks running without errors (min)

100
80
Ruby

60

●
40
20

●
●
●
●
●●● ●
0

0 20 40 60 80 100

F#

Figure 243: Scalability (min) of tasks running successfully (F# vs. other languages)

199
 Scalability: tasks running without errors Scalability: tasks running without errors
182

86
Go Go
Haskell Java
76

40
31

19
running time (min)

running time (min)

13

8
5

3
1

1
0

0
0 5 10 15 20 25 30 0 5 10 15 20 25 30 35

task task

Scalability: tasks running without errors Scalability: tasks running without errors
259

168

Go Go
Python Ruby
102

71
40

30
running time (min)

running time (min)

15

12
5

5
2

1
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Figure 244: Scalability (min) of tasks running successfully (Go vs. other languages)

200
 Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

● ●

80
150

60
100
Haskell

Java

40
●

●
50

●
●

20
● ●●
●

● ●
●
●
●
●
● ●
●
● ●
●
●
●
● ●
●
● ●
●
●
●
●
●●● ● ●●
●●
●●
●● ● ●● ●
0

0
0 50 100 150 0 20 40 60 80

Go Go

Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

● ●
250

150

●
200

100
150

●
Python

Ruby

●
100

●
●
●
●
50

● ●
● ●
●
● ●
●
●
50

●
● ●
●
●
● ●
● ●
●
●
●
● ●
●
●
●
● ●
●
●
●●
● ●● ● ●●●●● ● ●
0

0 50 100 150 200 250 0 50 100 150

Go Go

Figure 245: Scalability (min) of tasks running successfully (Go vs. other languages)

201
 Scalability: tasks running without errors Scalability: tasks running without errors
364

444
Haskell Haskell
Java Python
135

160
50

57
running time (min)

running time (min)

18

20
6

7
2

2
0

0
0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Scalability: tasks running without errors

517

Haskell
Ruby
182
63
running time (min)

22
7
2
0

0 5 10 15 20 25 30

task

Figure 246: Scalability (min) of tasks running successfully (Haskell vs. other languages)

202
 Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

400
300

300
200

Python
Java

200
100

100
●
●
●
●
●
● ●
●
●
●●
●●● ●
●●
● ● ● ● ● ●
●●
●● ● ● ● ●
0

0
0 100 200 300 0 100 200 300 400

Haskell Haskell

Scalability: tasks running without errors (min)

500
400
300
Ruby

200
100

●
●
●●
●
●
●●● ● ●
0

0 100 200 300 400 500

Haskell

Figure 247: Scalability (min) of tasks running successfully (Haskell vs. other languages)

Scalability: tasks running without errors Scalability: tasks running without errors
1718

1518

Java Java
Python Ruby
556

502
180

165
running time (min)

running time (min)

58

54
18

17
5

5
1

0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Figure 248: Scalability (min) of tasks running successfully (Java vs. other languages)

203
 Scalability: tasks running without errors (min) Scalability: tasks running without errors (min)

1500
1500

1000
1000
Python

Ruby

500
500

●
●
●
● ●
●
●
● ●
● ●
●
●
●
●
●●● ● ● ● ●
0

0
0 500 1000 1500 0 500 1000 1500

Java Java

Figure 249: Scalability (min) of tasks running successfully (Java vs. other languages)

Scalability: tasks running without errors

380

Python
Ruby
158
65
running time (min)

27
11
4
1

0 5 10 15 20 25 30

task

Figure 250: Scalability (min) of tasks running successfully (Python vs. other languages)

Scalability: tasks running without errors (min)

●
300
200
Ruby

100

●
●
●
●
●
●● ●
0

0 100 200 300

Python

Figure 251: Scalability (min) of tasks running successfully (Python vs. other languages)

204
 Scalability: tasks running without errors
2082

C
C#
F#
Go
582

Haskell
Java
Python
Ruby
162
running time (min)

45
12
3
0

1 2 3 4 5 6 7

task

Figure 252: Scalability (min) of tasks running successfully (all languages)

205
 Scalability: tasks running without errors Scalability: tasks running without errors

2082

2077
C C
C# F#

582

581
162

162
running time (mean)

running time (mean)

45

45
12

12
3

3
0

0
5 10 15 20 2 4 6 8 10 12

task task

Scalability: tasks running without errors Scalability: tasks running without errors
244

367
C C
Go Haskell

136
97
38

50
running time (mean)

running time (mean)

15

18
5

6
2

2
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Scalability: tasks running without errors Scalability: tasks running without errors
2742
859

C C
Java Python
278

732
195
89
running time (mean)

running time (mean)

28

51
13
9
2

3
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Scalability: tasks running without errors

403

C
Ruby
147
54
running time (mean)

19
6
2

206
0

0 5 10 15 20 25 30

task

Figure 253: Scalability (mean) of tasks running successfully (C vs. other languages)
 Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

● ●

2000
2000

1500
1500
C#

F#

1000
1000

500
500

● ●
●
●
●
●●
●● ●
●●
●
0

0
0 500 1000 1500 2000 0 500 1000 1500 2000

C C

Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)
250

● ●
200

300
●
150

200
Haskell
●
●
Go

100

● ●

100
●
50

●
●
●
●
●●
●
●
● ●
●
● ●
●
●
● ●
●
●
● ●
●
●●
●
●
● ●
●
●
0

0 50 100 150 200 250 0 100 200 300

C C

Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

● ●
800

2500
2000
600

1500
Python
Java

400

●
1000
200

●
500

●
●
●
● ●
●
● ●
●
● ●
● ●
●
●
● ●
●
●
●●
●
● ●
●
0

0 200 400 600 800 0 500 1000 1500 2000 2500

C C

Scalability: tasks running without errors (mean)

400

●
300

●
Ruby

●
200

●
●
100

●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ● ● ●
207
0

0 100 200 300 400

Figure 254: Scalability (mean) of tasks running successfully (C vs. other languages)
 Scalability: tasks running without errors Scalability: tasks running without errors

297
31
C# C#
F# Go

114
19
12

44
running time (mean)

running time (mean)

16
7
4

6
2

2
1

0
2 4 6 8 10 5 10 15 20

task task

Scalability: tasks running without errors Scalability: tasks running without errors
1041

859
C# C#
Haskell Java
326

312
102

113
running time (mean)

running time (mean)

31

40
14
9
2

4
0

5 10 15 5 10 15 20

task task

Scalability: tasks running without errors Scalability: tasks running without errors
2742

1301

C# C#
Python Ruby
822

441
246

149
running time (mean)

running time (mean)

73

50
21

16
6

5
1

5 10 15 20 5 10 15

task task

Figure 255: Scalability (mean) of tasks running successfully (C# vs. other languages)

208
 Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

300
●

30

250
25

200
20

150
Go
F#

15

100
10

●
●
●

50
5

●
● ●● ●
●
●
● ● ● ●
●
●●
● ●● ●
●● ● ●

0
0

0 5 10 15 20 25 30 0 50 100 150 200 250 300

C# C#

Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

●
1000

800
800

600
600
Haskell

Java

400

●
400

●
●
200
200

●
●
●
● ●
●●
● ● ●
●
●
●
●● ●
0

0 200 400 600 800 1000 0 200 400 600 800

C# C#

Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

● ●
1200
2500

1000
2000

●
800
1500
Python

Ruby

600
1000

400

●
500

200

●
●
●
●
● ●
●
●
●● ●
● ●
0

0 500 1000 1500 2000 2500 0 200 400 600 800 1000 1200

C# C#

Figure 256: Scalability (mean) of tasks running successfully (C# vs. other languages)

209
 Scalability: tasks running without errors Scalability: tasks running without errors

1038
297
F# F#
Go Haskell

128

326
102
55
running time (mean)

running time (mean)

23

31
10

9
4

2
1

0
2 4 6 8 10 12 14 2 4 6 8 10

task task

Scalability: tasks running without errors Scalability: tasks running without errors
216

163
F# F#
Java Python
98

78
45

37
running time (mean)

running time (mean)

20

17
9

8
3

3
1

2 4 6 8 10 12 14 2 4 6 8 10 12

task task

Scalability: tasks running without errors

403

F#
Ruby
166
68
running time (mean)

27
11
4
1

2 4 6 8 10 12

task

Figure 257: Scalability (mean) of tasks running successfully (F# vs. other languages)

210
 Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

300

1000
250

800
200

600
Haskell
150
Go

400
100

200
50

●
●
●
●
●●●
● ● ● ●
●
●
● ●

0
0

0 50 100 150 200 250 300 0 200 400 600 800 1000

F# F#

Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

●
200

150
150

100
Python
Java

100

50

●
50

●
●
● ● ● ●
● ●
●
●●●● ● ● ● ●
0

0 50 100 150 200 0 50 100 150

F# F#

Scalability: tasks running without errors (mean)

400

●
300
Ruby

200
100

●
●
●
●● ●
0

0 100 200 300 400

F#

Figure 258: Scalability (mean) of tasks running successfully (F# vs. other languages)

211
 Scalability: tasks running without errors Scalability: tasks running without errors
491

93
Go Go
Haskell Java
174

43
61

20
running time (mean)

running time (mean)

21

9
7

4
2

1
0

0
0 5 10 15 20 25 30 0 5 10 15 20 25 30 35

task task

Scalability: tasks running without errors Scalability: tasks running without errors
259

403

Go Go
Python Ruby
102

147
40

54
running time (mean)

running time (mean)

15

19
5

6
2

2
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Figure 259: Scalability (mean) of tasks running successfully (Go vs. other languages)

212
 Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)
500

● ●

80
400

60
300
Haskell

Java

40
200

●
●

20
100

●●
●
●
●
●●
● ●
●
●●
● ●
●
●
●
●
● ●
●●
●
●
● ●●● ● ●
●●●
●
●
●
● ●●●●● ●● ●
0

0
0 100 200 300 400 500 0 20 40 60 80

Go Go

Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)
400

● ●
250
200

300

●
150
Python

Ruby

●
200

●
●
●
100

●
●
●
●
100

●
●
50

●
●
● ●
●
●
●
●● ●
●
●
● ●
●●
●
●
● ●
●
●
● ●
● ●●
●
● ●
●●
● ●● ● ●
●
●● ●
●●●●
0

0 50 100 150 200 250 0 100 200 300 400

Go Go

Figure 260: Scalability (mean) of tasks running successfully (Go vs. other languages)

213
 Scalability: tasks running without errors Scalability: tasks running without errors

2742
491

Haskell Haskell
Java Python
174

732
195
61
running time (mean)

running time (mean)

21

51
13
7
2

3
0

0
0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Scalability: tasks running without errors

517

Haskell
Ruby
182
63
running time (mean)

22
7
2
0

0 5 10 15 20 25 30

task

Figure 261: Scalability (mean) of tasks running successfully (Haskell vs. other languages)

214
 Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

500
●

2500
400

2000
300

1500
Python
Java

200

1000
100

500
●
● ●
● ●
●
●
●●●
●●●●
●● ● ●● ● ●●●
●
● ● ●
0

0
0 100 200 300 400 500 0 500 1000 1500 2000 2500

Haskell Haskell

Scalability: tasks running without errors (mean)

500
400
300
Ruby

200
100

●
●
●
●
●● ●
●●
●
●
●
● ● ● ●
0

0 100 200 300 400 500

Haskell

Figure 262: Scalability (mean) of tasks running successfully (Haskell vs. other languages)

Scalability: tasks running without errors Scalability: tasks running without errors
2742

1518

Java Java
Python Ruby
822

502
246

165
running time (mean)

running time (mean)

73

54
21

17
6

5
1

0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Figure 263: Scalability (mean) of tasks running successfully (Java vs. other languages)

215
 Scalability: tasks running without errors (mean) Scalability: tasks running without errors (mean)

1500
●

2500
2000

1000
1500
Python

Ruby
1000

500
500 ●

● ●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●● ●● ● ●
0

0
0 500 1000 1500 2000 2500 0 500 1000 1500

Java Java

Figure 264: Scalability (mean) of tasks running successfully (Java vs. other languages)

Scalability: tasks running without errors

2742

Python
Ruby
822
246
running time (mean)

73
21
6
1

0 5 10 15 20 25 30

task

Figure 265: Scalability (mean) of tasks running successfully (Python vs. other languages)

Scalability: tasks running without errors (mean)

2500
2000
1500
Ruby

1000
500

●
●
●

●
●
● ●
0

0 500 1000 1500 2000 2500

Python

Figure 266: Scalability (mean) of tasks running successfully (Python vs. other languages)

216
 Scalability: tasks running without errors
2082

C
C#
F#
Go
582

Haskell
Java
Python
Ruby
162
running time (mean)

45
12
3
0

1 2 3 4 5 6 7

task

Figure 267: Scalability (mean) of tasks running successfully (all languages)

217
G. Maximum RAM

218
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

239
20
C C
C# F#

107
13

48
9
max RAM used (min)

max RAM used (min)

21
5
3

9
2

3
1

1
5 10 15 20 2 4 6 8 10 12

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

1998
70

C C
Go Haskell

631
38

199
21
max RAM used (min)

max RAM used (min)

11

62
19
6
3

5
1

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
414

788

C C
Java Python
170

290
107
69
max RAM used (min)

max RAM used (min)

28

39
11

14
4

4
1

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Maximum RAM usage: tasks running without errors

1216

C
Ruby
417
143
max RAM used (min)

48
16
5

219
1

0 5 10 15 20 25 30

task

Figure 268: Maximum RAM usage (min) of tasks running successfully (C vs. other languages)
 Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

20
● ●

●
●

200
15
●
●
●

150
●

C#

F#
10
●
●

100
●
●
●
● ●

50
5 ●

●
●
●
●
● ●
●
●
● ●
● ● ●

0
5 10 15 20 0 50 100 150 200

C C

Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

2000
70

●
60

1500
50
40

Haskell

1000
Go

30
20

500
10

●
●
●
●
●
● ●
●
●
●
● ●
●
● ●●
●● ● ● ●
0
0

0 10 20 30 40 50 60 70 0 500 1000 1500 2000

C C

Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)
800

● ●
400

600
300

Python

400
Java

●
200

●
●
200
100

●
●
●
●
●
●
● ●
●
●
● ●
●● ●
●
● ●●
0
0

0 100 200 300 400 0 200 400 600 800

C C

Maximum RAM usage: tasks running without errors (min)

1200

●
1000
800
Ruby

600
400
200

●
●

●
●
●●
●
220
0

0 200 400 600 800 1000 1200

Figure 269: Maximum RAM usage (min) of tasks running successfully (C vs. other languages)
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

353
14
C# C#
F# Go

148
10

62
7
max RAM used (min)

max RAM used (min)

26
4

10
3
2

4
1

1
2 4 6 8 10 5 10 15 20

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
24

12
C# C#
Haskell Java
15

9
10

6
max RAM used (min)

max RAM used (min)

6

4
4

3
2

2
1

5 10 15 5 10 15 20

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
46

71

C# C#
Python Ruby
27

38
15

21
max RAM used (min)

max RAM used (min)

11
9
5

6
2

3
1

5 10 15 20 5 10 15

task task

Figure 270: Maximum RAM usage (min) of tasks running successfully (C# vs. other languages)

221
 Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

14

350
●

300
12

250
10
●

200
8

Go
F#

150
6

100
4

50
●
●
●
2

●
●
● ● ●
●●●
●●
●●●
●●●● ●

0
2 4 6 8 10 12 14 0 50 100 150 200 250 300 350

C# C#

Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

● ●

12
20

10
●

8
15
Haskell

Java

●
6

●
●
10

●
●
4

●
●

●
●
5

●
●
●
● ●
●
●
2

●
●
●
●
●
●●●● ●
●●
●● ●
● ● ●

5 10 15 20 2 4 6 8 10 12

C# C#

Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

● ●
70
40

60
50
30

40
Python

Ruby
20

30
20
10

●
●
● ●
10

● ●

● ●
●
● ●
●
●
●●●● ● ●● ●
0
0

0 10 20 30 40 0 10 20 30 40 50 60 70

C# C#

Figure 271: Maximum RAM usage (min) of tasks running successfully (C# vs. other languages)

222
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

123

72
F# F#
Go Haskell

61

39
30

21
max RAM used (min)

max RAM used (min)

15

11
7

6
3

3
1

1
2 4 6 8 10 12 14 2 4 6 8 10

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

110
3.7

F# F#
Java Python
3.1

56
2.5

28
max RAM used (min)

max RAM used (min)

2.1

14
1.7

7
1.3

3
1

2 4 6 8 10 12 14 2 4 6 8 10 12

task task

Maximum RAM usage: tasks running without errors

5.3

F#
Ruby
4.2
3.3
max RAM used (min)

2.5
1.9
1.4
1

2 4 6 8 10 12

task

Figure 272: Maximum RAM usage (min) of tasks running successfully (F# vs. other languages)

223
 Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

120

70
60
100

50
80

40
Haskell
Go

60

30
40

20
20

10
●

●
●
● ●●● ●● ● ● ● ● ● ●
●●●●
● ● ● ●
0

0
0 20 40 60 80 100 120 0 10 20 30 40 50 60 70

F# F#

Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

●
3.5

100
3.0

80
●
2.5

60
Python

●
Java

●
2.0

40

●
●

●
1.5

●
20

●
1.0

● ● ● ● ● ●
●
●
●●
●●●●
● ●
0

1.0 1.5 2.0 2.5 3.0 3.5 0 20 40 60 80 100

F# F#

Maximum RAM usage: tasks running without errors (min)

●
5

●
4
Ruby

●
2

●
●● ● ● ● ● ●
1

1 2 3 4 5

F#

Figure 273: Maximum RAM usage (min) of tasks running successfully (F# vs. other languages)

224
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
871

653
Go Go
Haskell Java
316

248
114

94
max RAM used (min)

max RAM used (min)

41

35
14

13
5

4
1

1
0 5 10 15 20 25 30 0 5 10 15 20 25 30 35

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
381

587

Go Go
Python Ruby
158

227
65

87
max RAM used (min)

max RAM used (min)

27

33
11

12
4

4
1

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Figure 274: Maximum RAM usage (min) of tasks running successfully (Go vs. other languages)

225
 Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

● ●

600
800

500
600

400
Haskell

Java
400

300
200
●
200

100
●
●

● ●
● ●
●
● ●
●
● ●
●
● ● ●
●●
0

0
0 200 400 600 800 0 100 200 300 400 500 600

Go Go

Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)
600

● ●
500
300

●
400
Python

200

Ruby

300
200

●
100

●
●
100

●
●

● ●
●
● ●
●
●
●
●
●● ●
● ●
0
0

0 100 200 300 0 100 200 300 400 500 600

Go Go

Figure 275: Maximum RAM usage (min) of tasks running successfully (Go vs. other languages)

226
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

138
64

Haskell Haskell
Java Python
35

68
19

33
max RAM used (min)

max RAM used (min)

10

16
5

7
3

3
1

1
0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Maximum RAM usage: tasks running without errors

213

Haskell
Ruby
97
44
max RAM used (min)

20
8
3
1

0 5 10 15 20 25 30

task

Figure 276: Maximum RAM usage (min) of tasks running successfully (Haskell vs. other languages)

227
 Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

140
● ●

60

120
50

100
40

80
Python
Java

30

60
●
20

40
●

●
●
10

20
●
●
●
● ●
●
● ●
●
●
● ●
●● ● ●
● ● ● ●
●
●●
●●●
● ●● ● ● ●

0
0

0 10 20 30 40 50 60 0 20 40 60 80 100 120 140

Haskell Haskell

Maximum RAM usage: tasks running without errors (min)

●
200
150

●
Ruby

100

●
50

●
●

●
●
●
●
● ●●●● ●
0

0 50 100 150 200

Haskell

Figure 277: Maximum RAM usage (min) of tasks running successfully (Haskell vs. other languages)

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
60

33

Java Java
Python Ruby
34

20
19

12
max RAM used (min)

max RAM used (min)

10

7
5

4
3

2
1

0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Figure 278: Maximum RAM usage (min) of tasks running successfully (Java vs. other languages)

228
 Maximum RAM usage: tasks running without errors (min) Maximum RAM usage: tasks running without errors (min)

60
●

30
50

25
40

20
Python

Ruby
30

15
●

20

10
10

5
● ●
●
●
●
●
● ●
●●
●●
●●●
●
●●● ●●●
● ● ● ● ● ●●
●●●●●● ● ● ● ● ●
0

0
0 10 20 30 40 50 60 0 5 10 15 20 25 30

Java Java

Figure 279: Maximum RAM usage (min) of tasks running successfully (Java vs. other languages)

Maximum RAM usage: tasks running without errors

158

Python
Ruby
76
36
max RAM used (min)

17
8
3
1

0 5 10 15 20 25 30

task

Figure 280: Maximum RAM usage (min) of tasks running successfully (Python vs. other languages)

Maximum RAM usage: tasks running without errors (min)

150
100
Ruby

50

●
●
●
●
●
●
●
●●
●● ●
0

0 50 100 150

Python

Figure 281: Maximum RAM usage (min) of tasks running successfully (Python vs. other languages)

229
 Maximum RAM usage: tasks running without errors
128

C
C#
F#
Go
63

Haskell
Java
Python
Ruby
31
max RAM used (min)

15
7
3
1

1 2 3 4 5 6 7

task

Figure 282: Maximum RAM usage (min) of tasks running successfully (all languages)

230
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

239
20
C C
C# F#

107
13

48
max RAM used (mean)

max RAM used (mean)

9

21
5
3

9
2

3
1

1
5 10 15 20 2 4 6 8 10 12

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

1998
70

C C
Go Haskell

631
38

199
21
max RAM used (mean)

max RAM used (mean)

11

62
19
6
3

5
1

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
414

914

C C
Java Python
170

329
118
69
max RAM used (mean)

max RAM used (mean)

28

42
11

14
4

5
1

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Maximum RAM usage: tasks running without errors

1216

C
Ruby
417
143
max RAM used (mean)

48
16
5

231
1

0 5 10 15 20 25 30

task

Figure 283: Maximum RAM usage (mean) of tasks running successfully (C vs. other languages)
 Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

20
● ●

●
●

200
15
●
●
●

●
●

150
●

C#

F#
10
●
●

100
●
●
●
● ●

50
5 ●

●
●
●
●
● ●
●
●
● ● ●
●

0
5 10 15 20 0 50 100 150 200

C C

Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

2000
70

●
●

●
60

1500
50
40

Haskell

1000
Go

30
20

500
●
●
10

●
●
●
●
●
● ● ●
●
●
●
● ●
●
●
● ● ● ●●
●
0
0

0 10 20 30 40 50 60 70 0 500 1000 1500 2000

C C

Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

● ●
400

800
300

600
Python
Java

●
200

400

●
●
100

200

● ●

●
●
●
●
●
●
●
●
● ●
●
●● ●●
●● ●
●
●
0
0

0 100 200 300 400 0 200 400 600 800

C C

Maximum RAM usage: tasks running without errors (mean)

1200

●
1000
800
Ruby

600

●
400
200

●
●

●
●
●
●
●
232
0

0 200 400 600 800 1000 1200

Figure 284: Maximum RAM usage (mean) of tasks running successfully (C vs. other languages)
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

353
14
C# C#
F# Go

148
10

62
max RAM used (mean)

max RAM used (mean)

7

26
4

10
3
2

4
1

1
2 4 6 8 10 5 10 15 20

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
284

26
C# C#
Haskell Java
124

17
54

10
max RAM used (mean)

max RAM used (mean)

23

6
9

4
4

2
1

5 10 15 5 10 15 20

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
53

71

C# C#
Python Ruby
30

38
17

21
max RAM used (mean)

max RAM used (mean)

11
9
5

6
2

3
1

5 10 15 20 5 10 15

task task

Figure 285: Maximum RAM usage (mean) of tasks running successfully (C# vs. other languages)

233
 Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

14

350
●

300
12

250
10
●

200
8

Go
F#

150
6

100
4

50
●
●
●
2

● ●
●
● ● ●
●●
●●●
●●●
●●●● ●

0
2 4 6 8 10 12 14 0 50 100 150 200 250 300 350

C# C#

Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

25
250

20
200

●
15
Haskell

150

Java

●
100

10

● ●
●
50

●
●
5

●
●
●
● ●
●
●
● ●
●
●
●
●
● ●
●
●
●●
●
0

0 50 100 150 200 250 0 5 10 15 20 25

C# C#

Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

● ●
70
50

60
40

●
50
40
30
Python

Ruby

●
30
20

20

●
10

●
●
10

●
●
●
●

● ●
● ●
●●
●
● ● ● ●
●● ●
0
0

0 10 20 30 40 50 0 10 20 30 40 50 60 70

C# C#

Figure 286: Maximum RAM usage (mean) of tasks running successfully (C# vs. other languages)

234
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

123

72
F# F#
Go Haskell

61

39
30

21
max RAM used (mean)

max RAM used (mean)

15

11
7

6
3

3
1

1
2 4 6 8 10 12 14 2 4 6 8 10

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors

110
4.8

F# F#
Java Python
3.8

56
3.1

28
max RAM used (mean)

max RAM used (mean)

2.4

14
1.8

7
1.4

3
1

2 4 6 8 10 12 14 2 4 6 8 10 12

task task

Maximum RAM usage: tasks running without errors

63

F#
Ruby
35
19
max RAM used (mean)

10
5
3
1

2 4 6 8 10 12

task

Figure 287: Maximum RAM usage (mean) of tasks running successfully (F# vs. other languages)

235
 Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

120

70
60
100

50
80

40
Haskell
Go

60

30
40

20
20

10
●

●
●
● ●●● ●● ● ● ● ● ● ● ●●
●●● ● ● ●
0

0
0 20 40 60 80 100 120 0 10 20 30 40 50 60 70

F# F#

Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

100
4

80
60
Python
3
Java

●
40

●
2

● ●
20

●
●
● ●

● ● ●
● ● ● ● ● ●
●●
●●●●●
1

1 2 3 4 0 20 40 60 80 100

F# F#

Maximum RAM usage: tasks running without errors (mean)

●
60
50
40
Ruby

30
20
10

●
●
●
●
●●● ●●
0

0 10 20 30 40 50 60

F#

Figure 288: Maximum RAM usage (mean) of tasks running successfully (F# vs. other languages)

236
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
1589

653
Go Go
Haskell Java
521

248
171

94
max RAM used (mean)

max RAM used (mean)

55

35
18

13
5

4
1

1
0 5 10 15 20 25 30 0 5 10 15 20 25 30 35

task task

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
441

587

Go Go
Python Ruby
179

227
72

87
max RAM used (mean)

max RAM used (mean)

29

33
11

12
4

4
1

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Figure 289: Maximum RAM usage (mean) of tasks running successfully (Go vs. other languages)

237
 Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

● ●
1500

600
500
1000

400
●
Haskell

Java

300
500

200
●

100
●
●

●
●
●
●
● ●
● ● ●
●
● ● ●
● ●●
0

0
0 500 1000 1500 0 100 200 300 400 500 600

Go Go

Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)
600

● ●
400

500
400
300

●
Python

Ruby

300
200

200

●
●
●
100

●
●
100

●
● ●

●
● ●
●●
● ●
●
●
●
● ●
●●
●
0
0

0 100 200 300 400 0 100 200 300 400 500 600

Go Go

Figure 290: Maximum RAM usage (mean) of tasks running successfully (Go vs. other languages)

238
 Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
159

168
Haskell Haskell
Java Python
76

80
36

37
max RAM used (mean)

max RAM used (mean)

17

17
8

8
3

3
1

1
0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Maximum RAM usage: tasks running without errors

213

Haskell
Ruby
97
44
max RAM used (mean)

20
8
3
1

0 5 10 15 20 25 30

task

Figure 291: Maximum RAM usage (mean) of tasks running successfully (Haskell vs. other languages)

239
 Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

150
●

150
●

100

100
●

Python
Java
●
50

50
●

●
● ● ●
●
●
● ●
●
●
●
●
●
● ● ●
●
●
●
●
● ● ●● ● ●
●
●●●
●● ●● ● ● ● ●

0
0

0 50 100 150 0 50 100 150

Haskell Haskell

Maximum RAM usage: tasks running without errors (mean)

●
200

●
150

●
Ruby

100

●
50

●
●

●
● ●
●
●
● ●●●● ●
0

0 50 100 150 200

Haskell

Figure 292: Maximum RAM usage (mean) of tasks running successfully (Haskell vs. other languages)

Maximum RAM usage: tasks running without errors Maximum RAM usage: tasks running without errors
60

63

Java Java
Python Ruby
34

35
19

19
max RAM used (mean)

max RAM used (mean)

10

10
5

5
3

3
1

0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Figure 293: Maximum RAM usage (mean) of tasks running successfully (Java vs. other languages)

240
 Maximum RAM usage: tasks running without errors (mean) Maximum RAM usage: tasks running without errors (mean)

60
● ●

60
50

50
40

40
Python

Ruby
30

30
●

20

20
●
●
10

10
●

●
●
● ● ●
●
●●
●●
●●
●●
●●● ●
● ● ●
● ● ● ● ●
●
● ●
●●
●
● ●● ● ● ●

0
0

0 10 20 30 40 50 60 0 10 20 30 40 50 60

Java Java

Figure 294: Maximum RAM usage (mean) of tasks running successfully (Java vs. other languages)

Maximum RAM usage: tasks running without errors

158

Python
Ruby
76
36
max RAM used (mean)

17
8
3
1

0 5 10 15 20 25 30

task

Figure 295: Maximum RAM usage (mean) of tasks running successfully (Python vs. other languages)

Maximum RAM usage: tasks running without errors (mean)

150
100
Ruby

● ●
50

●
●
● ●
●
●
●●
●
●●
●● ●
0

0 50 100 150

Python

Figure 296: Maximum RAM usage (mean) of tasks running successfully (Python vs. other languages)

241
 Maximum RAM usage: tasks running without errors
128

C
C#
F#
Go
63

Haskell
Java
Python
Ruby
31
max RAM used (mean)

15
7
3
1

1 2 3 4 5 6 7

task

Figure 297: Maximum RAM usage (mean) of tasks running successfully (all languages)

242
H. Page faults

243
 Page faults: tasks running without errors Page faults: tasks running without errors

44
C C
C# F#

23
12
# page faults (min)

# page faults (min)

0

6
3
1
0
5 10 15 20 2 4 6 8 10 12

task task

Page faults: tasks running without errors Page faults: tasks running without errors

11
C C
Go Haskell

6.9
4.2
# page faults (min)

# page faults (min)

2.5
0

1.3
0.5
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Page faults: tasks running without errors Page faults: tasks running without errors

C C
Java Python
# page faults (min)

# page faults (min)

0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Page faults: tasks running without errors

C
Ruby
# page faults (min)

244
0 5 10 15 20 25 30

task

Figure 298: Page faults (min) of tasks running successfully (C vs. other languages)
 Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)

1.0
●

40
0.5

30
0.0
C#

F#
●

20
−0.5
●

10
●
−1.0

●
●

0
−1.0 −0.5 0.0 0.5 1.0 0 10 20 30 40

C C

Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)
1.0

10
●
0.5

8
●

Haskell
●

6
0.0
Go

4
−0.5

2
−1.0

−1.0 −0.5 0.0 0.5 1.0 0 2 4 6 8 10

C C

Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)
1.0

1.0
0.5

0.5
Python
Java

0.0

0.0

● ●
−0.5

−0.5
−1.0

−1.0

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

C C

Page faults: tasks running without errors (min)

1.0
0.5
Ruby

0.0

●
−0.5

245
−1.0

−1.0 −0.5 0.0 0.5 1.0

Figure 299: Page faults (min) of tasks running successfully (C vs. other languages)
 Page faults: tasks running without errors Page faults: tasks running without errors

44
C# C#
F# Go

23
12
# page faults (min)

# page faults (min)

6

0
3
1
0

2 4 6 8 10 5 10 15 20

task task

Page faults: tasks running without errors Page faults: tasks running without errors
10

C# C#
Haskell Java
6.4
3.9
# page faults (min)

# page faults (min)

2.3

0
1.2
0.5
0

5 10 15 5 10 15 20

task task

Page faults: tasks running without errors Page faults: tasks running without errors

C# C#
Python Ruby
# page faults (min)

# page faults (min)

0

5 10 15 20 5 10 15

task task

Figure 300: Page faults (min) of tasks running successfully (C# vs. other languages)

246
 Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)

1.0
●

40

0.5
30

0.0
Go
F#

●
20

−0.5
●
10

−1.0
●
●
0

0 10 20 30 40 −1.0 −0.5 0.0 0.5 1.0

C# C#

Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)

1.0
10

●
8

0.5
●
6
Haskell

Java

0.0
● ●

●
4

−0.5
2

−1.0
0

0 2 4 6 8 10 −1.0 −0.5 0.0 0.5 1.0

C# C#

Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)
1.0

1.0
0.5

0.5
Python

Ruby
0.0

0.0

● ●
−0.5

−0.5
−1.0

−1.0

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

C# C#

Figure 301: Page faults (min) of tasks running successfully (C# vs. other languages)

247
 Page faults: tasks running without errors Page faults: tasks running without errors

44

11
F# F#
Go Haskell

6.9
23

4.2
12
# page faults (min)

# page faults (min)

2.5
6

1.3
3

0.5
1
0

0
2 4 6 8 10 12 14 2 4 6 8 10

task task

Page faults: tasks running without errors Page faults: tasks running without errors
44

44
F# F#
Java Python
23

23
12

12
# page faults (min)

# page faults (min)

6

6
3

3
1

1
0

2 4 6 8 10 12 14 2 4 6 8 10 12

task task

Page faults: tasks running without errors

44

F#
Ruby
23
12
# page faults (min)

6
3
1
0

2 4 6 8 10 12

task

Figure 302: Page faults (min) of tasks running successfully (F# vs. other languages)

248
 Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)

40

10
●

8
30

Haskell

6
Go

20

4
●
10

2
● ●

●●● ● ●
0

0
0 10 20 30 40 0 2 4 6 8 10

F# F#

Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)
40

40
30

30
Python
Java

20

20
10

10

●●● ● ● ●●● ● ●
0

0 10 20 30 40 0 10 20 30 40

F# F#

Page faults: tasks running without errors (min)

40
30
Ruby

20
10

●●● ● ●
0

0 10 20 30 40

F#

Figure 303: Page faults (min) of tasks running successfully (F# vs. other languages)

249
 Page faults: tasks running without errors Page faults: tasks running without errors
11

Go Go
Haskell Java
6.9
4.2
# page faults (min)

# page faults (min)

2.5

0
1.3
0.5
0

0 5 10 15 20 25 30 0 5 10 15 20 25 30 35

task task

Page faults: tasks running without errors Page faults: tasks running without errors

Go Go
Python Ruby
# page faults (min)

# page faults (min)

0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Figure 304: Page faults (min) of tasks running successfully (Go vs. other languages)

250
 Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)

1.0
●
10

0.5
●
8

●
Haskell

●
6

Java

0.0
●

●
4

−0.5
2

−1.0
0

0 2 4 6 8 10 −1.0 −0.5 0.0 0.5 1.0

Go Go

Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)
1.0

1.0
0.5

0.5
Python

Ruby
0.0

0.0

● ●
−0.5

−0.5
−1.0

−1.0

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

Go Go

Figure 305: Page faults (min) of tasks running successfully (Go vs. other languages)

251
 Page faults: tasks running without errors Page faults: tasks running without errors
11

11
Haskell Haskell
Java Python
6.9

6.9
4.2

4.2
# page faults (min)

# page faults (min)

2.5

2.5
1.3

1.3
0.5

0.5
0

0
0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Page faults: tasks running without errors

11

Haskell
Ruby
6.9
4.2
# page faults (min)

2.5
1.3
0.5
0

0 5 10 15 20 25 30

task

Figure 306: Page faults (min) of tasks running successfully (Haskell vs. other languages)

252
 Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)

10

10
8

8
6

6
Python
Java

4
2

2
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
0

0
0 2 4 6 8 10 0 2 4 6 8 10

Haskell Haskell

Page faults: tasks running without errors (min)

10
8
6
Ruby

4
2

● ● ● ● ● ●
0

0 2 4 6 8 10

Haskell

Figure 307: Page faults (min) of tasks running successfully (Haskell vs. other languages)

Page faults: tasks running without errors Page faults: tasks running without errors

Java Java
Python Ruby
# page faults (min)

# page faults (min)

0

0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Figure 308: Page faults (min) of tasks running successfully (Java vs. other languages)

253
 Page faults: tasks running without errors (min) Page faults: tasks running without errors (min)

1.0

1.0
0.5

0.5
Python

Ruby
0.0

0.0
● ●

−0.5

−0.5
−1.0

−1.0
−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

Java Java

Figure 309: Page faults (min) of tasks running successfully (Java vs. other languages)

Page faults: tasks running without errors

Python
Ruby
# page faults (min)

0 5 10 15 20 25 30

task

Figure 310: Page faults (min) of tasks running successfully (Python vs. other languages)

Page faults: tasks running without errors (min)

1.0
0.5
Ruby

0.0

●
−0.5
−1.0

−1.0 −0.5 0.0 0.5 1.0

Python

Figure 311: Page faults (min) of tasks running successfully (Python vs. other languages)

254
 Page faults: tasks running without errors
44

C
C#
F#
Go
23

Haskell
Java
Python
Ruby
12
# page faults (min)

6
3
1
0

1 2 3 4 5 6 7

task

Figure 312: Page faults (min) of tasks running successfully (all languages)

255
 Page faults: tasks running without errors Page faults: tasks running without errors

44
C C

1
C# F#

0.8

23
0.6

12
# page faults (mean)

# page faults (mean)

0.4

6
0.3

3
0.1

1
0

0
5 10 15 20 2 4 6 8 10 12

task task

Page faults: tasks running without errors Page faults: tasks running without errors

1410
C C
1

Go Haskell

420
0.8

125
0.6
# page faults (mean)

# page faults (mean)

0.4

37
0.3

10
0.1

2
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Page faults: tasks running without errors Page faults: tasks running without errors

C C
1

Java Python
0.8

4.1
0.6

2.7
# page faults (mean)

# page faults (mean)

0.4

1.6
0.3

0.9
0.1

0.4
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Page faults: tasks running without errors

C
1

Ruby
0.8
0.6
# page faults (mean)

0.4
0.3
0.1

256
0

0 5 10 15 20 25 30

task

Figure 313: Page faults (mean) of tasks running successfully (C vs. other languages)
 Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)

1.0
●

40
0.8

30
0.6
C#

F#

20
0.4
● ●

10
0.2

●
●
0.0

● ●

0
0.0 0.2 0.4 0.6 0.8 1.0 0 10 20 30 40

C C

Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)

1400
1.0

1200
0.8

1000
0.6

800
Haskell
Go

600
0.4

400
0.2

200
0.0

● ● ●
●
0

0.0 0.2 0.4 0.6 0.8 1.0 0 200 400 600 800 1000 1200 1400

C C

Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)
1.0

●
6
5
0.8

4
0.6

Python
Java

3
0.4

2
0.2

●
0.0

● ● ● ●
0

0.0 0.2 0.4 0.6 0.8 1.0 0 1 2 3 4 5 6

C C

Page faults: tasks running without errors (mean)

1.0
0.8
0.6
Ruby

0.4
0.2

257
0.0

● ●

0.0 0.2 0.4 0.6 0.8 1.0

Figure 314: Page faults (mean) of tasks running successfully (C vs. other languages)
 Page faults: tasks running without errors Page faults: tasks running without errors

0.25
44
C# C#
F# Go

0.2
23

0.16
12
# page faults (mean)

# page faults (mean)

0.12
6

0.08
3

0.04
1
0

0
2 4 6 8 10 5 10 15 20

task task

Page faults: tasks running without errors Page faults: tasks running without errors
1410

0.25
C# C#
Haskell Java
420

0.2
0.16
125
# page faults (mean)

# page faults (mean)

0.12
37

0.08
10

0.04
2
0

5 10 15 5 10 15 20

task task

Page faults: tasks running without errors Page faults: tasks running without errors
0.25

0.25

C# C#
Python Ruby
0.2

0.2
0.16

0.16
# page faults (mean)

# page faults (mean)

0.12

0.12
0.08

0.08
0.04

0.04
0

5 10 15 20 5 10 15

task task

Figure 315: Page faults (mean) of tasks running successfully (C# vs. other languages)

258
 Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)

0.25
●

40

0.20
30

0.15
Go
F#

20

0.10
●
10

0.05
●

0.00
●
● ● ●
0

0 10 20 30 40 0.00 0.05 0.10 0.15 0.20 0.25

C# C#

Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)

0.25
1400

●
1200

0.20
1000

0.15
800
Haskell

Java
600

0.10
400

0.05
200

0.00

● ● ●
0

0 200 400 600 800 1000 1200 1400 0.00 0.05 0.10 0.15 0.20 0.25

C# C#

Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)
0.25

0.25
0.20

0.20
0.15

0.15
Python

Ruby
0.10

0.10
0.05

0.05
0.00

0.00

● ● ● ●

0.00 0.05 0.10 0.15 0.20 0.25 0.00 0.05 0.10 0.15 0.20 0.25

C# C#

Figure 316: Page faults (mean) of tasks running successfully (C# vs. other languages)

259
 Page faults: tasks running without errors Page faults: tasks running without errors

44

11
F# F#
Go Haskell

6.9
23

4.2
12
# page faults (mean)

# page faults (mean)

2.5
6

1.3
3

0.5
1
0

0
2 4 6 8 10 12 14 2 4 6 8 10

task task

Page faults: tasks running without errors Page faults: tasks running without errors
44

44
F# F#
Java Python
23

23
12

12
# page faults (mean)

# page faults (mean)

6

6
3

3
1

1
0

2 4 6 8 10 12 14 2 4 6 8 10 12

task task

Page faults: tasks running without errors

44

F#
Ruby
23
12
# page faults (mean)

6
3
1
0

2 4 6 8 10 12

task

Figure 317: Page faults (mean) of tasks running successfully (F# vs. other languages)

260
 Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)

40

10
●

8
30

Haskell

6
Go

20

4
●
●
●
10

2
● ●

●●● ● ●
0

0
0 10 20 30 40 0 2 4 6 8 10

F# F#

Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)
40

40
30

30
Python
Java

20

20
10

10

●●● ● ● ●
●●● ● ●
0

0 10 20 30 40 0 10 20 30 40

F# F#

Page faults: tasks running without errors (mean)

40
30
Ruby

20
10

●●● ● ●
0

0 10 20 30 40

F#

Figure 318: Page faults (mean) of tasks running successfully (F# vs. other languages)

261
 Page faults: tasks running without errors Page faults: tasks running without errors
1410

Go Go
Haskell Java
420
125
# page faults (mean)

# page faults (mean)

37

0
10
2
0

0 5 10 15 20 25 30 0 5 10 15 20 25 30 35

task task

Page faults: tasks running without errors Page faults: tasks running without errors
0.25

Go Go
Python Ruby
0.2
0.16
# page faults (mean)

# page faults (mean)

0.12

0
0.08
0.04
0

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30

task task

Figure 319: Page faults (mean) of tasks running successfully (Go vs. other languages)

262
 Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)
1400

1.0
●
1200

0.5
1000
800
Haskell

Java

0.0
●
600
400

−0.5
200

−1.0
●
●
0

0 200 400 600 800 1000 1200 1400 −1.0 −0.5 0.0 0.5 1.0

Go Go

Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)
0.25

1.0

●
0.20

0.5
0.15
Python

Ruby

0.0

●
0.10

−0.5
0.05

−1.0
0.00

0.00 0.05 0.10 0.15 0.20 0.25 −1.0 −0.5 0.0 0.5 1.0

Go Go

Figure 320: Page faults (mean) of tasks running successfully (Go vs. other languages)

263
 Page faults: tasks running without errors Page faults: tasks running without errors
1410

1410
Haskell Haskell
Java Python
420

420
125

125
# page faults (mean)

# page faults (mean)

37

37
10

10
2

2
0

0
0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Page faults: tasks running without errors

1410

Haskell
Ruby
420
125
# page faults (mean)

37
10
2
0

0 5 10 15 20 25 30

task

Figure 321: Page faults (mean) of tasks running successfully (Haskell vs. other languages)

264
 Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)

1400

1400
1200

1200
1000

1000
800

800
Python
Java

600

600
400

400
200

200
●
●
● ● ●
●
● ●
0

0
0 200 400 600 800 1000 1200 1400 0 200 400 600 800 1000 1200 1400

Haskell Haskell

Page faults: tasks running without errors (mean)

1400
1200
1000
800
Ruby

600
400
200

●
●
● ●
0

0 200 400 600 800 1000 1200 1400

Haskell

Figure 322: Page faults (mean) of tasks running successfully (Haskell vs. other languages)

Page faults: tasks running without errors Page faults: tasks running without errors
0.25

Java Java
Python Ruby
0.2
0.16
# page faults (mean)

# page faults (mean)

0.12

0
0.08
0.04
0

0 5 10 15 20 25 30 0 5 10 15 20 25 30

task task

Figure 323: Page faults (mean) of tasks running successfully (Java vs. other languages)

265
 Page faults: tasks running without errors (mean) Page faults: tasks running without errors (mean)

0.25

1.0
●

0.20

0.5
0.15
Python

Ruby

0.0
●

0.10

−0.5
0.05

−1.0
0.00

0.00 0.05 0.10 0.15 0.20 0.25 −1.0 −0.5 0.0 0.5 1.0

Java Java

Figure 324: Page faults (mean) of tasks running successfully (Java vs. other languages)

Page faults: tasks running without errors

0.25

Python
Ruby
0.2
0.16
# page faults (mean)

0.12
0.08
0.04
0

0 5 10 15 20 25 30

task

Figure 325: Page faults (mean) of tasks running successfully (Python vs. other languages)

Page faults: tasks running without errors (mean)

0.25
0.20
0.15
Ruby

0.10
0.05
0.00

● ●

0.00 0.05 0.10 0.15 0.20 0.25

Python

Figure 326: Page faults (mean) of tasks running successfully (Python vs. other languages)

266
 Page faults: tasks running without errors
44

C
C#
F#
Go
23

Haskell
Java
Python
Ruby
12
# page faults (mean)

6
3
1
0

1 2 3 4 5 6 7

task

Figure 327: Page faults (mean) of tasks running successfully (all languages)

267
I. Timeout analysis

268
 Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0
● ● ● ● ● ● ● ● ● ●

0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6
C C
C# F#
all all

0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 5 10 15 20 25 5 10 15

task task

Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0
● ● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6
C C
Go Haskell
all all
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 10 20 30 40 0 10 20 30 40

task task

Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0

● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6

C C
Java Python
all all
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 5 10 15 20 25 30 35 0 10 20 30 40

task task

Timeout analysis: scalability tasks

1.0

● ● ● ● ● ●
0.8
does it timeout? (binary.timeout.selector)

0.6

C
Ruby
all
0.4
0.2

269
0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 10 20 30 40

task

Figure 328: Timeout analysis of scalability tasks (C vs. other languages)

 Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

1.0
● ● ● ●

0.8

0.8
0.6

0.6
C#

F#
0.4

0.4
0.2

0.2
0.0

0.0
● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

C C

Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)
1.0

1.0
● ● ●
0.8

0.8
0.6

0.6
Haskell
Go

0.4

0.4
0.2

0.2
0.0

0.0

● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

C C

Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)
1.0

1.0

● ● ● ●
0.8

0.8
0.6

0.6
Python
Java

0.4

0.4
0.2

0.2
0.0

0.0

● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

C C

Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

●
0.8
0.6
Ruby

0.4
0.2

270
0.0

● ●

0.0 0.2 0.4 0.6 0.8 1.0

Figure 329: Timeout analysis of scalability tasks (C vs. other languages)

 Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0
● ● ● ● ● ● ● ● ● ● ●

0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6
C# C#
F# Go
all all
0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

5 10 15 0 5 10 15 20 25

task task

Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0
● ● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

C# 0.6 C#
Haskell Java
all all
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

5 10 15 20 25 0 5 10 15 20 25

task task

Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0

● ● ● ● ● ● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6

C# C#
Python Ruby
all all
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 5 10 15 20 25 0 5 10 15 20 25

task task

Figure 330: Timeout analysis of scalability tasks (C# vs. other languages)

271
 Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

1.0
● ● ●

0.8

0.8
0.6

0.6
Go
F#

0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

C# C#

Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)
1.0

1.0
● ●
0.8

0.8
0.6

0.6
Haskell

Java
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

C# C#

Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)
1.0

1.0

● ● ● ●
0.8

0.8
0.6

0.6
Python

Ruby
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

C# C#

Figure 331: Timeout analysis of scalability tasks (C# vs. other languages)

272
 Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0
● ● ● ● ● ● ● ● ● ● ● ●

0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6
F# F#
Go Haskell
all all
0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

5 10 15 5 10 15

task task

Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0
● ● ● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

F# 0.6 F#
Java Python
all all
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

5 10 15 5 10 15

task task

Timeout analysis: scalability tasks

1.0

● ● ● ● ●
0.8
does it timeout? (binary.timeout.selector)

0.6

F#
Ruby
all
0.4
0.2
0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

5 10 15

task

Figure 332: Timeout analysis of scalability tasks (F# vs. other languages)

273
 Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

1.0
● ●

0.8

0.8
0.6

0.6
Haskell
Go

0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

F# F#

Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)
1.0

1.0
● ● ●
0.8

0.8
0.6

0.6
Python
Java

0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

F# F#

Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

●
0.8
0.6
Ruby

0.4
0.2
0.0

● ●

0.0 0.2 0.4 0.6 0.8 1.0

F#

Figure 333: Timeout analysis of scalability tasks (F# vs. other languages)

274
 Timeout analysis: scalability tasks Timeout analysis: scalability tasks
1.0

1.0
● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6
Go Go
Haskell Java
all all
0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 10 20 30 40 0 5 10 15 20 25 30 35

task task

Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0

● ● ● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6

Go Go
Python Ruby
all all
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 10 20 30 40 0 10 20 30 40

task task

Figure 334: Timeout analysis of scalability tasks (Go vs. other languages)

275
 Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)
1.0

1.0
● ● ●
0.8

0.8
0.6

0.6
Haskell

Java
0.4

0.4
0.2

0.2
0.0

0.0
● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

Go Go

Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)
1.0

1.0

● ● ●
0.8

0.8
0.6

0.6
Python

Ruby
0.4

0.4
0.2

0.2
0.0

0.0

● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

Go Go

Figure 335: Timeout analysis of scalability tasks (Go vs. other languages)

276
 Timeout analysis: scalability tasks Timeout analysis: scalability tasks
1.0

1.0
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6
Haskell Haskell
Java Python
all all
0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 5 10 15 20 25 30 35 0 10 20 30 40

task task

Timeout analysis: scalability tasks

1.0

● ● ● ● ● ● ● ●
0.8
does it timeout? (binary.timeout.selector)

0.6

Haskell
Ruby
all
0.4
0.2
0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 10 20 30

task

Figure 336: Timeout analysis of scalability tasks (Haskell vs. other languages)

277
 Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

1.0
● ● ●

0.8

0.8
0.6

0.6
Python
Java

0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

Haskell Haskell

Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

● ●
0.8
0.6
Ruby

0.4
0.2
0.0

● ●

0.0 0.2 0.4 0.6 0.8 1.0

Haskell

Figure 337: Timeout analysis of scalability tasks (Haskell vs. other languages)

Timeout analysis: scalability tasks Timeout analysis: scalability tasks

1.0

1.0

● ● ● ● ● ● ● ● ● ●
0.8

0.8
does it timeout? (binary.timeout.selector)

does it timeout? (binary.timeout.selector)

0.6

0.6

Java Java
Python Ruby
all all
0.4

0.4
0.2

0.2
0.0

0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35

task task

Figure 338: Timeout analysis of scalability tasks (Java vs. other languages)

278
 Timeout analysis: scalability tasks (binary.timeout.selector) Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

1.0
● ● ● ●

0.8

0.8
0.6

0.6
Python

Ruby
0.4

0.4
0.2

0.2
0.0

0.0
● ● ● ●

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

Java Java

Figure 339: Timeout analysis of scalability tasks (Java vs. other languages)

Timeout analysis: scalability tasks

1.0

● ● ● ● ● ● ● ● ●
0.8
does it timeout? (binary.timeout.selector)

0.6

Python
Ruby
all
0.4
0.2
0.0

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 10 20 30 40

task

Figure 340: Timeout analysis of scalability tasks (Python vs. other languages)

Timeout analysis: scalability tasks (binary.timeout.selector)

1.0

● ●
0.8
0.6
Ruby

0.4
0.2
0.0

● ●

0.0 0.2 0.4 0.6 0.8 1.0

Python

Figure 341: Timeout analysis of scalability tasks (Python vs. other languages)

279
 Timeout analysis: scalability tasks
1.0

● ● ● ● ● ●
0.8
does it timeout? (binary.timeout.selector)

C
C#
0.6

F#
Go
Haskell
Java
Python
0.4

Ruby
all
0.2
0.0

● ● ● ● ● ● ● ● ● ● ● ● ●

2 4 6 8 10 12

task

Figure 342: Timeout analysis of scalability tasks (all languages)

280
J. Number of solutions

281
 Number of solutions per task Number of solutions per task

●
● ● ●

6
●
● ●

5
●●● ●

4
# solutions (length.na.zero)

# solutions (length.na.zero)
●● ● ●

4
●● ● ● ●● ● ●
● ● ●
● ●
●● ● ●● ● ● ●
●

3
C C
C# ●● ● ● ● ● ●● ● ● ●● ●
● ●
●● ● ●● ● ● ●
● F#

3
all all
●
●● ●
●●
●●
●●● ●
●● ●
●
●●
● ●
●
●●● ● ●●
● ●●
●●●●●
●●● ●● ●
●●●
●
●●●●● ● ●●
●
●● ● ● ● ●●
●●●●● ● ●●●●
●●●
●● ●●
●●
● ●

2
●●
●● ●●
●●● ●
●● ●
●● ●●● ● ●
●●
● ●●
●●●
●●●
●● ●● ●
●●●●
●●●● ●● ●●
●
●● ● ● ●
● ●●●
●●● ●●
●●●● ●●
●●
● ●

2
●
●
●●
●
●●
●
●●●
●●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●●●
●
●●
●
●●
●●●
●
●●
●●
●
●●
●●●●
●
●●
●
●●
●
●●
●●
●
●●●
●●
●●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●● ●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●●●●●
●●●
●●●
●●●
●
●●
●●●
●
●●
●
●●
●●●
●●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●
●●
1
●
●
●●
●
●●
●
●●●
●●●
●
●●
●
●●
●●
●●
●
●●
●●
●●●●
●
●●●
●●
●●
●●
●
●●
●●
●●●●
●
●●
●●
●
●●
●●
●●
●●
●●●
●
●●
●●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●●●
●
●●●
●
●●
●●
●
●●●
●
●●●
●
●●●●●
●●
●●
●●
●
●●●
●
●●
●●●
●●
●●●●●●●
●●
●●
●●●
●●●
●
●●
●●●
●
●●
●●
●
●●●
●
●●●●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●●
●
●●
●●
●
●●●
●●●
●
●●●
●●
●●

1
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●●
●●●
●●●
●●●
●
●●
●
●●
●●
●
●●
●●
●●●
●●
●
●●●
●●
●●
●
●●
●●
●●
●
●●●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●
●●
●
●
●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
● ●
●●
●
●●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●●
●●
●●
●
●●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●
●●
●
0

0
0 200 400 600 0 200 400 600

task task

Number of solutions per task Number of solutions per task

●
● ● ●
5

8
● ●
4

●● ●

6
# solutions (length.na.zero)

# solutions (length.na.zero)
●
● ● ●● ● ● ●
●● ●● ● ●● ● ● ●● ●● ● ●● ● ●
●●●
●● ● ●● ●
● ● ● ●
●
3

C C
Go ● ● ● ● ● ● ● Haskell
●

4
all all
●●
●● ●●
●●●
●
●●
●●● ●●
●●●●
●●
●● ●
●●
●● ●●
●
●●●●
●●●●●
●●●
●●●●
●●●●
●
●●●●●●●●● ● ● ●●
●
●●●● ● ●
● ● ●●●
●●● ●
●
●●●
● ●●
●●●● ●●
●●
●
● ● ●●
●
2

● ●●●
● ● ●
●●● ●● ●● ● ●●●
●● ● ●
● ●
●● ● ● ● ● ●● ● ●
●

●
●● ●
●●●
●●●● ● ●
●
●●●●
●●●
●●●●
●
●●●
● ● ●●
●●●●
●
●●
●●
●●●●●●●●●●●
●●●●●
● ●●
●● ●●
●
●●●●● ●
●●●● ●●●
●
●●● ●●● ●
●●●
●●
● ●●
●●
●● ●
●

2
●
●
●●
●
●●
●
●●●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●●●
●
●●●
●
●●
●●
●●
●●
●●
●●
●●●
●
●●
●●
●
●●
●●
●
●●●
●●
●●●
●
●●
●
●●
●●
●●
●●
●●
●●●
●
●●
●●
●
●●
●
●●
●
●●●●
●
●●●
●
●●
●
●●
●●
●
●●●
●●
●
●●●
●
●●●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●
●●●
●●●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●●
●
●●
●●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●●
●●●
●
●●●
●●
●●
1

●
●
●●
●
●●
●●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●●●
●
●●
●
●●
●●●
●
●●
●●●
●●
●●●
●
●●
●●
●
●●●
●
●●●
●●
●●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●●●
●
●●●
●
●●
●●
●
●●●
●
●●●
●
●●●●●
●
●
●●
●●
●●
●●
●●●
●
●●
●●●
●●
●●●
●●●●
●●●
●
●●●
●●
●●●
●●
●●●
●
●●
●●
●
●●●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●
●●●
●
●●●
●
●●
●●

●
●●
●●●
●
●●
●
●●
●
●●●
●●●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●●●
●
●●
●
●●●
●●
●●●
●
●●
●
●●●
●
●●
●
●●
●●
●●
●
●●●
●●
●●●●●
●●●
●●●
●
●●
●●
●
●●●●
●●
●●●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●●
●
●●●
●
●●
●●●
●
●●
●●
●
●●
●
●●
●
●● ●
●
●●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●●
●
● ●
●●
●●
●●
●
●●
●
●●
●●●
●●
●●
●●
●●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●●
●●
●●●
●
●●
●
●●
●
●●
●●
●●●
●
●●●
●
●●
●
●●●
●
●●
●●
●●
●●
●
●●
●●
●●
●●●●
●●●
●●
●●
●●
●●
●
●●
●●
●●
●●
●●
●●●
●●
●●
●●
●
●●
●
●●●
●●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●●
●
●●
●●
●●
●●
●
●●
●●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●●
●
●●
0

0 200 400 600 0 200 400 600

task task

Number of solutions per task Number of solutions per task

12

● ● ● ●
6

10

●
● ● ●
5

●
# solutions (length.na.zero)

# solutions (length.na.zero)

●● ●● ● ● ●
4

C C
● ●
●● ●
● ● ● ●● ● ● ● ●
● ●
●● ● ●● ●
● ● ● Java
●
●● ● Python
●
3

all all
●
● ● ● ●● ● ●● ●

●
●● ●●●
●● ● ●●
●● ●
●●
● ●●●●●
●●●●
●●
●●●●
●●●
●●●● ●
●●●●
●●● ●
●
●●●
●
●● ● ● ●
●●●
●
●● ● ●●
●●
●● ●
●●●
●●● ●
●●●●●●●
●●●
●● ●●
●●
● ● ●● ● ● ● ● ●● ●●
●● ●● ●
●●● ● ●● ● ●●
● ● ●
2

●
●
● ●● ●● ●
● ●●●● ●
●●● ●● ●●●● ● ●
●●●● ● ●●●
●●●●
●●● ●●● ●●●
●● ● ●● ●
● ●●
●

●
●
●●
●
●●
●●
●●
●●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●●●●
●
●●
●●
●
●●●
●
●●●
●●●
●
●●●
●
●●
●
●●
●●
●●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●●
●●●●
●●●
●●●●
●
●●●
●●
●●●
●
●●
●
●●●
●
●●●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●
●●●
●●●
●●●
●
●●
●● ●
●●●
●●●●
●●
●●
●●● ● ●
●●●
●●
●●●
●●●●●●●
●●● ●●
●
●●● ●
●●●
●
● ●●
●
●●●
●●
●
●●●●●●●
●●●
●●● ●● ●
● ●●
●
●●●● ●● ●
●● ●●●●
●
●●
●●● ●● ●
●●●
●●
●
●●
●●
●● ●
●●
●
●● ● ●
●●●●
1

●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●●
●
●●
●●
●
●●
●●●
●
●●
●●
●
●●●
●
●●●
●●
●●
●●
●●
●●
●●
●
●●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●●●
●
●●●
●
●●
●●
●
●●●
●
●●●
●
●●●●
●
●●
●●
●●
●
●●●
●
●●
●●
●●
●
●●
●
●●●
●●●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●●
●
●●●
●●●
●●●
●
●●
●●

●
●
●●●
●
●●
●●
●
●●
●
●●
●●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●●●
●
●●
●
●●●
●
●●
●●
●●
●●
●●
●●
●●●●●●
●●●●
●
●●
●●
●
●●
●●●
●
●●
●●●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●
● ●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●●
●
●● ●●
●●●
●
●●
●
●●●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●●●
●
●●
●
●●
●●
●
●●●●
●
●●●
●
●●
●
●●●
●
●●●
●●
●●
●
●●
●●
●
●●●●●●
●●●
●●
●●
●●
●●
●
●●●
●●●
●
●●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●●
●
●●
●●
●●
●
●●
●●
●●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●
●●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●
●
●●●
●
●●
●●
●
●
0

0 200 400 600 0 200 400 600

task task

Number of solutions per task

● ●

●
8
# solutions (length.na.zero)

●
● ●
C
Ruby
● ● ●● ● ● ● ● ● all
4

●
●● ●● ● ●
●● ● ● ●●●
●
●●●● ●●
● ●
●● ● ● ●● ●●
●● ●
● ● ● ●
● ●
●

●●●
●
●● ●
●●●
●●●● ●
●●●●
●
●● ●
●●
●●
●
●●● ●●●
●●● ●●
●●
●●●
●●●●
●●
●●●●
●●●●
●
●●●●●
●
● ●●
● ●
●●
●●●
●● ●
●● ●●
●●
●●●
●●●
● ●● ●●●
●●●
●●●
●
●● ●●
●●
●● ●
●●
2

●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●●
●
●●
●●
●●●
●●
●●
●
●●●
●●
●●●
●
●●
●●●
●●
●●●
●
●●
●●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●
●
●●
●●
●●
●●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●●
●●●●
●●
●●
●●●
●●●
●
●●●
●●
●●
●
●●●
●
●●●●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●●
●●●
●●●
●●●
●
●●
●●

●●
●
●●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●●
●
●●●
●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●
●●●●
●
●●●
●
●●
●
●●●
●
●●●
●●
●●
●●
●●
●●
●●●●
●●●●
●
●●
●●
●
●●
●
●●●
●●
●●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●
●●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●●
●
●●
●●
●
●
282
0

0 200 400 600

task

Figure 343: Number of solutions per task (C vs. other languages)

 Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

● ●

6
5
● ● ● ●

4
● ● ●

4
● ● ●

3
C#

F#
● ● ●

3
● ● ●

2
● ● ● ●

2
● ● ● ●
1
● ● ● ● ●

1
● ● ● ● ● ● ● ●
0

0
0 1 2 3 4 5 0 1 2 3 4 5 6

C C

Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

●
5

8
●
4

● ● ●

6
● ● ●
● ● ● ●
3

Haskell
Go

● ● ● ●

4
● ● ● ● ●
2

● ● ● ● ●

● ● ● ●

2
● ● ● ● ●
1

● ● ● ●

● ● ● ● ● ● ●
0

0 1 2 3 4 5 0 2 4 6 8

C C

Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)
12

● ● ● ●
6

10

●
5

● ● ● ● ●
4

●
Python
Java

● ● ● ●
3

● ● ● ●

● ● ● ● ● ● ● ●
2

● ● ● ●

● ● ● ● ● ● ● ●
1

● ● ● ●

● ● ● ● ● ● ● ●
0

0 1 2 3 4 5 6 0 2 4 6 8 10 12

C C

Number of solutions per task (length.na.zero)

● ●

●
8
6

●
Ruby

● ● ● ●
4

● ● ● ●

● ● ● ●
2

● ● ● ●

● ● ● ●
283
0

0 2 4 6 8

Figure 344: Number of solutions per task (C vs. other languages)

 Number of solutions per task Number of solutions per task

● ●
● ●

5
●
● ●

5
● ●●● ●

4
# solutions (length.na.zero)

# solutions (length.na.zero)
●●●● ● ● ●

4
●● ● ● ●● ● ● ●● ● ● ●
● ●● ● ●

3
C# C#
● ● ● ● ●● ● ● F# Go
3

all all
●● ● ●●●
●●●
●
●●●
● ●● ●●
●●● ●
●●
● ●●
●
●●●●
●●●●
●●●●
●●●
●●●● ●●●
● ●● ● ● ●● ●● ● ● ●●
● ●●
●
●●●
●● ●●●
●● ●
●● ●
●

2
●●
● ●●
●●● ● ●
● ●
●● ● ● ●●
●●● ● ●●●● ●● ● ● ● ●●
● ● ● ●●●
●●
2

●
●●
●●
●
●●●
●●●
●
●●
●
●●
●●●
●●
●●
●
●●●
●
●●
●
●●●●
●
●●●
●
●●
●●
●●●
●
●●
●●
●
●●
●●
●●●●
●
●●●
●
●●
●●
●
●●●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●
●
●●
●●
●●
●●
●●
●
●
●●
●●●
●
●●
●
●●
●●●
●●●
●●
●●●
●●
●●●
●●
●●
●●
●●
●●
●
●●
●
●●●
●
●●●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●●●●
●
●●●
●
●●
●●

1
●
●
●●
●●
●●●
●●●
●
●●
●
●●●
●●
●●
●
●● ●
●●●
●●
●●
●
●●
●● ●
●
●●
●●
●●
●●
●●●●
●
●●●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●●●
●
●●●
●●
●
●●
●●
●●
●
●●●
●
●●●
●
●●
●●
●●
●●●
●
●●●●
●
●●
●
●●
●●●●●
●●
●●
●●
●●●
●●●
●●
●●●
●●
●
●●●
●
●●●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●
●●●●●
●●●
●
●●
●
1

●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●● ●
●●
●
●●●
●
●●
●
●●
●
●●
●●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●●
●●●●●
●●●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●
●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
0

0
0 200 400 600 0 200 400 600

task task

Number of solutions per task Number of solutions per task

● ● ● ●
8

6
●
● ●

5
●● ●
6
# solutions (length.na.zero)

# solutions (length.na.zero)
●● ●●●● ● ●

4
●
● ● ●● ● ● ●

C# C#
● ●●● ● ● ● ● ● Haskell
● ● ●
● ●
● ● ● ● ●
● ● ● ● Java
●●
4

all all

● ●●● ●●● ● ●● ●●●

●● ● ●● ● ●● ●
●● ●
●
●●
●● ●● ●
● ●
●●●●
●●●●
●● ●
●●●● ●
●●●
●● ●
●●●
●●●● ● ●
●●● ● ●
●●● ●●
●● ● ●●
●●●● ●
●●●
●●
●● ● ● ●●
2

● ●
●●
●●
●●● ●●
●●●●●
●●●
●
●●
●●
●●●
●● ●●
●●
●●
●●●
● ●●
●●● ●
●●●● ●● ●
●●●● ●●● ●
●●● ● ● ●● ●●●
●●
● ●
●●● ●
2

●●
●
●●●
●●
●●●
●●
●●
●
●●
●
●●●●
●
●●
●
●●●●
●●
●
●●●
●●●
●●
●
●●
●●●●
●
●●
●
●●
●
●●
●●
●
●●●
●●●
●●
●
●●
●
●●●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●●
●
●●●
●
●●●●
●
●●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●●●●●
●●●
●●●
●●
●●●
●●
●●
●●
●●
●
●●●
●
●●●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●●
●
●●
●
●●
●●
●●●
●
●●
●
●●
●
●●●
●●●
●●●
●
●●
●●
1

●
●
●●
●●
●
●●●
●●
●●●
●
●●
●
●●
●●
●●●
●●
●
●●●●
●●
●●●
●●
●●
●
●●
●●
●●●
●
●●
●
●●
●
●●
●●
●●
●●●
●●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●●
●
●●●
●
●●
●●●
●●
●
●●●
●
●●
●●
●●
●●●●
●
●
●●
●●
●●
●●●
●
●●
●
●●●
●
●●
●●●
●
●●●●
●●●
●●●
●●
●●●
●●
●●
●●
●
●●
●●●
●
●●●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●
●●
●
●●
●●
●●●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●

●
●●
●
●●
●●
●●
●●
●●
●●
●
●●
●●●●
●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●
●●
●
●
●●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●●
●
●●●●
●●●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●●
●●
●●●
●●
●●
●
●●
●●
●●
●
●●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
● ●
●●
●
●●
●●●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●
●●
●●
●●●
●●●
●●
●
●●
●●
●●
●●
●
●●
●●
●●
●●
●
●●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●●●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●
0

0 200 400 600 0 200 400 600

task task

Number of solutions per task Number of solutions per task

12

● ● ●

●
8
10

●
# solutions (length.na.zero)

# solutions (length.na.zero)

● ●
8

●
●
● ●
C# C#
● Python
● Ruby
6

all ● ● ●●● ● ● ● ● ● ● all

4

●
● ● ● ●● ● ●● ●

● ● ● ●
●●●● ●●
●● ●● ●
●●● ● ●● ● ●●
● ● ● ●
● ● ●
● ● ● ●●●
●●● ● ●● ● ●●● ●
● ● ● ●
● ●
4

●
●
● ●● ● ●
● ●● ● ●●● ● ●●●● ●
●●●
●● ● ●● ●
●● ● ● ● ●●●● ● ●● ●
● ●●
●●
●
●● ●
●●
●● ●
●● ● ●●
●● ●
●●
●●●●● ●●● ●
●●
●●●
●●
●●●●●
● ●●
●
●●● ●
●● ●
● ● ●●● ●● ● ●
●●
● ● ●● ●● ●●●
●●
●●●
●● ●
●● ● ●
2

●●
●●●
●●●
●
●●●● ● ● ●
●●●
●●
●●●
●●●●●
●●● ●●
●●●●
●●●
●●●●●●
●●●●●●●
●●●
●
●● ●● ●
● ●
●●● ●● ●
●● ●●●
●●●
●●● ●●● ●
●●●●
●●
●●
●● ●
●●
● ●●
●●●●
2

●●
●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●
●●
●
●●●●
●
●●
●
●●●
●●
●
●●
●●
●
●●●
●●●
●●
●
●●
●
●●●
●●
●●
●●
●●●
●●
●
●●●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●●
●
●●
●●
●●
●●
●●
●●
●●●●
●
●●
●●
●●
●
●●●
●
●
●●●
●●
●●
●
●●
●
●●●
●●●●
●●
●●
●●●
●●●
●
●●
●●●
●
●●
●
●●
●●
●●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●● ●●
●
●●
●
●●
●●
●●●●●
●
●●●
●
●●●
●●
●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●●
●
●●
●
●●●●
●●
●
●●
●●●
●●
●●
●
●●
●●●●
●●
●
●●
●
●●
●●
●●
●●●●●
●
●●
●●●
●●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●●
●
●●●
●●●
●
●●●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●
●●●●●●
●●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●●●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●●
●● ●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●

●
●●
●
●●●
●
●●
●
●●
●
●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●●●●●
●●●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●●
●
●●●
●●●
●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●● ●
●●
●
●●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●●
●
●●
●●●
●●
●
●●
●
●●
●●
●●
●●●●
●●●
●●
●
●●
●●
●●
●
●●
●●
●●●
●
●●●
●
●●●
●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
0

0 200 400 600 0 200 400 600

task task

Figure 345: Number of solutions per task (C# vs. other languages)

284
 Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

5
5
●

4
● ●

4
● ● ●

3
Go
F#

● ● ●
3

● ● ● ● ● ●

2
● ● ● ●
2

● ● ● ● ●

1
● ● ● ● ● ●
1

● ● ● ● ● ● ●
0

0
0 1 2 3 4 5 6 0 1 2 3 4 5

C# C#

Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

● ● ● ●
8

6
5
● ● ●
6

● ● ●

4
● ● ● ●
Haskell

Java

● ● ● ● ● ●
4

● ● ● ● ●
● ● ● ● ●
2

● ● ● ●
2

● ● ● ● ●
1

● ● ● ●

● ● ● ● ● ● ●
0

0 2 4 6 8 0 1 2 3 4 5 6

C# C#

Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)
12

● ● ●

●
8
10

● ●
8

●
●
Python

Ruby

● ●
6

● ● ● ● ●
4

● ● ● ●

● ● ● ● ● ● ● ● ●
4

● ● ● ● ●
● ● ● ●
2

● ● ● ●
2

● ● ● ●
● ● ● ●

● ● ● ● ● ●
0

0 2 4 6 8 10 12 0 2 4 6 8

C# C#

Figure 346: Number of solutions per task (C# vs. other languages)

285
 Number of solutions per task Number of solutions per task

● ●

8
5
●● ●

6
# solutions (length.na.zero)

# solutions (length.na.zero)
● ● ● ●

4 ● ● ● ● ●

F# F#
●● ●
●●● ● ● ● ● ● ● ● ●● ● ● Go ● ● ● ● ● ● ● ● ● Haskell
●
3

4
all all

● ● ● ● ●●● ●● ●●●
● ● ● ●● ●
●● ●
●●● ● ● ●●
●
●●●
● ● ●●●● ●● ●●●●
●●●
●●●●
● ●●●
●
●●●●● ●●●● ● ● ● ● ● ● ●● ●
●
●●●
● ●
● ●
●● ●
●
2

●● ●
●●●
●
●● ● ●●●●●● ●
●●●●●
● ● ●●
●●●●
●●● ●●
●●●
● ●●●
●● ● ●
●●●● ●●●
● ● ●
●● ●● ●
●●
● ●
●● ● ●

2
●
●
●●
●●
●
●●●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●●●●
●●
●●
●
●●
●●
●●
●●●●
●
●●
●
●●
●●
●●●●
●
●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●
●●●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●
●●
●●
●●
●●●
●
●●
●●●
●
●●
●
●●
●●●
●●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●
●●
●
●●●●●
●●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●●
●
●●
●
●●
●
●●●●
●
●●●
●●
●●
1

●
●
●●
●●
●●●
●●
●●●
●
●●
●
●●
●●●
●●
●●●●
●●
●●●
●●
●●
●●
●●
●●●
●●
●
●●
●●
●●
●●
●●●
●
●●
●●
●
●●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●
●
●●●
●
●●●
●
● ●●
●
●●●
●
●●●
●
●●●●
●
●●
●●
●●
●●●
●
●●
●●
●●
●●●
●●●●●
●●
●●
●●●●
●●
●●
●●●
●
●●
●●
●
●●●●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●
●

●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
● ●
●●
●
●●
●
●●
●
●
●●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●●
●
●
0

0
0 200 400 600 0 200 400 600

task task

Number of solutions per task Number of solutions per task

12
● ● ● ● ●
6

10
●
5

●
# solutions (length.na.zero)

# solutions (length.na.zero)
●● ● ● ● ● ● ●
4

8
●

F# F#
● ● ● ●
● ● ● ● ● ● ● ● Java
●● ● ● Python
●
3

all all
● ● ● ● ● ●● ●

●●● ●● ● ● ● ●●
● ●●
●●
● ●
●●● ●
●●●● ●
●●●
●●●● ● ●
●● ● ●
●●● ● ● ● ●
●●●● ● ●
●
●● ● ● ●● ● ● ● ●
●● ●●
●● ●
●● ●
●●● ● ●● ● ●●
● ● ●
2

●
●
● ●● ● ●●
● ●● ● ●●● ●●●● ●
●●●●●● ●● ●
●● ● ● ● ●●●● ●● ●
● ●●

●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●●●
●
●●●●
●●
●
●●
●●
●●
●●
●
●●
●●●●
●
●●
●●
●
●●
●●
●
●●●●
●
●●●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●
●
●●
●
●●●
●
●●
●●
●
●●●
●
●●●
●
●●
●
●●
●●
●●
●
●●●
●
●●●●
●
●●
●●
●●
●●
●●
●●
●
●●●
●
●●
●
●●
●
●●●●●●●
●
●●
●●●
●●
●●
●●●
●
●●
●●
●
●●●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●
●●●
●●●
●●●
●
●●● ●●
●●●
●●●●
●●● ● ●
●●●●
● ●●
●●●
●●● ●
●●
●●●●●●
●
●●●●●
●●●●●●●●●
●
●● ●●
●●● ● ●● ●● ●
●●
● ●●● ●
●
●●● ●● ●
●●●●●
●
●●● ●
●●
● ●●
●●●●
1

●
●
●●
●●
●
●●●
●
●●●
●
●●
●
●●
●●●
●
●●●
●
●●●●
●●
●
●●
●●●
●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●
●
●●●
●●●
●●
●●●
●
●●
●
●●●
●
●●
●●
●
●●
●●
●
●●
●
●●●●
●●●
●
●●●
●
●●●
●
●●●
●
●●●
●
●●
●●
●●
●
●●●●
●
●●●
●
●●
●
●●
●
●●● ●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●●●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●

●
●
●●
●●
●●●
●●
●●
●
●●
●
●●●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●●
●
● ●
●●
●
●●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●●
●
●
0

0 200 400 600 0 200 400 600

task task

Number of solutions per task

● ●

●
8
# solutions (length.na.zero)

●
6

●
F#
Ruby
● ● ●● ● ● ● ● ● ● all
4

● ● ● ● ●
● ● ● ●●●
●●● ● ●● ● ●●● ●
● ● ● ●
● ●

●
●●●
●
●● ●
●●●●
●● ●●
● ●
●●●●●● ●●●● ●
●●
●●●●●●●●
● ●●
●
●●●
●●●● ●
●
● ● ●●● ●●
● ●●
●
● ● ● ●● ●●●
●● ●
●● ●
●● ● ●
2

●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●●●
●●
●●●●
●
●●
●
●●●
●●
●
●●
●●
●●
●●
●●●●
●
●●
●
●●
●●
●●
●●●
●
●●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●●
●●
●
●●●
●●
●
●●●●
●
●●●
●
●●
●●
●●
●●●●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●
●●●
●●●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●
●●

●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●●
●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●●
●
●●●
●
●●
0

0 200 400 600

task

Figure 347: Number of solutions per task (F# vs. other languages)

286
 Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

8
5
● ● ●

6
●

4 ● ● ●

Haskell
Go

● ● ● ● ● ● ●
3

4
● ● ●
● ● ● ● ●
2

● ● ● ●

2
● ● ● ● ●
1

● ● ● ●

● ● ● ●
0

0
0 1 2 3 4 5 6 0 2 4 6 8

F# F#

Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

12
● ● ● ●
6

10
●
5

● ● ● ● ●
4

8
●
Python
Java

● ● ● ● ●
3

● ● ●

● ● ● ● ● ● ● ●
2

● ● ●

● ● ● ● ● ● ● ● ●
1

● ● ● ●

● ● ● ●
0

0 1 2 3 4 5 6 0 2 4 6 8 10 12

F# F#

Number of solutions per task (length.na.zero)

● ●

●
8
6

●
Ruby

● ● ● ●
4

● ● ● ● ●

● ● ●
2

● ● ● ● ●

● ● ●
0

0 2 4 6 8

F#

Figure 348: Number of solutions per task (F# vs. other languages)

287
 Number of solutions per task Number of solutions per task

● ● ● ●
8

6
5
●● ●
6
# solutions (length.na.zero)

# solutions (length.na.zero)
●● ● ● ●

4
● ● ● ● ●

Go Go
● ● ● ● ● ● ● Haskell
● ●
●● ● ● ●●●
● ● ● ● ● ● ● ● ● ●● ● Java
●
●●
4

3
all all

●● ● ● ● ●● ●●● ● ●●
● ●●●
● ● ● ●●● ● ● ●● ●● ●
● ● ●●
●
●●●
●
●●●
● ●● ●●●● ●●●●
●●●●●
●●●
●●●
●
●●●
●●●●
●
●●●●
●●
●●●●●
● ●● ●●
●● ●● ●
●●● ● ●
● ● ● ●
●
●●
●●
●● ● ●
●●
● ●●
● ● ●●
●

2
● ● ●●
●●●
●●
●
●●●
● ●●●●
●●●●●●
●
●●●●●
●●●●●
●●●●
●●
●
●●
●●
●●
●●●●
●●●●
●
●●●
●●● ●
●● ● ● ●●
●●●●● ●●●●● ● ●
●● ●
●
●●●
●● ●
●●
● ●
●●
●
●●● ●
●●
2

●
●●
●
●●
●
●●●
●●●
●
●●
●
●●
●●
●●●
●
●●●●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●●
●●●
●
●●
●●
●
●●
●●
●
●●●
●●
●●
●●
●●
●
●●
●●
●
●●●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●●●
●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●●
●
●●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●●
●●●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●●
●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●●
●●●
●
●●●
●●
●●

1
●
●
●●
●●
●
●●●
●●●
●
●●
●
●●
●
●●●
●●
●
●●●●
●
●●
●●
●●●
●
●●
●●●
●●
●●●
●●
●●
●
●●
●●
●
●
●●●
●●
●●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●●●
●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
●●●
●
●●
●●
●●
●●●
●
●●
●
●●●
●
●●
●
●●
●●●
●●●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●●●●
●
●●●
●
●●
●●

●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●●●
●●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●●●
●●
●
●●
●
●●
●●
●●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●
●●
●
●●
●●
●●
●
●●●●
●●●
●●
●
●●
●●
●●
●
●●
●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●● ●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●●●
●
●●
●
●●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●●●●●
●●●
●●
●●
●●
●●
●
●●●●
●
●●
●●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●●
●●
●
●●●
●
●●
●●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●●
●●
0

0
0 200 400 600 0 200 400 600

task task

Number of solutions per task Number of solutions per task

12

● ● ●

●
8
10

●
# solutions (length.na.zero)

# solutions (length.na.zero)

● ●
8

●
●
Go Go
● Python
● Ruby
6

all ● ● ● ● ● ● ● ● ● all
4

● ● ● ● ● ●● ●

● ●● ● ● ● ●●
●● ●● ●
●●● ● ●● ● ●●
● ● ● ●● ● ● ●● ● ●
● ● ●● ●●●
●●●
● ●● ● ●● ● ●●● ●
● ●● ● ●
●● ●
4

●
●
● ●● ● ●●
●
●● ●● ● ●●● ● ●●●●● ●
●●●●●● ●●● ●
●● ● ● ● ●●●● ● ●● ●
●● ●●
●
●
●●●
●
●● ●
●
●●●
●
●●●
●
●● ● ●●●
●●●●
●●●●●●●●
●
●●●
●●●●●
●●
●●●●●
●
●●
●●●
●
●●
●●●
●
●●●● ●
●● ●
●● ●
●●●
●● ●●
●●●●
●● ● ●●
●
●●
●●
●●●●
●● ●
●
●● ●
●
● ● ●●
●●
2

●●●
●
●●●
●●●
●●●
●
●●
●●
●●● ●
●●●●
●●●
●●●●●
●●●
●
●●●
●●●
●
●●●●
●●●
●●●●
●
●●●
●●●●●●●●●
●
●● ●●●
● ●● ●
●● ●● ●
●
●●●
●●● ●
●
●●● ●
●●
●●
●●
●●●●●
●
●●● ●
●●
●●
●● ●
●●●
●●
2

●
●●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●●●
●●
●●
●●
●●
●
●●
●●
●●●
●
●●●
●●
●●
●●●●
●
●●
●
●●
●●
●●
●●●
●
●●
●●
●
●●
●
●●
●●
●●
●●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●●●
●
●●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●
●●●
●
●●
●●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●●
●
●●
●
●●
●●●
●●●
●
●●●
●
●●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●
●●●
●
●●
●●
●●
●●
●●
●
●●●
●
●●●
●
●●
●●
●●
●●●
●●
●●
●
●●
●●
●
●●●
●●
●●
●●
●●
●
●●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●
●●●●
● ●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●●●●
●
●●●
●
●●
●

●
●●
●●●
●
●●
●
●●
● ●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●●
●●
●●
●
●●
●●
●●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●
●●
●
●●
●●
●
●●
●●●●●
●●●
●
●●
●
●●
●●
●
●●●
●●●
●●●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●●
●●
●●
●●
●●
●
●●●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●●
●
●●●
●●
●●●
●●
●
●●
●
●●
●
●●
● ●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●●
●● ●
●●
●
●●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●●
●
●●
●
●●
●●
●●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●
●●
●
●●
●●
●●
●●●●
●●●
●●
●●
●●
●●
●
●●
●●●
●
●●●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●●
●●
●
●●●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●●
●●
●●
●●●
●●
●
●●
●
●●
●
●●
● ●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●●
●●
0

0 200 400 600 0 200 400 600

task task

Figure 349: Number of solutions per task (Go vs. other languages)

288
 Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

● ● ●
8

6
5
● ● ●
6

● ● ●

4
● ●
Haskell

Java
● ● ● ● ●
4

3
● ● ● ●
● ● ● ●

2
● ● ● ●
2

● ● ● ● ●

1
● ● ● ●

● ● ● ● ● ● ● ●
0

0
0 2 4 6 8 0 1 2 3 4 5 6

Go Go

Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)
12

● ● ●

●
8
10

● ●
8

●
●
Python

Ruby

● ●
6

● ● ●
4

● ● ● ●

● ● ● ● ● ●
4

● ● ● ● ●
● ● ● ● ●
2

● ● ● ●
2

● ● ● ●
● ● ● ●

● ● ● ● ● ● ● ●
0

0 2 4 6 8 10 12 0 2 4 6 8

Go Go

Figure 350: Number of solutions per task (Go vs. other languages)

289
 Number of solutions per task Number of solutions per task

12
● ●
8

10
●

● ●● ● ● ●
6
# solutions (length.na.zero)

# solutions (length.na.zero)
● ● ●

8
● ● ● ● ●
●

Haskell Haskell
●
●● ● ● ● ● ● ● ● Java● ● ●● ● Python
●
4

6
all all
● ●● ● ● ● ● ● ● ●● ●
●
● ●●● ●
● ●●● ●● ●●●
● ● ●● ● ●● ●
● ● ●●
● ● ● ● ● ● ●●
●● ● ● ● ●
●●● ● ● ●● ●● ●●
● ● ●
●

4
● ●
●
●●●●
● ● ●●●●●● ●
●●●●●
●
● ●●
●● ●
●●
●●
●●●●●
●●●●●●
●●
●●●
●●● ● ●
●●●
●●●● ●
●●
●● ● ●
●
●● ●
●●●
●● ● ●●
● ●
●●●
● ●
●● ●
●
● ●●● ● ●
● ●● ● ●
●●● ●●●●
●● ●●
●
●●●
●● ● ●● ●●
●● ●● ● ● ●●●● ●● ●
● ●●
2

●●
●●●
●●●
●●●
●
●●
● ●● ●●
●●
●●●
●●●●
●●●●
●●● ●●
●●●●
●
●●●
●
●●●●●●●
●●
●
●●●
●●●●●●
●
●● ●●
●●● ●
●●●●●●●●●
●● ●●● ●
●
●●● ●●●●
●●●●●
●●
●●●
●●
●
●●●
●●●
●●●

2
●
●
●●
●
●●
●●
●●
●●●
●●
●
●●
●
●●
●●●
●
●●●
●●●
●●
●●
●
●●●
●
●●
●●
●●
●●
●●●
●●
●
●●
●
●●●
●●
●●●
●●
●●
●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●●●●
●
●●●
●
●●
●
●●
●●●
●
●●●
●
●●●●●
●
●●
●●
●●●
●●
●
●●
●
●●●
●
●●
●
●●●●
●●●●
●●●
●●●
●●●
●●
●●●
●
●●
●
●●
●
●●●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●●
●
●●
●●
●
●●
●●●
●●●
●
●●●
●
●●
●●
●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●●
●●
●●
●●●
●
●●
●●●
●●
●
●●
●●
●●
●
●●
●
●●●
●●●
●
●●●
●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●●●
●
●●●
●
●●●
●●●
●
●●●
●
●●●●
●
●●
●●
●●
●●●
●
●●
●●●
●
●●
●
●●●●
●●●●●
●●
●●●●
●●
●●
●●
●●
●●
●
●●
●
●●●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
● ●
●
●●●
●
●●●
●●●
●
●●●
●
●●

●
●
●●
●●
●
●●
●●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●●
●
●●
●●
●●
●●
●
●●
●●
●●
●●●●
●●●
●●
●●
●●
●●
●
●●
●●
●●
●●
●
●●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●●
●
●●
●
●●
●
●● ●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●●
●
●●●
● ●
●●
●●
●●
●●
●
●●
●●●
●●
●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●●
●●●
●●
●
●●●
●
●●
●
●●●
●
●●
●●●
●●
●
●●
●●
●
●●
●●●●
●●●
●●
●●
●●
●●
●
●●
●●
●
●●●
●
●●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●
●●●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●●
●
●●●
●
0

0
0 200 400 600 0 200 400 600

task task

Number of solutions per task

● ●

● ●
8
# solutions (length.na.zero)

●● ●
6

● ● ● ● ● ●
Haskell
Ruby
● ● ● ● ● ● ● ● ● ● ● ● all ●
4

● ●●● ● ●●● ●
● ● ●● ● ●●
●
●●●
●●● ● ● ● ● ●● ● ●●● ●
● ●
●● ● ●
● ●

●●
●
●● ●
●●
●●
●●●●
●● ● ●●●
●●
●●● ●
●
●●●
●●
●●●
● ●●● ●
●●
●
●●
●●●
●●●●●
●
●●●●
●●
●
●●●
●●●
● ●
●
● ●●
●●●●● ●●
●● ● ●
●●●
●●● ●●
●●●●
●● ●
●● ●
●●
● ● ●● ●
2

●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●●
●
●●●
●●
●●
●●
●●●●
●
●●
●●
●
●●●
●●●
●●
●●
●
●●
●●
●
●●●
●●
●●●
●●
●
●●
●●
●●
●●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●●
●
●●●
●
●●
●●●●
●
●●●
●
●●●
●
●●●●●
●
●●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●●
●●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●
●●
●
●●●●●
●●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●●●
●
●●
●●
●
●●●
●●●
●
●●●
●
●●●

●
●
●●
●●
●●
●●●
●
●●
●●●
●●
●
●●
●
●●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●●
●●
●●●
●
●●
●
●●●
●
●●
●●●
●●
●
●●
●
●●●
●
●●●●
●●●
●●
●●
●●
●●
●
●●
●
●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●
●●●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●●
●
●●
●
●●●
●
●●●
●
0

0 200 400 600

task

Figure 351: Number of solutions per task (Haskell vs. other languages)

290
 Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

12
●

10
●

● ● ● ●

6
● ●

8
●

Python
Java
● ● ● ● ● ●

6
● ● ● ● ●
● ● ● ●
● ● ● ● ● ● ●

4
● ● ● ● ● ● ● ● ● ● ● ● ●
2

● ● ● ● ● ●

2
● ● ● ● ● ●
● ● ● ● ●

● ● ● ● ● ● ● ●
0

0
0 2 4 6 8 0 2 4 6 8 10 12

Haskell Haskell

Number of solutions per task (length.na.zero)

● ●

●
8
6

●
Ruby

● ● ● ● ● ●
4

● ● ● ● ● ●

● ● ● ● ●
2

● ● ● ● ● ●

● ● ● ●
0

0 2 4 6 8

Haskell

Figure 352: Number of solutions per task (Haskell vs. other languages)

Number of solutions per task Number of solutions per task

12

● ● ●

●
8
10

●
# solutions (length.na.zero)

# solutions (length.na.zero)

● ● ● ● ●
8

●
●
Java Java
● ● ● ● Python
● Ruby
6

all ●
●● ● ●● ● ● ● ● ● ● all
4

● ● ● ● ● ●● ●

●●● ● ● ●
●● ●●
●● ●● ●
●●● ● ● ●● ● ●●
● ● ● ● ● ●
● ● ●
● ● ● ●●●
●●● ● ● ●● ● ●●● ●
● ● ● ●
●● ● ●
●●
4

●
●
● ●● ● ●
●●● ●● ● ●●● ●●●● ●
●●●● ● ●● ●
●● ●● ● ● ●●●● ● ●● ●
●● ● ●●●
●●
●
●● ●●
●● ●
●●● ●●
●● ●
●●●●
●●
●●●
●●●
● ●
●
●●●
●●●●●
●●●
●
●● ●●
●●
●●●
●●●
● ●
● ●
●●
●●●● ●
●●
●● ●●
●
● ●● ●
●●●
●●●●●●
●● ●
●●● ●
●●●
●●●●●
2

●●
●●●
●●●
●●●● ●● ●
●●●●
● ●●
●●●
●●
●●●
●●●
●
●●●●●
●●
●●●●
●●●●●
●
●●
●●●
●
●●
●●
●●●
●
●● ●● ●
●●
●●● ●● ●●●
●●
●●
● ●●●●
●●
●●● ●
●●
●●●
●●●●●
●●
●●●
●●
● ●●
●●●
●●
2

●●
●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●
●●●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●●
●●
●●●
●
●●
●
●●
●●
●●
●●●
●
●●●
●
●●
●
●●●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●●
●●
●●●●
●
●●
●●
●●
●●●
●
●●
●
●●●
●
●●
●●●●
●●●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●●
●
●●
●
●●
●●●
●
●●
●●
●
●●
●
●●
●●
●
●●●
●
●●
●
●●
●
●●●
●●●
●●●
●
●●
●●
●●
●
●●
●●
●●
●
●●●
●
●●
●
●●
●●
●●
●●
●●
●●
●●●
●
●●
●●
●
●●
●●
●
●●
●●
●●●
●●●
●●
●
●●
●
●●●
●●
●●●
●●
●
●●
●
●●●
●●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●●
●
●●
●●●
●
●●●
●
●●●●●
●
●●
●●
●●
●●●
●●
●
●●●
●
●●
●
●●●●
●●●●
●●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●●
●●
●●●
●
●●
●
●●
●●
●●●
●
●●
●●
●
●●
●●
●
●●
●●●
●●
●●
●
●●●
●●●
●●●
●
●●
●●

●
●
●●●
●●
●●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●
●●●
●
●●
●●
●
●●
●●
●●
●
●●
●●
●●●●
●
●●
●●
●●
●
●●
●●
●
●●●●●●
●●●●
●
●●
●●
●●
●
●●●
●●●
●
●●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●●
●●
●
●●●
●
●●
●●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●●
●● ●
●
●●●
●●
●●
●●
●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●
●●
●●
●●
●●●
●
●●
●
●●●
●●
●●
●●
●●
●●
●
●●●
●●●●●
● ●●
●
●●
●●
●
●●
●
●●●
●
●●
●●●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●●
●●
●
●●●
●
●●
●●
●●●
●●
●
●●
●
●●
●
●●
●
● ●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●●
●●
0

0 200 400 600 0 200 400 600

task task

Figure 353: Number of solutions per task (Java vs. other languages)

291
 Number of solutions per task (length.na.zero) Number of solutions per task (length.na.zero)

12
● ● ●

8
10
●

● ●

6
●
●

Python

Ruby
● ●

6
● ● ● ●

4
● ● ●

● ● ● ● ● ● ● ● ●

4
● ● ● ● ●
● ● ● ● ● ●

2
● ● ● ●
2

● ● ● ● ●
● ● ● ● ●

● ● ● ● ● ●
0

0
0 2 4 6 8 10 12 0 2 4 6 8

Java Java

Figure 354: Number of solutions per task (Java vs. other languages)

Number of solutions per task

12

●
10

● ●●
# solutions (length.na.zero)

● ●●
8

Python
● Ruby
●
6

all
●
● ● ● ● ● ●● ●

● ● ● ● ● ●●● ●●
●● ●● ●
●●●
● ● ● ●● ● ● ●●
●● ● ●
4

●
●
● ●● ● ●
● ●● ● ●●● ●
● ●●
●●● ● ●
●●●
●●
●●● ●● ●
●● ●
●●● ●● ●●●● ●
● ●● ● ●
●
● ●●
●

●●
●
●●●
●●●●
●●●
●●
●● ●
●
●●
●●●
●●●
●●
●
●●●●
●●●●●
●●
●●●●
●●
●●●
●
●●●
●●●
●
●●
●●●
●●
●
●●●
●●●
●● ●
●● ●
● ●●
●●● ●●●●
●● ● ●
●
●●●
●
●●●● ●●
●
●●●●●
●●
●●●
●●
●● ●
●●
●●●
●●●
●●
●●
2

●●
●
●●
●
●●●
●
●●●
●
●●
●●●●
●
●●
●●
●
●●●
●●
●
●●
●●
●●
●●
●
●●
●●
●
●●●
●●●
●●
●●
●
●●
●●●●
●●
●●●
●
●●
●●●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●●●
●
●●●
●
●●
●●●
●
●●●
●
●●●●●
●
●●
●●
●●
●●●
●
●●
●
●●●
●
●●
●
●●●●
●●●●
●●
●●
●●●
●●
●●
●●
●●●
●
●●
●
●●
●● ●
●●
●●
●
●●
●
●●
●●●
●●
●●
●
●●
●
●●
●●
●●●
●
●●
●●
●
●●●
●●●
●●●●
●
●●

●●
●●●
●●
●
●●●
●
●●
●
●●
●●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●●●●
●●
●
●●
●
●●
●
●●●●
●
●●●
●
●●
●
●●●
●●●
●●●
●
●●
●
●●
●●
●●●●●
●●●●
●
●●
●●
●●
●
●●
●
●●●
●
●●●
●●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●●
●●
●●
●
●●●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●●
●●
●●
●●●
●●
●
●●
●
●●●
●
● ●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●●
●
●●
●
●●●
●
●●●
●
●
0

0 200 400 600

task

Figure 355: Number of solutions per task (Python vs. other languages)

Number of solutions per task (length.na.zero)

12
10

● ●

●
8
Ruby

● ● ● ● ●
4

● ● ● ● ● ●

● ● ● ● ● ●
2

● ● ● ● ● ● ● ●

● ● ● ● ●
0

0 2 4 6 8 10 12

Python

Figure 356: Number of solutions per task (Python vs. other languages)

292
 Number of solutions per task
12

●
10

● ●●
# solutions (length.na.zero)

● ●● ● C
8

C#
● F#
Go
● ● ●● ● ● Haskell
●
6

Java
●
● ●● ● ●● ● ● ● ● ●● Python
●

Ruby
●
●●● ●● ● ● ●
●
●●●● ●●
●● ● ●
●● ●
●●●
● ● ● ● ●● ●● ● ●●
●● ●all ●
●
4

●
●
● ●●
●●●● ●
●●●
●●●
● ●●●● ●
●
●●● ●
●● ●●
●●●
●●
●● ●●
●
●●●
●
●●
●●●● ●●●●
●●●●
●●●
●●●●●● ●●●
●● ●
● ●
●● ● ●
●
●●● ● ●●
●●

●
●●●
●
●●●
●●
●●
●
●●
●●
●
●●
●
●●
●●
●●●●●
●
●●●
●
●●●
●●
●●●
●
●●
●●
●●
●●
●●
●●●
●
●●●
●●
●
●●
●●
●●●
●
●●
●
●●
●●
●
●●
●●●
●●
●
●●
●●
●●
●
●●●
●
●●
●●
●
●●●
●
●●●
●●● ●
●●
●●●●
●●
●●
●●●
●
●●●●●●●
●●
●●●●
●●●
●●
●●●
●
●●●●●●●
●
●●
●
●●
●●
●●●
●●
●●
●
●●
●●
●
●●
●●●
●
●●
●●●
●●●
●●
●●●
2

●
●
●●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●●●
●
●●
●●
●
●●
●●
●●●
●
●●
●
●●
●
●●
●●
●
●●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●●
●
●●●
●
●●
●●
●●
●●●
●●
●
●●
●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●
●
●●
●●
●●●●●
●●
●●
●●●
●●●
●
●●
●●
●●
●●
●
●●
●
●●●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●●
●●●●●
●
●●●
●
●●
●●

●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●●
●
●●
●
●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●
●●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●●
●
●●
●
0

0 200 400 600

task

Figure 357: Number of solutions per task (all languages)

293