SoftwareX 15 (2021) 100711

Contents lists available at ScienceDirect

SoftwareX
journal homepage: www.elsevier.com/locate/softx

Original software publication

Cnerator: A Python application for the controlled stochastic generation

of standard C source code
∗
Francisco Ortin a,b , , Javier Escalada a
a
University of Oviedo, Computer Science Department, Federico Garcia Lorca 18, 33007, Oviedo, Spain
b
Munster Technological University, Department of Computer Science, Rossa Avenue, Bishopstown, Cork, Ireland

article info a b s t r a c t

Article history: The Big Code and Mining Software Repositories research lines analyze large amounts of source code to
Received 25 January 2021 improve software engineering practices. Massive codebases are used to train machine learning models
Received in revised form 7 May 2021 aimed at improving the software development process. One example is decompilation, where C code
Accepted 13 May 2021
and its compiled binaries can be used to train machine learning models to improve decompilation.
Keywords: However, obtaining massive codebases of portable C code is not an easy task, since most applications
Big code use particular libraries, operating systems, or language extensions. In this paper, we present Cnerator,
Mining software repositories a Python application that provides the stochastic generation of large amounts of standard C code.
Machine learning It is highly configurable, allowing the user to specify the probability distributions of each language
C programming language construct, properties of the generated code, and post-processing modifications of the output programs.
Stochastic program generation Cnerator has been successfully used to generate code that, utilized to train machine learning models,
Python has improved the performance of existing decompilers. It has also been used in the implementation
of an infrastructure for the automatic extraction of code patterns.
© 2021 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).

Code metadata

Current code version v.1.0.1

Permanent link for this code version https://github.com/ElsevierSoftwareX/SOFTX-D-21-00022
Legal code license BSD 3-Clause
Code versioning system used Git
Software code language used Python 3.7+
Compilation dependencies numpy PyPI Python package
Developer documentation https://computationalreflection.github.io/Cnerator
Support email for questions [email protected]

Software metadata
Current software version v.1.0.1
Permanent link to executables of this version https://github.com/ComputationalReflection/Cnerator/releases
Legal software license BSD 3-Clause
Operating system Linux, Windows and Mac OS
Installation requirements & dependencies numpy PyPI Python package
User manual https://github.com/ComputationalReflection/Cnerator/blob/main/user-manual.md
Support email for questions [email protected]

1. Introduction

∗ Corresponding author at: University of Oviedo, Computer Science ‘‘Big code’’ is a recent line of research that brings together big
Department, Federico Garcia Lorca 18, 33007, Oviedo, Spain.
data and source code analysis [1]. It is based on using the source
E-mail addresses: [email protected] (Francisco Ortin),
[email protected] (Javier Escalada). code of millions of programs to build different types of tools to
URLs: http://www.reflection.uniovi.es/ortin (Francisco Ortin), improve software development [2]. Machine learning is used to
https://www.javierescalada.es (Javier Escalada). create useful predictive models that learn common patterns from

https://doi.org/10.1016/j.softx.2021.100711
2352-7110/© 2021 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Francisco Ortin and Javier Escalada SoftwareX 15 (2021) 100711

a large number of source code applications [3,4]. For example, The rest of this article is structured as follows. Section 2 details
JavaScript programs are used to train a model capable of deob- the related work, and the software functionality and architecture
fuscate variable names from their usage [5]; Java and C# files are presented in Section 3. An illustrative example is described
with the same behavior are used to learn automatic translation in Section 4. Section 5 evaluates Cnerator and compares it with
between these two languages [6]; and vulnerable C source code related approaches. Conclusions are presented in Section 6.
is used to train models that predict vulnerabilities by analyzing
the source code of new programs [7]. 2. Problems and background
Likewise, the Mining Software Repositories (MSR) field ana-
lyzes the rich data available in source code repositories to un- As mentioned, there are tools for the random generation of
cover information about software systems and projects [8]. In
C code. Most of them are aimed at finding bugs in C compilers,
this case, the source code data is enriched with other information
rather than at training machine learning models.
taken from defect tracking systems, archived communications be-
Csmith is a well-known generator of random C programs [25]
tween project personnel, version control systems, and question-
created as a fork of randprog [26]. Its main purpose is the de-
and-answer sites [9]. Examples of MSR projects include software
tection of bugs in C compilers. Generated programs conform to
repair models that analyze bug fix transactions in software repos-
the C99 standard, and they avoid the undefined behavior con-
itories [10], change prediction systems that identify the code
structs specified in C99. To find compiler bugs, each generated
prone to change in subsequent releases [11], and the automatic
program is compiled by different compilers and executed. If a
retrieval of help information for source code fragments using
checksum of the global variables upon program termination is
question-and-answer websites [12].
One of the languages used to build those machine learning different from the rest of executions, the compiler that produced
models is the C programming language. From its creation in that binary has an error (i.e., randomized differential testing).
the 70s, C is still in use, particularly for the development of Csmith implements different safety mechanisms such as pointer
systems software, embedded system applications, and programs analysis, bounded loop constructs, and different dynamic checks.
that access specific hardware addresses [13]. Its low demand for Csmith has been used to detect more than 325 errors in existing
runtime system resources and its wide availability have made compilers, including the verified CompCert C compiler [27].
it a usual candidate to implement language interpreters and ldrgen is a tool for the random generation of C programs
computationally intensive programs. According to the Tiobe [14], to test compilers and program analysis tools [28]. Existing sys-
LangPop [15], and the Transparent Language Popularity Index [16] tems generate large amounts of dead code that the compiler
programming language rankings, C is still the most widely used reduces to little relevant binary, because dead code is deleted.
language in January 20211 , obtaining the fifth position in the For this reason, ldrgen implements a liveness analysis algo-
PYPL [17], Redmonk [18] and Trendy Skills [19] rankings. rithm during program generation to avoid producing dead code.
There exist many different variants of the C programming It is implemented as a plugin for the Frama-C extensible frame-
language, which include language extensions and modifications work [29]. ldrgen has been used to detect missed compiler
depending on the operating system, compiler and target hard- optimizations [30].
ware. Therefore, different ANSI/ISO standardizations of C are de- YARPGen is a random test-case generator for C and C++ com-
fined to facilitate the development of portable software [20]. pilers, created to find and report compiler bugs [31]. YARPGen
However, it is still difficult to find applications written in 100% is created to overcome the saturation point reached by existing
standard C source code that could be compiled with many differ- compiler testing methods, where very few bugs are found. This is
ent compilers. Most of the existing open-source applications have not because compilers are bug-free, but rather because generators
particular dependencies on non-portable code. This is an issue contain biases that make them incapable of testing specific parts
when building predictive models from source code, since a large of compiler implementations. YARPGen generates programs free
number of programs is usually required [21]. of undefined behaviors without dynamic safety checks, unlike
There exist tools capable of generating random C source code, Csmith. Its approach is to implement different static analyses
but they are mainly aimed at testing compilers, rather than cre- to generate code that conservatively avoids undefined behaviors.
ating machine learning models [22]. Therefore, they are not de- It also implements generation policies that systematically skew
signed to build massive amounts of standard C code, and they do probability distributions to cause certain optimizations to take
not cover every language construct—we detail them in Section 2 place more often. YARPGen has found more than 220 bugs in GCC,
and evaluate them in Section 5. LLVM, and the Intel C++ compiler. Those bugs were not previously
For this reason, we developed Cnerator, a Python applica- found by other compiler testing tools.
tion that generates large amounts of standard ANSI/ISO C source The family of Orange random C code generators is focused
code [20] to train machine learning models. Cnerator is highly on generating arithmetic expressions [32]. Instead of differential
customizable to generate all the syntactic constructs of the C testing, they track the expected values of each test after execu-
language, necessary to build accurate predictive models with tion, checking whether the obtained values are the expected ones.
machine learning algorithms. The code it generates is ready to
The programs generated by Orange generators are safe, avoiding
be compiled by any standard language implementation. Cnerator
the undefined behaviors of the C programming language. Orange
has been used to improve state-of-the-art decompilers [23] and to
code generators do not include important language features such
implement an infrastructure for the automatic extraction of code
as control flow statements, structs, arrays or pointers.
patterns [24]. The stochastic generation of source code programs
Quest is a code generator tool aimed at finding several com-
has also been used to detect bugs in existing compilers [25].
piler bugs related to calling conventions [33]. It generates func-
Another potential use of Cnerator is testing whether a compiler
tion declarations randomly, and then generates type-driven test
implements the ANSI/ISO standard specification correctly.
cases that invoke each function. A global variable is generated
for each parameter and return value. Assertions are used to
1 These rankings measure the popularity of each programming language,
check that each value received and returned is the appropriate
using different criteria. For example, the Tiobe language ranking uses 25
search engines to calculate each language index, depending on the number
one. Quest avoids undefined behavior by simply not generating
of searches done by users (https://www.tiobe.com/tiobe-index/programming- potentially dangerous constructs (e.g., arithmetic expressions). It
languages-definition). was used to find 13 bugs in 5 different compilers [33].
2
Francisco Ortin and Javier Escalada SoftwareX 15 (2021) 100711

3. Software framework constructs, and provides different helper functions to facilitate

its specification. As shown in Fig. 3 (explained in Section 4),
In this section, we first describe the main functionalities of JSON probability specification files permit the use of those helper
Cnerator. Then, we present its architecture and a brief description functions to modify the default probability distributions.
of each module. The second type of JSON input allows the user to control the
number and characteristics of all the functions to be generated.
3.1. Software functionalities For example, we can enforce Cnerator to generate a program with
as many functions as built-in types in the language, and make
These are the main functionalities provided by Cnerator: each function return an expression of each built-in type. The
Controlled Function Generation module interprets the JSON file to
1. ANSI/ISO standard C. All the source code generated by Cn- drive the process of program generation. To this aim, it asks the
erator follows the ISO/IEC 9899:2018 (C17) standard speci- main Program Generation module to generate random functions,
fication [20]. The code strictly follows the syntax grammar, and discards those not fulfilling the requirements specified in the
type system and semantic rules of the standard specifi- JSON file. If no function generation file is provided, Program Gen-
cation. This makes the generated code to be able to be eration just produces a random program following the existing
compiled by any compiler implementing the standard. probability distributions.
2. Probabilistic randomness. C language constructs are ran- The third type of argument is an ordered collection of Python
domly generated, following different probability distribu- post-processing specification files. When the user wants the out-
tions specified by the user. For example, it is possible to put program to fulfill some requirements not guaranteed by the
describe the probability of each kind of statement and ex- stochastic generation process, these post-processing files can be
pression construct, the number of statements in a function, used to modify the generated code in order to meet such re-
and the types of their arguments and return values. To quirements. By following an introspective implementation of the
this aim, the user can specify fixed probabilities of each Visitor design pattern [35], the user can specify the traversal of
element, or use different probability distributions, such as the program representation produced by Cnerator (an example
normal, uniform, and direct and inverse proportional. is presented in Section 4). We use the singledispatch Python
3. Highly customizable. Many features of the programs to package [36] to traverse program representations.
be generated are customizable. Some examples include The Program Representation module is mainly an in-memory
the types of each language construct, array dimensions representation of Abstract Syntax Trees (AST) [37]. Cnerator pro-
and sizes, struct fields, maximum depth of expression and duces ASTs modeling the generated program before generating
statement trees, number of function parameters and state- the output code. The AST data structure implements the Inter-
ments, global and local variables, structures of control flow preter design pattern [38] to convert a program representation
statements, and type promotions, among others—see the into a set of output compilation units [39].
detailed documentation [34].
4. Large amounts of code. Cnerator is designed to allow gen- 4. Illustrative example
erating large amounts of C source code (see Section 5). One
parameter indicates the number of independent compila- In this section, we show how Cnerator was used to build
tion units to be created for the output application, so that classifiers for inferring the return type of C functions from bi-
each unit could be treated as an independent module. This nary code, outperforming the existing decompilers [23]. We used
feature, together with the probabilistic randomness, makes Cnerator to produce abundant C source code in order to train
Cnerator an ideal tool to build predictive models, because supervised machine learning models. Such models are capable of
the input programs used to train such models comprise inferring function return types by just analyzing their compiled
abundant and varied code patterns. binary code. As shown in Fig. 2, state-of-the-art decompilers pro-
vide 30% F1 -measure2 for this classification problem [23], while
3.2. Software architecture our machine learning model achieves 79.1%.
Initially, we searched for 100% standard C applications in
Fig. 1 shows a UML package diagram describing the archi- GitHub, Bitbucket and SourceForge, finding 2329 instances (func-
tecture of Cnerator. When executing the tool, three types of tions) to be used in our classification problem [23]. We then
optional arguments may be passed: command-line arguments, realized that model accuracy could be increased if more programs
JSON specification files, and Python post-processing traversals. were added to the dataset (F1 -measure showed a high coefficient
If no parameter is passed, Cnerator creates a random output of variation). Therefore, we used Cnerator to generate additional
program, using the default probability values [34]. The generated programs and included them in the dataset. Fig. 2 shows how
program consists of a group of compilation units (a pair of .h and F1 -measure of the classifier grows with an increasing number of
.c files) that can be compiled independently, even though they instances, obtaining a coefficient of variation below 2% for 20,000
commonly depend on other compilation units. functions. Gradient boosting was the classifier with the best
As command-line arguments, the user may pass parameters performance (accuracy and F1 -measure) out of the 14 machine
such as the number of output compilation units, probability val- learning algorithms tested [23].
ues of syntactic constructs, and the output directory and file Fig. 3 shows an excerpt of two of the JSON files used to
names, among others (all the parameters are detailed in the user customize Cnerator. The one on the left overwrites some default
manual [34]). The Parameter Processing module takes all the probabilities. The first entry (function_basic_stmt_prob) de-
parameters passed by the user and customizes the behavior of fines the probability of building basic statements (i.e., statements
Cnerator accordingly. not containing other statements, unlike for and switch), and
Cnerator accepts two types of JSON configuration files as pa- the second one states that 10% array definitions should initialize
rameters (examples are presented in Section 4). The first one their values. These two examples specify fixed probabilities that
allows specifying the probability values and probability distribu-
tions of multiple C syntactic constructs. The Probabilities mod- 2 Although different compilers were measured, Fig. 2 only shows IDA
ule stores the default probability distributions of all the syntax (Hex-Rays) because it outperforms the rest of decompilers [23].

3
Francisco Ortin and Javier Escalada SoftwareX 15 (2021) 100711

Fig. 1. UML package diagram describing the architecture of Cnerator.

The _instrument_statements function takes a list of state-

ments (represented as AST nodes) and adds a unique label –
prefixed with __RETURN__ (line 09) – before each return state-
ment. That function is later used in the traversal of function
definitions (line 18), and do, while, for and block statements
(line 26)—if and switch control flow statements follow the
same template. The code in Fig. 4 is an instance of an introspec-
tive implementation of the Visitor design pattern [35]. The visit
annotations indicate the AST node to be traversed, and default
tree traversal is performed with reflection [40].

5. Implementation and empirical results

We compare Cnerator with the random C code generators

discussed in Section 2. We select the following evaluation criteria
related to the generation of large amounts of standard C code to
Fig. 2. F1 -measure of gradient boosting classification model for increasing train machine learning models (Section 3.1). It is worth noting
number of instances in the dataset. Whiskers represent 95% confidence intervals, that the comparison is not aimed at identifying the best tool,
for 30 training/test iterations for each number of instances. The x-axis has a but at analyzing their appropriateness to generate abundant and
logarithmic scale.
varied source code.
1. Generation of standard ANSI/ISO C code, which can be
must sum zero. The three remaining entries use uniform/equal, compiled by any standard compiler implementation.
proportional and normal distributions to specify, respectively, the 2. It can be specified the probability of each language con-
usage of primitive types, and the number of function parameters struct.
and statements in the main function. 3. The tool can be customized to describe properties fulfilled
The right-hand JSON file in Fig. 3 shows the controlled by the different language constructs (e.g., types returned by
function-generation method used to build the dataset for the functions, array dimensions, maximum depth of expression
decompiler scenario. The two first entries are examples of how and statement trees, or number of function parameters and
we made Cnerator generate 1000 functions returning each of statements).
the types defined in the standard specification.3 The condition 4. Generation of a configurable number of independent com-
in the lambda expression checks that the returned type is the pilation units.
expected one. The last entry shows a different example, not used 5. Avoidance of dynamic undefined and unspecified behav-
in the decompiler scenario, where the user demands Cnerator to iors.
generate a function containing at least one if statement with an 6. Generation of code for all the language constructs.
else clause. 7. Generation of large numbers of functions (and their invo-
Fig. 4 shows an example Python post-process specification file. cations).
The code traverses the representation of the generated program
(its AST), and adds a unique label before each return statement. The results of the comparison are detailed in Table 1. All
The purpose of this instrumentation is to identify in the com- the tools but ldrgen and Orange generate standard C code.
piled code the binary patterns used for each high-level return Csmith is the only system, besides Cnerator, that allows setting
statement. Those binary code patterns are later labeled with the the probability of some language constructs (inline functions,
high-level return type to build predictive models with supervised array accessing loop, and built-in function invocation for Csmith;
machine learning algorithms [23]. much more for Cnerator [34]). Csmith supports the customization
of some basic properties of the generated code (criterion 3),
3 Only void and bool are shown for the sake of brevity. such as the maximum depth of blocks, pointer indirections, array
4
Francisco Ortin and Javier Escalada SoftwareX 15 (2021) 100711

Fig. 3. Two example JSON files used to customize program generation with Cnerator. The left-hand side shows a sample probability specification file, and the
right-hand side specifies an example of controlled function generation.

dimensions and expression complexity. ldrgen allows the speci- Table 1

fication of the maximum number of statements per block, expres- Qualitative comparison of random C code generators (a = yes, d = no, and
r = partially).
sion depth, functions, function arguments, and statement nesting
Criterion Csmith ldrgen YARPGen Orange Quest Cnerator
depth. Cnerator is the only system that allows the generation of
any number of independent compilation units. 1 a d a d a a
2 r d d d d a
The feature of avoiding dynamic undefined and unspecified
3 r r d d d a
behaviors (criterion 5) shows a different pattern. All the tools but 4 d d d d d a
Cnerator provide this feature because their objective, unlike ours, 5 a r a a a d
is detecting bugs in compilers (Section 2). On the contrary, Cn- 6 r d r d d a
erator is the only tool that generates all the language constructs 7 d d d d d a
(criterion 6), because the generated code is used to train machine
learning models that perform better when the code has more
variability. Both Csmith and YARPGen provide a lot of language
These results show how the differences between Cnerator
constructs, but do not support others.4
In order to evaluate the capability of generating large amounts and the rest of the random C generators are caused by the
of source code (criterion 7), we measure the tools that allow spec- purpose they are designed for. All the tools but Cnerator are
ifying the number of functions in a program (Csmith, ldrgen and
Cnerator) and ask them to generate programs with an increasing aimed at testing compiler implementations, and that is why, for
number of functions. Both Csmith and ldrgen show a runtime those tools, the avoidance of dynamic undefined behaviors is so
memory error when they are asked to generate 100 functions,
important. However, this feature makes code generation to be
in an Intel Core i7 2.5 GHz computer with 16 GB RAM running
Ubuntu 20.04.2.0 LTS. The results for Cnerator are detailed in more difficult, even requiring the implementation of backtracking
Table 2. Cnerator generates the 20,000 functions for the example algorithms [31]. Moreover, this complexity limits the number
in Section 4 in 5.2 min, producing more than 2 million non-empty
source lines of code (SLOC). of language constructs to be generated, and the production of
programs with large amounts of source code. On the contrary,
4 Csmith does not generate assignments as statements, array-typed struct those two last features are very important to Cnerator, designed
fields, strings, dynamic memory allocation, floating-point types, unions, recur-
sion, and function pointers. YARPGen does not produce function calls, ++ and to train machine learning models. It also provides some other fea-
-- operators, pointer arithmetic operations, assignments as expressions, non- tures necessary for its aim, such as the specification of language
integer local variables, and has some restrictions when generating floating-point
values and loops. construct probabilities and a high degree of customization.
5
Francisco Ortin and Javier Escalada SoftwareX 15 (2021) 100711

Fig. 4. Python code excerpt of an AST post-processing example.

Table 2 Cnerator is distributed with different examples, configuration

Increasing sizes of C programs generated by Cnerator. SLOC stands for source files and complete documentation. Its source code is available for
lines of code and counts non-empty lines of source code, excluding comments.
download at GitHub under a permissive BSD 3-clause license.
Number of functions Seconds SLOC
10 0.335 1,078 Declaration of competing interest
50 0.433 4,159
100 0.696 13,184
The authors declare that they have no known competing finan-
500 2.907 73,935
1,000 4.269 139,756 cial interests or personal relationships that could have appeared
2,500 12.601 342,669 to influence the work reported in this paper.
5,000 30.062 717,562
10,000 74.771 1,345,993 Acknowledgments
20,000 312.237 2,692,157
This work has been partially funded by the Spanish Depart-
ment of Science, Innovation and Universities: project RTI2018-
6. Conclusions 099235-B-I00. The authors have also received funds from the
University of Oviedo, Spain through its support to official research
groups (GR-2011-0040).
Cnerator is a Python application that provides the controlled
stochastic generation of standard ANSI/ISO C code to train ma- References
chine learning models. It is highly configurable, allowing the user
[1] Defense Advanced Research Projects Agency. MUSE envisions mining ‘‘big
to define the probability distributions of each language construct, code’’ to improve software reliability and construction. 2014, http://www.
specify the properties of the generated functions, perform post- darpa.mil/news-events/2014-03-06a.
[2] Allamanis M, Barr ET, Devanbu P, Sutton C. A survey of machine learning
processing modifications of the generated programs, and define for big code and naturalness. ACM Comput Surv 2018;51(4).
the number of output compilation units. [3] Ortin F, Escalada J, Rodriguez-Prieto O. Big code: new opportunities for
Cnerator has been successfully used to build machine learning improving software construction. J Softw 2016;11(11):1083–8.
[4] Ortin F, Rodriguez-Prieto O, Pascual N, Garcia M. Heterogeneous tree struc-
models that improve state-of-the-art decompilers [23]. It has also ture classification to label java programmers according to their expertise
been utilized to implement an infrastructure for the automatic level. Future Gener Comput Syst 2020;105:380–94.
extraction of code patterns [24]. It could also be used to test [5] Raychev V, Vechev M, Krause A. Predicting program properties from
‘‘big code’’. In: Proceedings of the 42nd annual ACM SIGPLAN-SIGACT
existing C compilers, including the correct implementation of the symposium on principles of programming languages. POPL ’15, New York,
ANSI/ISO standard. NY, USA; 2015, p. 111–24.

6
Francisco Ortin and Javier Escalada SoftwareX 15 (2021) 100711

[6] Karaivanov S, Raychev V, Vechev M. Phrase-based statistical translation [25] Yang X, Chen Y, Eide E, Regehr J. Finding and understanding bugs in
of programming languages. In: Proceedings of the 2014 ACM international c compilers. In: Proceedings of the 32nd ACM SIGPLAN conference on
symposium on new ideas, new paradigms, and reflections on programming programming language design and implementation. PLDI ’11, New York,
& software. Onward! 2014, New York, NY, USA: ACM; 2014, p. 173–84. NY, USA: Association for Computing Machinery; 2011, p. 283–94.
[7] Yamaguchi F, Lottmann M, Rieck K. Generalized vulnerability extrapolation [26] Eide E, Regehr J. Volatiles are miscompiled, and what to do about it.
using abstract syntax trees. In: Proceedings of the 28th annual computer In: Proceedings of the 8th ACM international conference on embedded
security applications conference. ACSAC ’12, New York, NY, USA: ACM; software. EMSOFT ’08, New York, NY, USA: Association for Computing
2012, p. 359–68. Machinery; 2008, p. 255–64.
[8] Herzig K, Zeller A. Mining your own evidence. In: Making software. O’Reilly [27] Leroy X. Formal verification of a realistic compiler. Commun ACM
Media, Inc.; 2010. 2009;52(7):107–15.
[9] Jung W, Lee E, Wu C. A survey on mining software repositories. IEICE Trans [28] Barany G. Liveness-driven random program generation. In: Fioravanti F,
Inf Syst 2012;E95.D(5):1384–406. Gallagher JP, editors. Logic-based program synthesis and transformation.
[10] Martinez M, Monperrus M. Mining software repair models for reasoning Cham: Springer International Publishing; 2018, p. 112–27.
on the search space of automated program fixing. Empir Softw Eng [29] Cuoq P, Kirchner F, Kosmatov N, Prevosto V, Signoles J, Yakobowski B.
2015;20(1):176–205. Frama-c. In: Eleftherakis G, Hinchey M, Holcombe M, editors. Software
[11] Malhotra R, Khanna M. Software change prediction: A systematic review engineering and formal methods. Berlin, Heidelberg: Springer Berlin
and future guidelines. e Inf Softw Eng J 2019;13(1):227–59. Heidelberg; 2012, p. 233–47.
[12] Ponzanelli L, Bavota G, Di Penta M, Oliveto R, Lanza M. Mining stack- [30] Barany G. Finding missed compiler optimizations by differential testing. In:
overflow to turn the IDE into a self-confident programming prompter. Proceedings of the 27th international conference on compiler construction.
In: Proceedings of the 11th working conference on mining software CC 2018, New York, NY, USA: Association for Computing Machinery; 2018,
repositories. MSR 2014, New York, NY, USA: Association for Computing p. 82–92.
Machinery; 2014, p. 102–11. [31] Livinskii V, Babokin D, Regehr J. Random testing for c and c++ com-
[13] Meyerovich LA, Rabkin AS. Empirical analysis of programming language pilers with YARPGen. In: Proceedings of the ACM on programming
adoption. In: Proceedings of the international conference on object- languages. Vol. 4, (OOPSLA). New York, NY, USA: Association for Computing
oriented programming systems languages & applications. OOPSLA ’13, New Machinery; 2020.
York, NY, USA: Association for Computing Machinery; 2013, p. 1–18. [32] Nagai E, Awazu H, Ishiura N, Takeda N. Random testing of C compilers
[14] Tiobe. Tiobe programming language index for january 2021. 2021, https: targeting arithmetic optimization. In: Proceedings of workshop on synthe-
//www.tiobe.com/tiobe-index. sis and system integration of mixed information technologies, SASIMI ’12,
[15] LangPop. Programming language popularity. 2021, http://65.39.133.14. 2012, p. 48–53.
[16] SourceForge. The transparent language popularity index. 2021, http://lang- [33] Lindig C. Random testing of c calling conventions. In: Proceedings of the
index.sourceforge.net. sixth international symposium on automated analysis-driven debugging.
[17] PYPL. Popularity of programming languages. 2021, https://pypl.github.io. AADEBUG’05, New York, NY, USA: Association for Computing Machinery;
[18] Redmonk. The redmonk programming language ranking. 2021, https:// 2005, p. 3–12.
redmonk.com/sogrady/2020/07/27/language-rankings-6-20. [34] Ortin F, Escalada J. Cnerator user manual. 2021, https://github.com/
[19] Trendy Skills. Extracting skills that employers seek in the IT industry. 2021, ComputationalReflection/Cnerator/blob/main/user-manual.md.
https://trendyskills.com. [35] Ortin F, Quiroga J, Redondo JM, Garcia M. Attaining multiple dispatch in
[20] ISO. ISO/IEC 9899:2018 - information technology – programming languages widespread object-oriented languages. Dyna 2014;81(186):242–50.
– c. 2018, https://www.iso.org/standard/74528.html. [36] Langa L. Singledispatch python package 3.4.0.3. 2021, https://pypi.org/
[21] Babii H, Janes A, Robbes R. Modeling vocabulary for big code machine project/singledispatch/.
learning. 2019, http://arxiv.org/abs/1904.01873. [37] Rodriguez-Prieto O, Mycroft A, Ortin F. An efficient and scalable platform
[22] Chen J, Patra J, Pradel M, Xiong Y, Zhang H, Hao D, Zhang L. A survey of for java source code analysis using overlaid graph representations. IEEE
compiler testing. ACM Comput Surv 2020;53(1). Access 2020;8:72239–60.
[23] Escalada J, Scully T, Ortin F. Improving type information inferred by [38] Erich G, Richard H, Ralph J, John V. Design patterns: elements of reusable
decompilers with supervised machine learning. 2021, http://arxiv.org/abs/ object-oriented software. Addison-Wesley Professional Computing Series;
2101.08116. 1995.
[24] Escalada J, Ortin F, Scully T. An efficient platform for the automatic [39] Ortin F, Escalada J. Cnerator developer manual. 2021, https:
extraction of patterns in native code. Sci Program 2017;2017:1–16. //computationalreflection.github.io/Cnerator.
[40] Ortin F, López B, Pérez-Schofield JBG. Separating adaptable persistence
attributes through computational reflection. IEEE Softw 2004;21(6):41–9.