Delft University of Technology

The Adoption of JavaScript Linters in Practice: A Case Study on ESLint

Tómasdóttir, Kristín; Aniche, Maurício; van Deursen, Arie

DOI
10.1109/TSE.2018.2871058
Publication date
2020
Document Version
Accepted author manuscript
Published in
IEEE Transactions on Software Engineering

Citation (APA)
Tómasdóttir, K., Aniche, M., & van Deursen, A. (2020). The Adoption of JavaScript Linters in Practice: A
Case Study on ESLint. IEEE Transactions on Software Engineering, 46(8), 863-891. [8468105].
https://doi.org/10.1109/TSE.2018.2871058

Important note
To cite this publication, please use the final published version (if applicable).
Please check the document version above.

Copyright
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent
of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Takedown policy
Please contact us and provide details if you believe this document breaches copyrights.
We will remove access to the work immediately and investigate your claim.

This work is downloaded from Delft University of Technology.

For technical reasons the number of authors shown on this cover page is limited to a maximum of 10.
 1

The Adoption of JavaScript Linters in Practice:

A Case Study on ESLint
Kristı́n Fjóla Tómasdóttir, Maurı́cio Aniche, Arie van Deursen
Delft University of Technology - The Netherlands
[email protected], {m.f.aniche, arie.vandeursen}@tudelft.nl

Abstract—A linter is a static analysis tool that warns software devel- how such tools are configured in the wild [9], [3]. Research
opers about possible code errors or violations to coding standards. By already showed that using static analysis tools does not
using such a tool, errors can be surfaced early in the development come without its challenges. They are known to produce
process when they are cheaper to fix. For a linter to be successful, it is a high number of warnings which includes many false
important to understand the needs and challenges of developers when
positives [8], [10]. Moreover, some warnings need not to
using a linter.
In this paper, we examine developers’ perceptions on JavaScript
be relevant for all projects and can therefore be perceived
linters. We study why and how developers use linters along with the as false positives when tools are not configured appropri-
challenges they face while using such tools. For this purpose we perform ately [11], [2], [6].
a case study on ESLint, the most popular JavaScript linter. We collect Most of the current research does not focus on JavaScript,
data with three different methods where we interviewed 15 developers an evergrowing language with a vibrant community.
from well-known open source projects, analyzed over 9,500 ESLint JavaScript has become a very popular programming lan-
configuration files, and surveyed 337 developers from the JavaScript guage in the last years and in fact has been the most
community.
commonly used language on GitHub since 2013 [12]. It is
Our results provide practitioners with reasons for using linters in their
known as the language of the web and has recently also
JavaScript projects as well as several configuration strategies and their
advantages. We also provide a list of linter rules that are often enabled become popular for server side development, serving as
and disabled, which can be interpreted as the most important rules a general-purpose language. A notable characteristic of
to reason about when configuring linters. Finally, we propose several JavaScript is its dynamic nature, which is unlike other pop-
feature suggestions for tool makers and future work for researchers. ular programming languages such as Java. For example, it
allows for generating new code during program execution,
dynamic typing, and use of undeclared variables.
1 I NTRODUCTION Partly due to its dynamic features, JavaScript is consid-
An important part of software development is to maintain ered an error-prone language [13]. For example, it can be
code by keeping it readable and defect free. This is where easy to introduce unexpected program behavior with simple
static analysis tools can step in: they automatically examine syntactic or spelling mistakes, which can go unnoticed for
source code and look for defects or any issues related to best a long time [14], [15]. A linter can therefore be especially
practices or code style. These tools aid in finding issues and useful for JavaScript to detect these types of mistakes. Ad-
refactoring opportunities early in the software development ditionally, as JavaScript has become widespread, it becomes
process, when they require less effort and are cheaper to more important to have tool support that aids developers in
fix [1], [2]. Due to their many benefits, static analysis tools keeping JavaScript code maintainable, secure, and correct.
have become commonly used in software development [3]. In recent years, linters have increasingly become commonly
There is an abundance of available static analysis tools, used tools for dynamic languages such as JavaScript [3].
ranging from academic research prototypes to tools widely We thus hypothesize that our current knowledge about how
used in industry. These tools vary in functionality, use di- developers make use of static analysis tools may not directly
verse approaches for static analysis and can be used for dif- apply to the JavaScript ecosystem.
ferent languages [4]. Some tools focus on coding styles, code This study therefore aims at complementing the exist-
smells or general maintainability issues, while others try to ing body of knowledge by understanding why and how
identify faults in code, perhaps examining specific types of developers use static analysis tools in real world JavaScript
defects such as related to security or concurrency [5], [6]. software systems, and to see which challenges they face.
One type of a static analysis tool is a linter, which often uses Furthermore, linters need to be incorporated to the devel-
a relatively simple analysis method to catch non-complex opment process and configured appropriately for a project.
errors and violations of coding standards. This can be done in different ways and can be a demanding
In fact, a good amount of research has already been process when there are many rules to choose from. We in-
conducted on general static analysis tools, including how vestigate what methods developers use to configure linters
developers use and perceive these tools [7], [8], [6] as well as and how they maintain those configurations.
 2

We use a mixed methods research approach which in- which disallows the use of the notorious eval function in
volves collecting a combination of qualitative and quanti- JavaScript [27], and indent (Stylistic Issues), which enforces
tative data [16]. We choose ESLint [17] as a case study as consistent use of indentation. The description of each rule
it is currently the most popular JavaScript linter [18]. First, and category can be found in the tool manual [28]. Develop-
we apply a qualitative method, inspired by Grounded The- ers are required to specify which of these 236 rules should
ory [19], to conduct and analyze interviews with 15 devel- be turned on, or instead use publicly available presets. A
opers from reputable open source software (OSS) projects. preset is a set of rules that are made available to the public,
These developers were identified to be actively involved such as the ones from Airbnb [29], Standard [30], or even
with enabling and configuring ESLint. Next we perform ESLint’s recommended settings [28].
a quantitative analysis on the usage and configurations of ESLint does not come with a configuration that is en-
ESLint in 9,548 JavaScript projects on GitHub. Finally, to abled by default, and is instead extremely customizable [31].
challenge and generalize the previous two analyses, we sur- There are several different ways to configure rules for
vey 337 developers from the JavaScript community. We ask ESLint: 1) specifying individual rules in a configuration
them about their experiences and perceptions with using file, 2) specifying individual rules with comments in a file
linters, employing the previously acquired knowledge as containing code, 3) using a preset, and 4) using a plugin
input to a questionnaire. (additional rules that can be created by any developer and
This paper extends our previous work “Why and How plugged directly into ESLint).
JavaScript Developers Use Linters” that appeared at the When specifying an individual rule, it is possible to
32nd IEEE/ACM International Conference on Automated disable a rule (with the settings off or 0) or to enable it,
Software Engineering (ASE), 2017 [20]. Our previous work either as a warning (warn or 1) or as an error (error or 2).
included a qualitative analysis of interviews with JavaScript The rule is turned off when applying off and will have no
developers, which is now extended with an extensive anal- effect whatsoever3 . When a rule is set to warn, each instance
ysis of linter configurations in OSS projects, along with a of the rule being broken will appear in the output when
survey distributed in the JavaScript community. running the linter. Lastly, error will have the same effect as
The main contributions of this paper are: warn except it gives an exit code of 1, meaning it can break
• Perceptions on the usage of linters, after interviewing the build when ESLint is a part of the build process.
15 developers that have actively used and configured a Some rules have more detailed options that further
linter in reputable OSS projects. specify how the rule is applied. As an example, the rule
• An extensive analysis of linter configurations in over indent can have the setting tab to enforce the usage of
9,500 JavaScript projects on GitHub, shedding light on tabs for indentation, or the setting 2 to enforce using two
the most common linter configuration patterns. spaces. Even further customization is possible with this
• A study on the experiences and perceptions of 337 rule by enforcing explicit indentation settings for different
JavaScript developers on linters and specific ESLint statements and expressions, such as switch statements or
rules, via a survey distributed in the JavaScript com- variable declarations.
munity. We chose ESLint as the linter to be analyzed in this study
as it is the most commonly used linter in the JavaScript
community, with over 72M downloads in npm (JSHint,
2 BACKGROUND : L INTERS FOR J AVA S CRIPT
the second most popular, has approximately 56.5M down-
Well known and much researched static analysis tools in- loads4 ) [18]. Of all JavaScript linters, ESLint also has the
clude FindBugs [21], CheckStyle [22] and PMD [23]. These most active community around it where it has the most
tools all have a different focus. For example, FindBugs de- contributors on GitHub, highest number of commits and
tects numerous defects, such as infinite recursive loops and frequent releases. In addition, it offers the greatest amount
references to null pointers, CheckStyle is focused towards of functionality and flexibility out of all well known lin-
following coding standards, and PMD detects both code ters, thus not excluding nor focusing on any specific type
style violations and possible defects. JavaScript linters work of linting such as only analyzing styling issues or solely
in a similar fashion where the best known and most popular identifying possible errors.
ones include ESLint, JSHint [24], JSCS1 [25] and, the first
linter created for JavaScript, JSLint [26].
ESLint is the newest of these and has gained much 3 M ETHODOLOGY
popularity in the last two years in the JavaScript community. The goal of this study is to understand how developers use
ESLint was designed to be an extremely flexible linter that JavaScript linters in real world software systems as well as
is both easily customizable and pluggable. ESLint provides the challenges that they face. To that aim, we propose the
236 base rules2 , grouped in seven categories designed to following research questions:
help developers understand their purpose: Possible Errors, • RQ1 : Why do JavaScript developers use linters?
Best Practices, Strict Mode, Variables, Node.js & CommonJS, • RQ2 : Which linter configuration strategies do develop-
Stylistic Issues, and ECMAScript 6. In Table 1, we provide ers adopt in their projects?
the description and the number of rules in each of these • RQ3 : What linter rules do developers commonly enable
categories. Example rules include no-eval (Possible Errors), and disable?
1. JSCS is no longer supported and the maintainers have joined forces 3. Disabling rules often occurs when projects make use of a pre-
with ESLint as of April 2016. defined preset and developers want to disable some of its rules.
2. As of release v3.13.0 in January 2017. 4. Measured in May 2017.
 3

Category Description Available rules

Possible Errors Possible syntax or logic errors in JavaScript code 31
Best Practices Better ways of doing things to avoid various problems 69
Strict Mode Strict mode directives 1
Variables Rules that relate to variable declarations 12
Node.js and CommonJS For code running in Node.js, or in browsers with CommonJS 10
Stylistic Issues Stylistic guidelines where rules can be subjective 81
ECMAScript 6 Rules for new features of ES6 (ES2015) 32
Total 236

TABLE 1: ESLint rule categories with ordering and descriptions from the ESLint documentation [28]

• RQ4 : What are the challenges in using a JavaScript maintain it?. Other questions were more specific, such as:
linter? Do you experience false positives? if so, which?. The complete
To answer these research questions, we perform three list of questions is available in the paper appendix and in
different steps that involve three different sources of in- the extended online version [35].
formation: first, we interview developers with the goal of Interviewees were asked to participate in an online video
understanding why and how they use linters as well as call. The interviews were recorded with permission and
the challenges they face; next, we mine existing JavaScript lasted from 16 to 60 minutes, with an average duration of
open source repositories to analyze their linter configura- 35 minutes. Three out of the 15 participants were not able
tions; finally, to generalize the results obtained in the inter- to participate in an online call and instead received a list of
views, and to further explain what we observed in GitHub questions via e-mail and provided written responses.
repositories, we survey developers from different JavaScript
communities. Each step is described in the following sub- 3.1.2 Analysis
sections.
Continuously after each interview, memoing was conducted
to note down ideas and to identify possible categories in the
3.1 Part I. Interviewing JavaScript Developers data. The interview recordings were then ultimately man-
To answer RQs 1, 2, and 4, we followed a qualitative ually transcribed. First, we performed open coding where
research approach in our study [16], inspired by many the transcripts were broken up into related sentences and
concepts of classic Grounded Theory [19], [32] where the grouped together into the three main topics that drove
aim is to discover new ideas emerging from data instead our interviews (why and how developers use linters and
of testing any preconceived hypotheses. We also followed the challenges they face). Secondly, we performed selective
Stol et al.’s guidelines [33] that were derived after a system- coding where more detailed categories were identified which
atic literature review on the usage of Grounded Theory in became the topics we present in the Results (Section 4). In
Software Engineering. this process we took advantage of the memos that had been
With an open mind we wanted to understand how written over the course of conducting the interviews. The
and why developers use a linter for JavaScript. For that complete list of codes can be found in the appendix [35].
purpose we collected data by conducting 15 interviews with
developers from reputable JavaScript projects on GitHub. 3.1.3 Interview Participants
We explain the process of conducting these interviews in In order to find potential participants for the interviews we
Section 3.1.1. The interview recordings were manually tran- examined the most popular JavaScript projects on GitHub,
scribed and analyzed with continuous memoing and coding, according to their number of stars in December, 2016. We
which is further described in Section 3.1.2. Finally, we detail conjecture that by observing the top projects on GitHub we
our participants in Section 3.1.3. can obtain an insight into active and reputable projects with
many contributors, providing more interesting and relevant
3.1.1 Interview Procedure and Design information about the usage of linters. We detected projects
The interviews were conducted in a semi-structured fash- that 1) use ESLint, 2) have some custom configurations
ion as it is commonly done in software engineering re- (e.g., not only using a single preset) and 3) where one or
search [34]. With this method, specific questions are com- two contributors could be identified that have been more
bined with open-ended questions to also allow for un- involved than others in editing the configuration file. We
expected information in responses. Hove and Anda [34] then sent an e-mail to one or two main contributors of
encourage interviewers to talk freely, to ask relevant and the ESLint configuration file of the corresponding project,
insightful questions and to follow up and explore interesting briefly explaining the purpose of the study and asking for
topics. These guidelines were followed while performing a short interview. These requests were sent out in batches
the interviews. Each interview was built upon a list of 13 of 5-10 e-mails (starting with the most popular projects)
standard questions. as it was difficult to predict how many positive replies we
To begin with, participants were asked broad questions would receive. Batches were sent out until we had received
which often provided an opportunity for further discussion. a sufficient number of positive replies back, where the goal
Example questions include: Why do you use a linter in your was to perform at least 10 interviews, or until we were
project? and How do you create your configuration file and satisfied with the amount of information we had collected.
 4

TABLE 2: All participants’ codenames, number of months whether they rely more on pre-made settings (presets) or
using ESLint in the corresponding OSS project and the range their own configurations, and which types of rules are most
for the project placement in the top 120 JavaScript projects commonly used.
on GitHub
3.2.1 Data Collection
Code Months Placement
To collect projects we chose GitHub as a data source due
P1 25 11-20 to the high number of available JavaScript projects and the
P2 22 11-20
P3 5 21-30
convenience of retrieving the data [36].
P4 14 21-30 The original data selection consists of all JavaScript
P5 8 31-40 projects on GitHub that have at least 10 stars and are not
P6 7 41-50 forks of other projects. The purpose of giving a star to
P7 1 61-70
P8 23 71-80 a project on GitHub is to keep track of projects that a
P9 5 81-90 user finds interesting or simply to show appreciation to a
P10 3 81-90 repository [37]. By only including repositories with at least
P11 4 91-100
P12 16 91-100
10 stars the intent is to analyze “real” software projects.
P13 15 111-120 Kalliamvakou et al. [36] showed that a large portion of
P14 24 111-120 GitHub repositories are not for software development but
P15 22 111-120 for other functions such as for experimental or storage
purposes. It is expected that repositories that were created
TABLE 3: Experience of participants, showing the lowest for experimentation or testing purposes only, or pet projects
and highest answers along with the average of all answers that were started and abandoned, will not receive 10 stars
from other users. Furthermore, forks of other projects were
Low High Average excluded to avoid having duplicate configuration files in
Years as developer 3.5 27 11.8 the dataset. This resulted in 86,366 projects being collected
. Years as JavaScript developer 1.3 20 8.9 to analyze.
Years in project 0.6 5.0 2.7
Project age 1.0 8.0 5.1 To retrieve data on these projects, we used Google Big-
Query [38] on the most recent GHTorrent [39], [40] dataset
on GitHub projects from April 1st, 2017. The precise SQL
A number of 120 projects were eventually examined query that was used to obtain the data can be found in our
where 37 requests were sent out. These resulted in 15 in- online appendix [35].
terviews being performed, thus with a response rate of 40%. After retrieving the 86,366 project entries, additional
The information from these 15 interviews was considered filtering was performed on the dataset. First of all, there
enough to provide us with theoretical saturation [19]. Table 2 were examples of duplicate entries in the dataset where they
shows the developers who participated in the interviews point to the same GitHub API URL for the project. This can
where, in order to keep the participants’ anonymity, they happen when the project name has been modified or when
are given names starting with the letter P and a number the owner has been changed for a repository, in which case a
from 1 to 15. The months each corresponding project had new entry is created in the GHTorrent dataset. The duplicate
used ESLint is also displayed5 , where most projects had entry that had a more recent date for its last commit was
migrated from another linter such as JSHint. The table kept in the analysis. This filtering resulted in 1,596 projects
also shows the placement of the projects in the top 120 being removed from the dataset.
JavaScript projects on GitHub within a range of 10 projects Secondly, even though the query includes a statement to
each (to maintain the participants’ anonymity). Participants not include deleted projects, some repositories could not be
are ordered by the projects’ number of stars on GitHub, and accessed. In seven cases, an HTTP error status code of 451
not by the order we interviewed the developers. A summary (unavailable for legal reasons) was returned when trying to
of the participants’ experience is shown in Table 3 where access the repository. More commonly, or in 871 cases, the
the average experience as a professional software developer repository’s URL could not be found, returning an HTTP
was 11.8 years. Among the 15 participants, four are founders status code of 404. Due to this filtering, a number of 878
of the project, seven identified themselves as lead or core additional projects could not be analyzed, resulting in a final
developers and four are project maintainers. number of 83,892 possible projects.
Besides removing forked, duplicated and deleted
projects, no additional filters were applied such as regarding
3.2 Part II. Mining Linter Configurations in Open project size or activity. We decided not to filter the projects
Source Systems by a minimum size, as the intention was to analyze all dif-
To answer RQ3 , we performed a quantitative analysis to ferent types of JavaScript projects: big or small, collaborative
know exactly how developers configure their linters and or personal. However it could therefore be the case that
what the most common configuration patterns are. For the resulting dataset includes projects that are not suited to
this purpose, we analyzed 9,548 ESLint configuration files ever use a linter, e.g., a repository that is not a software but
extracted from 86,366 JavaScript projects on GitHub. We an- simply a collection of scripts or even tips for developers.
alyzed how much configurations are applied by developers, Additionally, these could be projects that have not been
active for several years, and perhaps even not active since
5. As of February 2017. ESLint was created in June 2013.
 5

3.2.2 Extracting the Linter Configuration Linter Projects with linter % of all projects
While we only analyze the linter configurations for ESLint, ESLint 9,548 11.4%
JSHint 9,447 11.3%
we note down the usage of other linters as well, namely: Standard 1,651 2.0%
JSHint, JSCS and Standard. For each linter the tool searches JSCS 1,578 1.9%
for a configuration file with a specific known name and
Total 20,292 24.2%
file ending. The configuration file is typically located in
the main directory of a project (as it will then be used for TABLE 4: Estimation of the number of projects using ESLint,
the whole project), so in order to save execution time and JSHint, Standard and JSCS in our dataset
to simplify the tool, it is the only location where the tool
searches for the file. Linters used Number of projects % of all projects
If a configuration file (either .eslintrc or package.json) is
1 18,450 22.0%
found for ESLint then it is retrieved. For the other linters the 2 1,753 2.1%
tool merely notes down their presence in order to measure 3 88 0.1%
their prevalence. 4 1 0.0%
With the configuration file in hand, our tool extracts Total 20,292 24.2%
all the relevant information about enabled and disabled
rules. We make the tool available for download in our TABLE 5: Estimation of the number of projects using multi-
appendix [35]. ple linters in our dataset

3.2.3 Dataset Characteristics

The sizes of the projects in GHTorrent are expressed in configuration files on GitHub, we surveyed 337 developers
kilobytes (KB) where the first, second and third quartiles from the JavaScript community about their perceptions and
are the following: 126 KB ; 364 KB ; 1,928 KB. Moreover, experiences with using linters. The survey was built upon
55,9% of the projects are 500 KB or smaller. the previously acquired knowledge where the information
was used as input to both open and closed questions.
For the last commit date of the projects, 7.4% of the
projects had their last commit before ESLint was first re-
leased (v0.0.2) in June 2013 and 35.4% of the projects before 3.3.1 Survey Design
the first major release (v1.0.0) in July 20156 . Thus when an- This survey was constructed under guidelines from both
alyzing the prevalence of linter usage in JavaScript projects social sciences and software engineering research, e.g., from
or even more specifically, ESLint usage, some projects in the Fink [45], De Vaus [46] and Kitchenham [47]. A shortened
dataset are less likely to have used a linter. Moreover, when version of the survey is present in the paper appendix and
manually inspecting the dataset, we found examples of pro- a full PDF version is available online [35].
gramming guides (example at [43]) and tutorials (example at We mainly took advantage of closed questions to make
[44]). Nevertheless, these types of projects were not filtered the survey more reliable and more compelling for partic-
out as we wanted to avoid making any assumptions about ipants to complete. To minimize the risk of forcing opin-
the dataset and for it to be as broad and general as possible. ions on participants, each closed question, where possible,
Table 4 shows the estimation of the usage of four linters contained an option where the user could write his or her
in our dataset, where ESLint was the most commonly used own response, along with having a neutral option in ordinal
linter followed by JSHint. We observe a similar pattern for questions. Moreover, to minimize the risk of bias due to the
npm, where ESLint is also the most downloaded linter, order of options in the survey, the options were randomly
followed by JSHint, but with a greater difference [18]. In shuffled for each participant, wherever possible. To choose
total 20,292 out of the 83,892 of the analyzed projects, or the options for the closed questions, we used input from the
24.2%, used a linter. We also show the number of projects results of the two previous data sources.
that use more than one linter in Table 5. Only 9.1% of the Early on in the survey, participants were asked if they
projects that use a linter, use two or more linters, where 4.3% had ever used a linter in a project. If the answer to that
use ESLint and another linter. The percentage of projects question was negative, they were only presented with two
using any of these linters appears to be higher for those
that have more stars. Table 6 shows the number of projects
Top projects Projects with linter % of top projects
that use any of these linters while observing only the top
starred projects, such as the top 10 or top 1,000 projects. 10 7 70.0%
100 65 65.0%
The percentage of projects using a linter steadily decreases 300 185 61.7%
when observing more projects, going from 70.0% for the top 1,000 535 53.5%
10 projects to 28.5% for the top 30,000 projects. 3,000 1,371 45.7%
5,000 2,082 41.6%
10,000 3,675 36.8%
3.3 Part III. Surveying Developers 20,000 6,306 31.5%
30,000 8,554 28.5%
To generalize the findings we obtained from the interviews Total 20,292 24.2%
and to further explain the results we obtained after mining
TABLE 6: Estimation of the number of projects using a linter
6. The exact dates that were used for this comparison were 08:00 June
30th 2013 [41] and 08:00 July 15th 2015 [42]. in our dataset
 6

additional questions, asking how important they consider a

linter and why they have never used a linter. Other partici-
JS Reddit 268
pants received questions on the usage of linters, including a
question asking which linters they had used. If a participant
had not used ESLint in any project, he or she was not
presented with questions that had specifically to do with Echo JS 44
ESLint rules. Instead they (and also those that had used
ESLint) received more generic questions about why they use
a linter, what methods they use to configure it and which JS.is 15
challenges they have faced.
One of the main goals of the survey is to collect the
perceptions of participants on the importance of individual Twitter 10
ESLint rules. For this we derive a set of 14 rules in the
four most important ESLint categories, as identified by the 0 100 200 300
interviewees and in project configuration files, except for the
Variables category where there are only 12 available rules. Fig. 1: Number of full responses from each location where
These sets were created with the following criteria: First, the survey was distributed.
we select the five most commonly enabled rules where a
preset is not used (according to our previous analysis) and
five where a preset is also used. Some rules can appear in 1) JS.is. An Icelandic JavaScript user group on Facebook
both these sets where additional rules are then chosen. If an with 789 members [48]7 .
even number of additional rules were needed, one rule was 2) JS Reddit. A “subreddit” about JavaScript on the popu-
added from each list of commonly enabled rules (e.g., the lar community-driven social news site reddit.com with
sixth top enabled rule without a preset and also the sixth 111,649 subscribers [49].
top enabled rule with using a preset). If an odd number of 3) Echo JS. A community-driven news site about
additional rules were needed, the last chosen rule is the next JavaScript development [50]8 .
rule of either set that has been more commonly enabled. The 4) Twitter. A news and social networking site [51].
same process was applied to select the remaining four rules,
except this time observing the most commonly disabled Distributing the survey in these four locations resulted
rules, consisting of two rules where a preset is used and two in a total number of 337 completed responses, as shown
where no preset is used. Like with many other questions, the in Figure 1. Furthermore, additional 476 partial responses
order of the rules was randomized, along with the order of were received, which we did not include in the analysis.
the questions themselves as they appeared for each of the The completion rate is therefore 42.0%. The survey was first
four categories. posted on the web on June 1st and eventually closed on June
Before publishing the survey we performed pilot studies, 14th, thus being available for 14 days.
but rather in the nature of moderating focus groups. More
specifically, six participants with different characteristics
3.3.3 Analysis of the Survey Results
were recruited to take the survey in individual sessions with
the first author. The pilot participants were colleagues of We apply descriptive statistics to report the findings of the
the authors that were known to have used JavaScript, with closed questions. For the open questions, further manual
different levels of experience. The participants were asked analysis was needed. There are eight questions in the survey
to read each question out loud and to express their thoughts that are completely open (essay questions) but many more
and understanding of the text. They were especially encour- that are closed but additionally contain an open answer
aged to speak out if they thought any questions were un- possibility to add an extra option. Similarly as when pro-
clear or vague and to express whether answer options were cessing the interviews, an inductive method [45] was used
appropriate. Furthermore, before answering the questions, where the main themes of each question were identified as
the pilot participants were asked to read the introductory they emerged when processing the answers. Each identified
text to evaluate how appropriate and motivating it was. At category received a code and each answer was labeled with
the end, we discarded the data generated in the pilot studies. at least one code. When needed, the coding was conducted
on different levels of detail, where broad categories were
3.3.2 Target Population identified which were then sub-classed into more detailed
The survey was specifically directed towards JavaScript ones [46]. In cases where answers described more than one
developers that have used a linter, and in particular those item, e.g., listing several different reasons for something, it
that have used ESLint, as they can answer more of the was decided to give multiple codes instead of choosing only
survey’s questions and have more knowledge on the topic. one and then possibly disregarding parts of an answer. The
We applied convenience sampling where the survey was derived analysis can be found in our online appendix [35].
advertised in several places on the web where it was likely
to find members of the target population. More specifically,
7. All information about the four resources was collected on June 14th
the survey was distributed in the following four locations 2017.
(the references point to the direct post where the survey 8. This reference does not contain a direct link to the promotion post
was promoted): as it is not made accessible by the Echo JS website.
 7

United States 38.4% Developer 86.1%

Germany 6.9%
Team leader 6.3%
United Kingdom 6.3%
Student 4.8%
Canada 6.0%
Manager 1.2%
Iceland 4.2%
Project manager 0.6%
Netherlands 4.2%
Other 0.6%
France 3.3%

All others 30.3% Designer 0.3%

0 10 20 30 40 50 0 30 60 90

Fig. 2: The country of residence of participants Fig. 3: The primary roles of participants in software devel-
opment

3.3.4 Participants’ Characterization

the proposed reasons aforementioned9 . Sections are ordered
Nearly all of the survey’s participants are male, or 96.4%,
by their importance, according to the survey participants.
with only four female participants, accounting for 1.2%. One
participant chose Gender Variant/Non-conforming while
4.1.1 Maintain Code Consistency
the rest preferred not to answer the question. The respon-
dents’ country of residence covers 53 different countries Every single interview participant mentioned that one of
where more than half of all participants, or 69.7%, come the reasons why they use a linter is to maintain code consis-
from seven dominating countries as shown in Figure 2, most tency. Moreover, 97% of our survey respondents also agree
commonly from the United States. or strongly agree, and this was in fact the most agreed topic
A large majority of the participants identified their pri- in the entire survey (Figure 5, Maintain code consistency).
mary role in software development as a developer, or 86.1%, Having a consistent code style in a project is beneficial
as shown in Figure 3. Other noticeable roles are team leader for many reasons, one being that it improves the readability
(6.3%) and student (4.8%). and understandability of the code. As an example, P10
The experience of the participants in software develop- reported that inconsistent code, such as having different
ment and with JavaScript is shown in Figure 4. The average spaces and semi colons, makes the code very difficult to read
experience of the participants in software development is and understand since in those cases these inconsistencies
8.0 years, and their average experience in working with consume all his attention. This might even be especially
JavaScript is 5.8 years. All participants claimed to have used relevant in the case of JavaScript since it is a language where
JavaScript in the last year and 96.1% reported that it was the developer has substantial freedom in how to write the
one of their main programming languages. code (P12, P14): “With JavaScript you can write code in many
ways, and it can be hard to read other people’s code if you write it
The majority regularly works with commercial software
in a different way.” (P12).
(87.0%) while around half of the participants work with
This topic relates mostly to the category Stylistic Issues
open source software (50.6%) (some participants chose both
where there are many different rules available to enforce
options). Finally, 93.7% claimed to have used a linter in
specific code styles. Even though every participant men-
a JavaScript project. We thus consider that our survey
tioned this matter in the interviews, it does not seem to
achieved our targeted population. The 21 participants that
be of high priority for them. When participants were asked
had never used a linter were not presented with the greater
which category of rules they thought were the most and
part of the rest of the survey.
least important ones, two considered Stylistic Issues to be
the most important category while 10 thought it to be the
least important one.
4 R ESULTS
Some participants were bothered by the fact that choos-
In the following we present our results on why and how ing which style to follow is a very subjective decision and
JavaScript developers use linters, along with the challenges developers generally have very different opinions on how
that they face. code should be written (P1, P3, P5): “Stylistic Issues - they’re
all opinion based.” (P5). On the other hand, four participants
explained that Stylistic Issues was indeed the least important
4.1 RQ1 . Why Do JavaScript Developers Use Linters? category simply because other categories can catch bugs
We first introduce the six reasons to why JavaScript devel- which is far more important (P2, P4, P9, P13). This category
opers use linters, which we derived from the 15 interviews.
Each of the following sub-sections represent the reasons that 9. The reasons presented in Figure 5 have been shortened to fit in the
figure, and two of those reasons have been combined into one section in
emerged from our qualitative analysis. In addition, in Fig- the text (namely, Automate Code Review and Avoid Giving Negative
ure 5, we present the survey participants’ agreements with Comments).
 8

39 49
40 50 47
48

35
45
35
32
31
40
Number of participants

Number of participants
37
30
26 35
25 23
22 22 30
26
24
20 18 25
19
14 20 18
15
15
11 14
10 15
10 8
10 8
5 5 5
4 4 5
5 4
2 2 2 2 2 2 5 2 2
3
2
3
1 1 1 1 1 1
0 0
−2 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 −2 0 2 4 6 8 10 12 14 16 18 20 22 24
Years of experience Years of experience
(a) Experience in software development (b) Experience with using JavaScript

Fig. 4: Participants’ experience in software development and with JavaScript. Axes show years of experience and number
of participants with the corresponding answer.

More explicitly, when asked about the most important

Maintain code consistency 2% 2% 97% category of warnings in ESLint, 10 interviewees answered
that Possible Errors was the most important one (P1, P2, P3,
Prevent errors 3% 4% 93%
P4, P5, P7, P8, P9, P11, P12): “Possible Errors is the #1 most
important, the biggest reason to use a linter is to catch errors
Save discussion time 6% 12% 82%
the programmer missed, before they become a runtime bug.” (P9).
Avoid complex code 13% 20% 68% These rules can be especially useful for bugs that are hard
to find and to debug (P15).
Automate code review 12% 24% 64% A special category of bugs in JavaScript has to do with
the declaration of variables because of the dynamic nature
Learn about JS 42% 28% 31%
of the language. Two of our interview participants reported
that Variables was the most important category (P10, P15).
Avoid negative comments 39% 35% 27%
When a developer e.g., mistypes a variable or uses the
100 50 0 50 100 wrong variable name, the linter can catch it and warn the
developer: “It’s very easy to write JavaScript code that has errors,
Fig. 5: Level of agreement with reasons as to why the partic- you might use a variable that hasn’t been declared or you might
ipants have used a linter. Bars show Strongly Disagree, Dis- have a typo in your variable name and because JavaScript is often
agree, Neutral/Not Applicable, Agree, and Strongly Agree, not compiled, you’ll only discover that much later when you run
respectively. (N = 309) the code.” (P1).
Nine participants (P1, P3, P4, P5, P8, P10, P12, P13,
P15) mentioned the importance of the rule no-unused-vars
thus still provides a lot of value and they would not want to
(identifies variables that have been declared but never used)
omit it: “They make [the code] harder to read but they just don’t
and five (P1, P3, P5, P13, P15) mentioned no-undef (identifies
cause issues as much.” (P13).
variables that have not been defined) which are both useful
to identify mistyped variables. In addition, no-dupe-keys and
4.1.2 Prevent Errors and Augment Test Suites no-dupe-args, two duplication rules for keys in objects and
Using a dynamic language such as JavaScript is not free names in function parameters, were also mentioned by P3,
of risks: “Without a linter [JavaScript] is a very dangerous P4, P8, and P15.
language. It’s very easy to make a very big problem and then spend While linters are being used to catch errors in code, there
30 minutes to find it.” (P7). The majority of the interview is another popular and widely accepted method to catch
participants reported that the number one reason as to why bugs which is to write unit tests. It is therefore interesting
they use a linter is to catch possible errors in their code: to know how these two methods are combined for this
“There are things which are easy mistakes to make and are obvious purpose. Some participants mentioned that they use a linter
errors and I think those provide the highest value. Because you on top of unit testing as a complementary approach to the
have a one to one correspondence between times that a rule catches regular tests (P1, P3, P8). P1 and P8 pointed out that unit
something and bugs that you’ve prevented.” (P4). This is also tests commonly do not cover all code, which can result in
inline with the results of our survey, as 93% of participants problems being easily missed: “You need to seek all possible
agree or strongly agree that this is an important category of cases for unit tests, but sometimes it’s very hard, and of course in
issues (Figure 5, Prevent errors). all projects, unit tests don’t cover all possible cases. So this is why
 9

a linter is a second protection line.” (P8). Furthermore, the tests 4.1.5 Automate Code Review
can also take substantial time to run and thus the linter can Several participants mentioned that they use the linter to
be seen as a much faster version of smaller subtests (P1). avoid having to manually review the code style in pull
On the other hand, participant P4 believes that unit requests (P1, P2, P3, P4, P8, P11, P14, P15). Furthermore,
tests and manual tests can usually cover all errors, so it saves time for the contributor of the pull request since he
even though ESLint would not be used, the errors would or she receives much faster feedback from a linter than from
eventually be caught by the various tests that are applied. a person that would conduct a review (P4). In the survey,
However, P4 says that the linter can catch them earlier in we see that 64% of participants agree or strongly agree with
the process and is also better at identifying code that is the usage of linters for improving code reviews. However,
ambiguous. only 12% disagree or strongly disagree with it (Figure 5,
Automate code review).
4.1.3 Save Discussion Time
Maintaining code consistency with a linter can also make
Having a set of rules regarding code style can also save time
pull requests much easier to review. When there is a set of
that is spent on discussing different styles and opinions (P2,
stylistic rules in a project to which everyone has to conform,
P4, P5, P6, P7, P10). 82% of our survey participants also
all pull requests have minimal stylistic changes. If there are
agree or strongly agree (Figure 5, Save time on discussions).
no rules, there can be multiple code changes of e.g., only
In big projects with many contributors there can be
whitespaces or line formatting which might be caused by
many pull requests in circuit and discussions can occur
different editors being used. This can make it difficult to see
where developers disagree on a certain style that is used. P2
the actual changes that were done in the contribution since
explains that discussing code styles is not worth the effort
they are hidden by these formatting changes (P3, P12).
when there are other more important things to discuss. He
In addition, when receiving comments from a code re-
further describes that comments regarding code style on
view, developers can sometimes be sensitive to criticism
pull requests can be different depending on which devel-
(P2, P8, P11). This can particularly be the case for new
oper is conducting the review. In some cases, contributors
developers: “If you tell to a new developer that he or she made a
can therefore receive contradictory advice if no rules exist
mistake, it will be very sensitive. He may feel very uncomfortable
that everyone goes by.
because somebody found a mistake in his work. But if a linter tells
The discussions about code style that can occur in pull
you about a mistake, it is not a problem, because it’s not personal.”
requests or in issues can also even lead to arguments be-
(P8). A new developer might also look up to the person
tween people since developers have very different opinions
that is conducting the code review which can make the
on the matter (P1, P2, P3, P5). All this can be avoided by
criticism especially dispiriting (P2). Having a linter doing
deciding upon a set of rules to begin with: “It’s almost like a
this job can also contribute to people feeling more equal in
code contract. There may be things that each of you have assumed
a project if there is no senior person telling others to do
and you don’t know what your own assumptions are, and what
things differently (P11). However, this does not seem to be a
could possibly lead to conflict down the road, so you have a written
common reason for using a linter as only 27% of the survey
contract to try to address everything up front.” (P7).
participants agree or strongly agree with using a linter as a
4.1.4 Avoid Ambiguous or Complex Code way to avoid giving negative comments to other developers
(Figure 5, Avoid giving negative comments).
It can be difficult to understand code correctly where the
intention is not perfectly clear. The category Best Practices
tries to tackle this problem where, according to its docu- 4.1.6 Learn About JavaScript
mentation [17], it contains rules that relate to better ways ESLint can be used to learn about new JavaScript features
of doing things to help developers avoid problems. While or new syntax. P12 used ESLint in helping him to learn the
only one participant recognized this category to be the most new syntax of ECMAScript 6 (ES6): “When I switched to ES6, I
important one (P6), others identified it as the second most used it as an educational tool. It’s so easy to continue to use var for
important (P4, P8, P13, P15). In our survey, 68% of partici- variable declarations. I used ESLint very strictly to enforce ES6
pants agree or strongly agree that avoiding ambiguous and syntax, otherwise I would probably still use ES5 when I write
complex code is important, while 20% were neutral, and code. But with the help of the linter it forces you to switch to ES6,
13% disagree or strongly disagree (Figure 5, Avoid complex which is a good habit.” (P12).
code). Even though linters can be beneficial to all JavaScript
Some of these rules try to prevent code from being developers, they can be especially helpful for new devel-
misunderstood: “It helps enforce code which says what it does, opers, either those who are new to a project or those who
so that it’s easy to understand.” (P4). In some cases code is are new to programming in general (P6, P7, P9, P13, P14).
actually doing something else than it appears to and a linter Contributors in OSS projects usually have different levels of
can help to detect these situations (P2, P4, P6). For exem- experience and using a linter can help with “leveling the play-
plary beneficial rules, participants mentioned restricting the ing field and helping people to understand what’s actually going
usage of switch statements by forbidding the use of ”fall on” (P13). This particular example came from a developer
throughs”10 (P4), and disallowing unreachable code (P2, P3, that had been working with students who were accustomed
P6, P15). to getting errors from the Java compiler, telling them what
they can and can not do. However when using JavaScript,
10. A ”fall through” occurs when all statements after a matching
case are executed until a break statement is reached, regardless if the one can run code that includes various coding mistakes and
subsequent cases match the expression or not. not get notified about it (P13).
 10

Interestingly, our survey participants did not fully agree

Use an existing preset 70.2%
with using linters as a way to learn JavaScript: only 31%
of them agree or strongly agree, whereas 42% disagree or Project fit 56.0%
strongly disagree with it (Figure 5, Learn about JS). Default configurations 51.1%
Pull Request discussions 37.9%
4.1.7 Why Not to Use a Linter?
Most used style within team 33.0%
There were 21 survey participants that had never used
a linter in a JavaScript project. These participants only Minimal configurations 21.0%
received two additional questions in the survey, including Effort of enforcing a rule 19.1%
why they do not use a linter. Three participants explained Consistent rules 10.4%
that they simply do not need to use a linter, e.g., as they
Automatically generated 7.4%
already use an editor formatter or rather rely on TypeScript
for typing analysis and on team discipline and code reviews I never configured a linter 1.3%
for code quality. Two participants did not find it sufficiently 0 20 40 60 80
beneficial to use a linter since it can complicate the build
process, can require demanding configuration, and it can Fig. 6: Strategies used by developers to configure a linter
increase the cost of the development life cycle. Two others (N = 309). The full survey question can be found in our ap-
did not have sufficient knowledge about linters, one was pendix. Number in the bars are represented as percentages.
not in charge of the repository settings and three simply did
not know why they had never used a linter. Other reasons default configurations or presets to be carefully thought, as
given once were: there is too much effort involved in setting they are highly adopted by practitioners.
up a linter in a big project, it is difficult to integrate the linter
with some IDEs and that other tasks have taken precedence 4.2.2 Project Fit
before setting up a linter. It is important that the stylistic and best practice rules fit the
existing code when the rules are chosen (P3, P4, P6, P12): “I
wanted them to fit the code as it was, I wanted the linting in place
RQ1 : JavaScript developers use linters for the follow-
with as little mess as possible.” (P12).
ing reasons: to maintain code consistency, to prevent
According to P4, if there is already some sense of style
errors and augment test suites, to save discussion
in the existing code, it is not very sensible to change it to
time about which style to use, to avoid ambiguous
something else since it would create more work than nec-
and complex code, and to automate code reviews.
essary when setting up a linter. P4 also considers whether
a particular rule will be useful for the project or if it will
need to be disabled or overridden in multiple locations in
4.2 RQ2 . Which Linter Configuration Strategies Do De- the code. If it needs to be disabled frequently, it is not worth
velopers Adopt in Their Projects? it to enforce it.
In our survey, we also observed that choosing rules that
In the following we present eight strategies that developers
fit the current project is a common configuration strategy, as
use to configure their linters, which emerged from our
56% of our participants affirmed to use it.
qualitative analysis by conducting the interviews. These
strategies were presented to our survey participants who 4.2.3 Pull Request Discussions
could select all methods they had ever used, and also had Three participants (P2, P4, P10) reported that when some-
the chance to add their own methods to the list. The follow- thing is discussed in a pull request that can be enforced with
ing sections are ordered by their importance, according to a rule, they use the opportunity to enable the corresponding
the survey participants, which is also displayed in Figure 6. rule. According to P2, since the topic surfaced in the pull
request, then there was obviously a need to make a decision.
4.2.1 Use an Existing Preset That way, the topic will not surface again and time will not
There are many publicly available presets that anyone can be spent on discussing it (P2).
use in a project instead of creating a custom configuration, In the survey, 37.9% of the participants affirm to use
or that one can use as a part of a custom configuration. team discussions as input to choose which rules to enable
Many of these presets have been carefully constructed by or disable.
their creators and have been changed attentively over time:
“They thought about the code standard quite extensively and put 4.2.4 Most Used Style Within Team
a lot of thought in it.” (P12). Several interviewees use a preset P7 reported that in a new project he would most likely go
as a part of their configuration file (P1, P6, P10, P12, P13, with the code style that is the most common one amongst
P15) and one participant normally tries to solely use the the developers in the team, consequently adding rules that
preset as is (P8). enforce that style. Generally when working with a new
A large portion of our survey participants (70.2%) also team, the first discussion he often has with the team mem-
use an existing preset. We also took the opportunity to bers is regarding which code style people are used to.
ask our survey participants about how often they use the 33% of our survey participants also affirm to use the
default configurations that come with the linter. 51.1% of style that is the most commonly preferred one amongst
them affirm to do so. This highlights the importance of such developers of a team.
 11

4.2.5 Minimal Configurations 4.2.9 Additional Methods

Some prefer to keep the configuration as simple and mini- Survey participants mentioned 15 additional methods,
malistic as possible (P1, P5, P8, P15): “We don’t want people to where three participants claimed to configure the linter
feel like they have to jump through unnecessary hoops to get their according to their personal preferences. Another three par-
PR’s in, so turning on every single thing wouldn’t be great.” (P1). ticipants use a company style guide, where one explained
Furthermore, P8 thinks that if too many rules are enabled in that there is a special committee at his workplace which
a project, people will not trust the configurations: “They will maintains JavaScript coding standards for the whole com-
think that it is a bureaucracy and that it’s not important.” (P8). pany. Other strategies include using the strictest setting
Both P1 and P8 prefer to only enable rules that can prevent possible, choosing rules that prevent bugs or to have a big
errors. team discussion on which style to use.
In the survey, we see that 21% of the survey participants
have chosen to also have configurations that are as minimal-
istic as possible. RQ2 : The most common linter configuration strate-
gies are: to use an already existing preset, to choose
4.2.6 Effort of Enforcing a Rule rules that fit the current style of a project, to enable
P15 described that he commonly enables a set of rules, rules that surface in discussions (e.g., on a pull re-
e.g., a known preset, and then sees how it works out for the quest), to use the most commonly preferred coding
project. If some of the rules are starting to be bothersome style within a team, to be as minimalistic as possi-
for the project, e.g., needing to be disabled with inline ble, and to use rules that involve the least effort to
comments or if too much refactoring is required to fit the enforce.
rule, it is permanently disabled: “Just start to use it and see
how much pain it causes, where it’s beneficial. But usually it’s
turning things off when it’s apparent that it’s creating more effort 4.3 RQ3 . What Linter Rules Do Developers Commonly
than it really helps.” (P15). He also describes the process as a Enable and Disable?
feedback cycle where it is important that contributors agree
on the rules that are used: “The disagreement between people is As stated before, ESLint does not come with default con-
very important, you have to get everyone on the same page.”. figurations. It therefore has to be configured in some way,
Although this was only mentioned by one interviewee, most commonly by specifying a configuration file. This
enabling rules that involve less effort to follow is used as a configuration can include presets, and/or some rules that
strategy by 19.1% of our survey participants. are enabled, disabled or set as warnings. We first discuss
the prevalence of presets, followed by an analysis on the
4.2.7 Consistent Rules prevalence of individual rules that are specified.
To some developers it is not important which exact rules Throughout the results, we separate the results between
are actually enabled (P2, P6, P10): “I almost don’t even care projects that make use of a preset as basis and projects that
what the rules are, I have some opinions, but I’m much more do not make use of a preset as basis. We conjecture their
interested in there just being consistent rules, than having a point behavior are different: when not using a preset, developers
of view about any particular rule.” (P10). There simply has need to think about which rules to enable and which rules
to be some fixed set of rules to enforce consistency and not to enable. When using a preset, developers already start
to prevent unnecessary time being spent on discussions: with a set of enabled (and, as a consequence, disabled) rules;
“Having a linter forces us to make choices, even if it’s arbitrary however, they may disagree with some of the decisions of
choices in some situations.” (P2). the preset and, in this case, they have to overrule what was
Interestingly, only 10.4% of our survey participants af- there (in other words, developers make a decision to go
firm that they do not care about which rules are there, as “against” the preset). Thus, in our results, we control this
long as they are consistent. We see this as evidence that possible factor of influence.
choosing the right rules matters to most developers.
4.3.1 Presets
4.2.8 Automatically Generated Out of the 9,548 projects that were analyzed, 6,413 or 67.2%
ESLint provides an automatic method to ensure that a used a preset in their configurations, with a total of 6,967
configuration fits well to a specific project, based on its presets being used (some projects using more than one pre-
source code in addition to a small questionnaire to the set). This result is inline with the results of our survey, where
developer. Two interviewees used this method to create 70.2% of participants affirm to use presets as a configuration
their configuration file (P11, P14). P11 then reviewed the strategy (Figure 6).
generated rules and the errors in the output to see if he The 10 most popular presets are displayed in Table 7. It is
agreed with them. In general, he does not consider it wise evident that only a handful of presets are extremely popular
to use a preset since linter configurations are generally very amongst these projects. Only the five most popular presets
project dependent: “I didn’t even consider Airbnb or Google account for 67.2% of all presets that are used. The recom-
because I think every project is a little different.” (P11). mended setting from ESLint [31] is the most popular preset,
This strategy’s lack of popularity among our intervie- followed by the Airbnb preset [52]. It is interesting that in
wees is also observed in our survey, where only 7.4% of the top five places, there are two presets by Airbnb: the
participants affirmed to use it (making it the least frequent base version (airbnb-base [53]) and the extended version that
strategy amongst our survey participants). includes rules for some external JavaScript libraries (airbnb).
 12

Preset Projects % of all presets Total Mean # %

rules per proj projects projects
eslint:recommended 2,226 32.0%
airbnb 1,166 16.7% Projects w/ preset 43,340 7 3,389 53%
standard 627 9.0% Project w/o preset 115,500 37 2,735 87%
airbnb-base11 557 8.0%
google 104 1.5% (a) Enabled rules
standard-react 81 1.2% Total Mean # %
prettier 58 0.8% rules per proj projects projects
rackt 49 0.7%
react-app 48 0.7% Projects w/ preset 5,159 1 1,125 18%
semistandard 44 0.6% Project w/o preset 20,122 6 1,601 51%
Total 4,960 71.2% (b) Warned rules
Total Mean # %
TABLE 7: The 10 most popular presets, showing the number rules per proj projects projects
of projects using each preset and the percentage of the times
Projects w/ preset 17,263 3 3,395 53%
the preset is used out of all used presets Project w/o preset 45,024 14 2,344 75%

(c) Disabled rules

If these two presets are counted together, they have a total of Total Mean # %
1,723 instances, making airbnb and eslint:recommended (2,226) rules per proj projects projects
by far the most popular ones (56.7% of all presets that are Projects w/ preset 65,762 10 4,465 70%
used). Project w/o preset 180,646 58 2,979 95%

(d) All rules

4.3.2 Frequency of Specified Rules
Here we examine the frequency of rules that are specified TABLE 8: All used rules, divided into the different settings:
in configurations, for each type of rule setting and differen- enabled, disabled and warned rules. Each shows the total
tiating between projects that use presets and those that do number of used rules, the round up arithmetic mean of rules
not. The quantity of rules that are configured are displayed used per project, the number of projects that use at least one
in Tables 8a, 8b, 8c, and 8d. rule, and the percentage of projects that use at least one rule.
First of all, it is evident that the warning mechanism that Total number of projects = 9,548, total number of projects
is provided by the tool is not often used, as we observe with a preset = 6,413, total number of projects without a
only three warnings used on average per project (with and preset = 3,135.
without presets). We will therefore not focus on this setting
in the rest of the analysis. Rules are however most frequently
turned on, or on average 17 times per project (with and
without presets). In addition, developers also turn off an
average of seven rules per project.
It is visible that projects that do not use presets, specify
more rules in general than projects that use a preset. More Enabled Categories. For projects that do not use presets,
specifically, 95% of the projects that do not use a preset, four categories are enabled in more than 50% of all projects:
use at least one rule and specify 58 rules (to be enabled or Possible Errors, Best Practices, Variables and Stylistic Issues
disabled) on average in their configurations (the remaining (Table 9a). The category Stylistic Issues is by far the most
5% only use plugins or do not configure the rules correctly). commonly enabled one, where 82% of all projects that do
Furthermore, 70% of the projects that do use a preset have not use a preset, enable at least one rule. The next category
at least one rule specified, and use 10 rules on average. For in line is Best Practices where 62% of projects without a
these projects, the same percentage of projects enable and preset enabled at least one rule. Furthermore, most rules
disable at least one rule (53%) but more rules are enabled on are enabled on average per project for these two categories.
average (seven rules enabled and three rules disabled). However since these two categories also include the highest
number of available rules (81 and 69), they are not the most
4.3.3 Common Categories commonly used categories in proportion with the amount
of available rules. In that sense, Possible Errors is the most
ESLint provides a set of rules that is grouped into seven
commonly used category with 24% of available rules being
different categories, as explained in the Background section.
used on average per project, whereas with Stylistic Issues
We examine which categories are most commonly enabled
and Best Practices the ratio is 13% and 19%, respectively. The
or disabled, separately between projects that use presets and
other three categories, Strict Mode, Node.js & CommonJS and
those that do not. Tables 9 and 10 show the average number
ECMAScript 6, all have a similar percentage of projects with
of rules that are enabled and disabled, respectively, from
at least one rule, or around 28-29%.
each category per project along with how many projects use
at least one rule from each category.
There is a similar story when it comes to projects that
11. There were two entries for the Airbnb base configurations, do use a preset (Table 9b). Much fewer rules are generally
namely airbnb-base and airbnb/base, with 378 and 179 instances, respec-
tively. The latter is however only a deprecated npm entry point for the enabled as shown in the previous analysis on rule frequency
previous, so these two were counted as one preset in the analysis. (Table 8a), but the most popular categories remain the same.
 13

Category Avg % available # proj % proj Category Avg % available # proj % proj
rules rules
Possible Errors 7.5 24.2% 1,631 52.1% Possible Errors 1.0 3.2% 1,180 37.7%
Best Practices 13.1 19.0% 1,929 61.6% Best Practices 3.7 5.4% 1,687 53.9%
Strict Mode12 0.3 30.0% 895 28.6% Strict Mode 0.3 30.0% 980 31.3%
Variables 2.7 22.5% 1,882 60.1% Variables 0.8 6.7% 1,111 35.5%
Node.js & CommonJS 0.9 9.0% 917 29.3% Node.js & CommonJS 1.0 10.0% 886 28.3%
Stylistic Issues 10.6 13.1% 2,562 81.9% Stylistic Issues 6.8 8.4% 2,096 67.0%
ECMAScript 6 1.8 5.6% 897 28.7% ECMAScript 6 0.9 2.8% 532 17.0%

(a) Enabled rules per category for projects that do not use a (a) Disabled rules per category for projects that do not use a
preset preset
Category Avg % available # proj % proj Category Avg % available # proj % proj
rules rules
Possible Errors 0.4 1.3% 777 12.1% Possible Errors 0.3 1.0% 1,569 24.5%
Best Practices 2.1 3.0% 1,171 18.3% Best Practices 0.6 0.9% 1,505 23.5%
Strict Mode 0.1 10% 448 7.0% Strict Mode 0.1 10.0% 394 6.1%
Variables 0.4 3.3% 1,043 16.3% Variables 0.2 1.7% 709 11.1%
Node.js & CommonJS 0.2 2.0% 279 4.4% Node.js & CommonJS 0.1 1.0% 404 6.3%
Stylistic Issues 3.0 3.7% 3,020 47.1% Stylistic Issues 1.2 1.5% 2,087 32.5%
ECMAScript 6 0.6 1.9% 799 12.5% ECMAScript 6 0.3 0.9% 975 15.2%

(b) Enabled rules per category for projects that do use a preset (b) Disabled rules per category for projects that do use a preset

TABLE 9: Enabled rules per category, showing the average TABLE 10: Disabled rules per category, showing the average
number of rules used per project, the average percentage number of rules used per project, the average percentage
of rules used per project out of all available rules in the of rules used per project out of all available rules in the
category, the number of projects with at least one rule and category, the number of projects with at least one rule and
the percentage of projects with at least one rule out of all the percentage of projects with at least one rule out of all
projects. Categories are listed in the order as they appear in projects. Categories are listed in the order as they appear in
the ESLint documentation [28]. the ESLint documentation [28].

Disabled Categories. For projects that use a preset, Stylis- Enabled Rules. We show in Table 11a that the three most
tic Issues is the most commonly disabled category, with 33% commonly enabled rules for projects without presets, all
of projects disabling at least one rule (Table 10b). The second belong to the Stylistic Issues category. These are formatting
category in this case is Possible Errors with 25% of projects rules that enforce which type of quotation marks should be
disabling a rule. A singularity in these results is that the used (quotes), whether semicolons should be placed at the
Variables category is by far the least disabled category out of end of a line (semi) and what kind of indentation should
the four popular ones, with only 11% of projects disabling a be used (indent). The next four rules in line are from the
rule. Best Practices category and Variables, namely eqeqeq which
As seen in Table 10a, some projects that do not use requires the use of the type-safe triple equality operator
presets still disable some rules, even though it does not (=== instead of ==), no-undef which disallows undeclared
change any of the linter’s functionality. In fact, more than variables, no-unused-vars which disallows unused variables
half of all projects that do not use a preset disable at least one and curly that enforces consistent use of curly braces for
rule from the Stylistic Issues and Possible Errors categories, control statements. More rules in Best Practices follow, along
and disable as much as seven stylistic rules on average. We with rules from the Possible Errors category and Stylistic
discuss possible reasons for why rules are disabled in these Issues. The rule no-dupe-keys prevents errors when two keys
projects in the next section when examining commonly in object literals are identical, no-caller disallows use of
disabled rules. callers and callees which can otherwise make several code
optimizations impossible, and no-unreachable disallows un-
reachable code, e.g., after a return, throw or break statement.
4.3.4 Common Rules
In general, all rules in this list belong to the four categories
Examining which categories are used gives a good overview that have been popular in the previous analyses: Possible
of which types of rules are commonly used and what devel- Errors, Best Practices, Variables and Stylistic Issues.
opers find important. In this part we dive deeper into these Table 11b shows the top 20 enabled rules for projects that
categories and examine which individual rules are enabled do use presets. Interestingly, these rules mostly belong to
and disabled most often. the Stylistic Issues category with 14 out of the 20 rules orig-
inating from the category. The same three stylistic rules as
12. The category Strict Mode is a special case in this analysis since it
is the only category that includes only one rule (others have 10 or more before are the most popular ones, followed by linebreak-style
rules). For that reason, it is less appropriate to report values for this which enforces consistent line endings, comma-dangle which
category on the average percentage of used rules out of all available enforces or disallows trailing commas in object literals, and
rules, as that percentage will always be relatively high compared to space-before-function-paren which enforces consistent use of
other categories. To maintain consistency, the values are reported in
this table and in the following tables, but are not specifically analyzed spaces before function parameter parentheses. These rules
in the text. are followed by some of the few rules from other categories
 14

Rule Category Freq % proj

1 quotes Stylistic Issues 1,898 60.6% Possible Errors 2% 5% 92%

2 semi Stylistic Issues 1,506 48.1%

3 indent Stylistic Issues 1,356 43.3% Best Practices 3% 8% 89%
4 eqeqeq Best Practices 1,338 42.7%
5 no-undef Variables 1,270 40.6% ECMAScript 6 4% 9% 87%
6 no-unused-vars Variables 1,260 40.3%
7 curly Best Practices 1,232 39.4%
8 no-dupe-keys Possible Errors 1,228 39.2% Variables 3% 11% 86%

9 no-caller Best Practices 1,171 37.4%

10 no-unreachable Possible Errors 1,163 37.2% Stylistic Issues 9% 13% 78%
11 no-eval Best Practices 1,155 36.9%
12 brace-style Stylistic Issues 1,126 36.0% NodeJS and CommonJS 9% 29% 63%
13 no-with Best Practices 1,123 35.9%
14 wrap-iife Best Practices 1,123 35.9%
15 no-irregular-whitespace Possible Errors 1,090 34.8% Strict mode 18% 24% 58%
16 comma-style Stylistic Issues 1,089 34.8%
17 no-cond-assign Possible Errors 1,085 34.7% 100 50 0 50 100
18 no-func-assign Possible Errors 1,083 34.6%
19 no-redeclare Best Practices 1,083 34.6% Fig. 7: Perception of participants on the importance of
20 no-invalid-regexp Possible Errors 1,079 34.5% each category. Bars show Unimportant, Slightly important,
Neutral/Not Applicable, Important, and Very Important,
(a) Top 20 enabled rules for projects that do not use a preset
respectively. (N = 266)
Rule Category Freq % proj
1 semi Stylistic Issues 1,641 25.6%
2 indent Stylistic Issues 1,567 24.4% Interestingly, as before, we also observed some rules be-
3 quotes Stylistic Issues 1,336 20.8%
4 linebreak-style Stylistic Issues 888 13.8%
ing disabled even in projects that do not use a preset (Table
5 comma-dangle Stylistic Issues 714 11.1% 12a). We conjecture two possibilities to why that happens:
6 space-before- 1) Some configurations are written in a way where every
function-paren Stylistic Issues 605 9.4% single available rule is listed, and each is set to being enabled
7 eqeqeq Best Practices 601 9.4%
8 no-unused-vars Variables 595 9.3% or disabled. This might make changes clearer, for example
9 curly Best Practices 515 8.0% showing which rules existed when the configuration file was
10 no-trailing-spaces Stylistic Issues 491 7.7% last updated, and thus differentiating between newly added
11 brace-style Stylistic Issues 482 7.5%
12 strict Strict Mode 451 7.0%
rules by ESLint and older rules that the developers decided
13 space-before-blocks Stylistic Issues 424 6.6% to later enable, and 2) Developers want to explicitly show
14 max-len Stylistic Issues 419 6.5% that some rules should absolutely not be turned on. For
15 no-use-before-define Variables 419 6.5% example, the rule no-underscore-dangle is disabled by 49.3%
16 keyword-spacing Stylistic Issues 416 6.5%
17 object-curly-spacing Stylistic Issues 396 6.2% of projects that do not use a preset, where developers might
18 eol-last Stylistic Issues 386 6.0% want to show that they indeed allow the naming pattern
19 new-cap Stylistic Issues 366 5.7% where variables start with an underscore. This would make
20 no-eval Best Practices 365 5.7%
the style obvious to a newcomer and it would also be clear
(b) Top 20 enabled rules for projects that do use a preset that the rule was not simply forgotten or overseen by not
being included in the configurations. Future research needs
TABLE 11: Top 20 enabled rules for projects that use and do to be conducted in order to better understand why this
not use a preset, showing the total number of projects using behavior occurs.
them, along with the percentage of those projects out of all
projects that use or do not use a preset 4.3.5 Developers’ Perceptions on the Importance of Cate-
gories and Rules
The survey participants were asked which ESLint categories
which were also detailed in the previous list. they considered important to include in configurations.
Disabled Rules. Table 12b shows the most commonly Figure 7 shows that Possible Errors is the most important cat-
disabled rules for projects that use a preset. The rule no- egory where 92.5% of the participants consider it to be either
console from the Possible Errors category is by far the most important or very important. Possible Errors is then followed
commonly disabled rule, with 19% of projects specifying by Best Practices, ECMAScript 6, and Variables, where over
the rule. Logging with the console is mainly used for de- 86% of the participants consider each of them to be either
bugging purposes which should normally not be present important or very important. For these four categories, only
in production code. This is followed by no-underscore-dangle four participants considered any of them to be unimportant
from the Stylistic Issues category which disallows dangling (one for Possible Errors and three for ECMAScript 6). The
underscores in identifiers, which are commonly used to three remaining categories, Strict Mode, Node.js & CommonJS
indicate a private variable. Interestingly, here we observe and Stylistic Issues, are all widely considered to be important
again comma-dangle as the third one on the list, thus proving as well, but to a lesser extent than the other categories. Strict
to be a controversial rule. Other rules include func-names to Mode is the category that is most commonly considered to
require or disallow named function expressions for debug- be unimportant by the participants (11%).
ging purposes, and no-param-reassign from the Best Practices Interestingly, we observe that there is a mismatch be-
category which disallows reassigning function parameters. tween the perceptions of participants on how important
 15

Rule Category Freq % proj which should be further studied: 1) Although participants
1 no-underscore-dangle Stylistic Issues 1,543 49.3% perceive these categories as highly important for a linter,
2 strict Strict Mode 982 31.4% they do not completely provide what developers need, or
3 no-ternary Stylistic Issues 686 21.9%
2) Participants are not actually spending time configuring
4 func-names Stylistic Issues 681 21.8%
5 new-cap Stylistic Issues 634 20.3% their linters to provide feedback on what they believe is
6 one-var Stylistic Issues 608 19.4% important.
7 sort-vars Stylistic Issues 597 19.1% To know which individual rules are considered to be the
8 padded-blocks Stylistic Issues 582 18.6%
9 no-use-before-define Variables 581 18.6% most important ones, we asked the survey participants to
10 consistent-return Best Practices 572 18.3% rate the importance of a chosen set of 54 rules. By mining
11 no-console Possible Errors 570 18.2% projects on GitHub we saw that there are four categories
12 camelcase Stylistic Issues 565 18.1% that are most commonly used in configurations: Possible
13 no-extra-parens Possible Errors 560 17.9%
14 func-style Stylistic Issues 549 17.5% Errors, Best Practices, Variables and Stylistic Issues, and these
15 no-plusplus Stylistic Issues 535 17.1% were also discussed most often in the interviews. For each
16 vars-on-top Best Practices 525 16.8% of these categories we asked the participants to rate the
17 no-shadow Variables 525 16.8%
18 no-warning-comments Best Practices 514 16.4%
importance of 14 rules, where the individual rules were
19 valid-jsdoc Possible Errors 499 15.9% chosen as described in Section 3.3. We show the results in
20 wrap-regex Stylistic Issues 495 15.8% Figure 8.
The rules in the Possible Errors category are shown in Fig-
(a) Top 20 disabled rules for projects that do not use a preset
ure 8a. Using the median rating value, eight of the 14 rules
Rule Category Freq % proj
are thought to be either important or very important. The
1 no-console Possible Errors 1,216 19.0% rules that originate from the list of enabled rules without
2 no-underscore-dangle Stylistic Issues 646 10.1%
3 comma-dangle Stylistic Issues 479 7.5%
using a preset are generally thought to be the more impor-
4 func-names Stylistic Issues 430 6.7% tant ones, where no-dupe-keys is the most important rule,
5 no-param-reassign Best Practices 411 6.4% followed by no-unreachable and no-invalid-regexp. Moreover,
6 strict Strict Mode 395 6.2% the rule no-console seems to be one of the most debatable
7 no-use-before-define Variables 358 5.6%
8 consistent-return Best Practices 356 5.6% rules where it is thought to be unimportant by 25.6% of
9 max-len Stylistic Issues 315 4.9% participants but also important by 35.2% of participants.
10 padded-blocks Stylistic Issues 305 4.8% This rule was further commented on by four survey re-
11 arrow-parens ES6 266 4.1%
spondents, saying that it is sometimes not necessary to
12 new-cap Stylistic Issues 262 4.1%
13 camelcase Stylistic Issues 259 4.0% include the rule on single-person projects and that most
14 global-require Node.js 254 4.0% logging should be disallowed except for logging errors.
15 no-shadow Variables 243 3.8% In addition, one participant mentioned the importance of
16 arrow-body-style ES6 224 3.5%
17 no-plusplus Stylistic Issues 204 3.2% the rule no-await-loop (not included in list) due to potential
18 vars-on-top Best Practices 196 3.1% performance problems. Others claimed that the rules that
19 space-before- cause actual bugs are generally the most important ones
function-paren Stylistic Issues 194 3.0%
(without mentioning any specific rules).
20 id-length Stylistic Issues 193 3.0%
The Best Practices rule set is shown in Figure 8b. In this
(b) Top 20 disabled rules for projects that do use a preset case, most rules are thought to be (very) important or 12 out
of the 14 listed rules. The rules eqeqeq and no-eval appear
TABLE 12: Top 20 disabled rules for projects that use and do to be the most important ones where 90.4% and 81.1%
not use a preset, showing the total number of projects using consider them to be either important or very important.
them, along with the percentage of those projects out of all These are then followed by no-unused-expressions and curly.
projects that use or do not use a preset Furthermore, the rules vars-on-top and no-warning-comments
are notably the least important ones, which were originally
retrieved from the lists of the most commonly disabled rules
each category is and the number of projects that actually in GitHub projects. One participant shared his discontent
enable these categories (Figure 7 and Table 9). More specifi- with the rule no-warning-comments, saying that he did not
cally, the Possible Errors category is considered important by know about it previously and that he “would probably flip
92.50% of our participants, while only 52.10% of the projects a table if it broke a CI build”. Another participant expressed
actually enable at least one rule in the category (among that this rule should only be a part of a specific setup for
projects that do not use a preset). Similar ratios are found deployment or production and not for development.
for Best Practices and Variables, and an even larger difference A general observation was also shared by another par-
exists for ECMAScript 6. We also highlight an interesting ticipant saying that it is good to enforce some rules for
fact about the Stylistic Issues category: while this category is production code but that they “should not hinder one’s de-
by far the most enabled one in projects (81.9% of projects velopment style”. The less rated rule vars-on-top was also said
that do not use a preset and 47.1% of projects that do use a to be unimportant as with ES6 it is possible to have block
preset enable at least one of its rule), it is only the fifth most scope local variables with let and const (instead of using the
important category according to the developers’ perceptions var statement which defines a variable regardless of block
(although considered important or very important by 78% scope). Lastly, one respondent discussed the rule no-extend-
of participants). native and the two opposing views about extending native
We hypothesize a few reasons for such a phenomenon, objects. Being in favor of the feature, he described it as one
 16

no−dupe−keys 5% 7% 88% eqeqeq 4% 6% 90%

no−unreachable 7% 7% 86% no−eval 7% 12% 81%
no−invalid−regexp 7% 12% 81% no−unused−expressions 9% 12% 79%
no−irregular−whitespace 17% 14% 70% curly 12% 12% 75%
no−func−assign 17% 17% 67% no−redeclare 14% 11% 75%
no−cond−assign 19% 21% 60% no−extend−native 20% 21% 60%
no−empty 21% 26% 53% wrap−iife 20% 23% 57%
no−extra−semi 30% 19% 51% no−param−reassign 27% 18% 55%
no−constant−condition 25% 29% 46% consistent−return 27% 20% 52%
no−extra−parens 32% 24% 44% no−with 20% 29% 51%
no−extra−boolean−cast 28% 32% 40% no−multi−spaces 26% 23% 51%
no−console 44% 20% 35% no−caller 24% 29% 47%
no−inner−declarations 39% 27% 34% vars−on−top 48% 19% 33%
valid−jsdoc 40% 28% 32% no−warning−comments 44% 38% 18%
100 50 0 50 100 100 50 0 50 100

(a) Possible Errors (N = 256) (b) Best Practices (N = 251)

no−unused−vars 6% 6% 88% indent 8% 5% 86%

no−undef 8% 10% 82% quotes 15% 4% 81%

brace−style 8% 13% 79%
no−use−before−define 9% 12% 79%
semi 10% 15% 75%
no−shadow−restricted−names 12% 28% 60%
comma−style 15% 10% 75%
no−undefined 23% 17% 59% linebreak−style 15% 10% 75%
no−shadow 19% 23% 58% comma−dangle 18% 15% 67%
no−restricted−globals 18% 27% 55% space−before−blocks 19% 16% 65%

no−label−var 17% 29% 54% space−before−function−paren 23% 15% 61%

eol−last 28% 19% 53%
no−catch−shadow 18% 29% 53%
max−len 38% 19% 44%
no−delete−var 30% 26% 44%
func−names 32% 30% 38%
no−undef−init 36% 26% 38%
no−underscore−dangle 44% 28% 28%
init−declarations 37% 39% 25% no−ternary 78% 12% 10%
100 50 0 50 100 100 50 0 50 100

(c) Variables (N = 252) (d) Stylistic Issues (N = 244)

Fig. 8: Participants’ perceptions on the importance of different rules. Bars show Unimportant, Slightly Important,
Neutral/Not Applicable, Important, and Very Important, respectively.

of the advantages of JavaScript where avoiding it is “akin to been available for 2,5 years at the time of analysis. Lastly,
leaving the protective plastic on a couch”. the rule comma-dangle was mentioned by two participants to
For the Variables category in Figure 8c, the set simply be particularly good to catch errors, where it was described
consists of all available rules as there are only 12 in the as “unquestionably the source of most of my lint errors”13 . Other
category. Nine out of the 12 rules are considered to be participants generally described rules from this category to
important by the participants. Three rules seem to be partic- be beneficial for code readability within a team and for
ularly important, no-unused-vars, no-undef and no-use-before- simple code merging.
define, where 87.5%, 81.9% and 79.4% of the participants,
respectively, consider them to be either important or very
important. No specific comments were made about any of RQ3 : Presets are used by 67% of all ESLint projects,
these rules. and projects that do not use presets specify a high
number of 58 rules on average, whereas those that
The subset of rules for the Stylistic Issues category is
do use presets specify 10 rules on average. Stylistic
shown in Figure 8d. 10 out of the 14 rules are considered
Issues is the most commonly enabled category, fol-
to be important, but those are exactly the rules that were
lowed by Best Practices and Variables. The Possible
retrieved from the lists of the most commonly enabled rules.
Errors category is considered to be the most impor-
One rule is considered to be very important (based on the
tant one, according to survey participants.
median value), indent, where 86.3% consider it to be either
important or very important. There are three other rules
which over 40% of the participants consider to be very
important: quotes, semi and linebreak-style. On the contrary,
the rule no-ternary is unquestionably the least important rule 13. The rule comma-dangle was previously a part of the Possible Errors
where 63.8% of participants consider it to be unimportant category due to the known errors that dangling commas could cause in
and only 9.6% consider it to be important. Interestingly, Internet Explorer version 8 and below. The rule was however moved to
the Stylistic Issues category when older versions of Internet Explorer
this rule had only been enabled 8 times in the 3,135 (0.3%) stopped being supported by Microsoft, thus making this error less
analyzed projects that do not use a preset, despite having relevant.
 17

Agreeing on rules as a team 47.0% Very rare 27.8%

Creating and maintaining configurations 38.3%

Never 22.8%

False positives 33.9%

Rare 22.4%
Enforcing rules 33.6%
Occasionally 19.3%
Enabling rules in existing projects 33.2%
N/A 3.1%
Too many warnings/errors 29.2%

Frequently 2.7%
The lack of dynamic analysis 24.5%

No challenges 8.4% Very Frequently 1.9%

0 10 20 30 40 50 0 10 20 30 40

Fig. 9: The challenges that participants face when using a Fig. 10: The frequency of how often developers experienced
linter (N = 298). The full survey question can be found false positives when using ESLint.
in our appendix. Numbers in the bars are represented as
percentages.
a set of 236 rules that ESLint has available which can be a
tedious process: “Most of it was read through every rule, it was
4.4 RQ4 . What Are the Challenges in Using a kind of a very painful process to set it up.” (P3). Another par-
JavaScript Linter? ticipant used very similar wording when it came to setting
up the tool: “Sometimes a pain to set up in your editor with the
In this sub-section, we discuss the results that we derived
right configuration.” (P5). Meanwhile, the other participant
from the interviews, together with the agreement of our
that manually created the configuration file claimed that it
survey participants. We show the most commonly faced
was actually quite easy to set up (P9).
challenges in Figure 9. We provide one additional option in
Others have been frustrated with keeping their configu-
the survey that did not emerge from the interviews, namely:
rations up to date after they have been created, especially
too many warnings/errors outputted from the linter, as it was
when using presets (P1, P8, P12). When the presets are
suggested by one of the pilot test participants and has been
updated, there are often new rules that are enabled or older
reported in related research to be a common problem with
rules that are changed which can cause a high volume
using static analysis tools [8], [54].
of new warnings or errors (P12). P12 explained that these
In the following, sections are ordered by their frequency,
changes are sometimes beneficial and he happily fixes the
according to the survey participants.
warnings, but at other times the change is not useful and
4.4.1 Agreeing on Rules as a Team they have to be overwritten. Moreover, P1 discussed that it
can be frustrating when the presets update very frequently
In an open source project it is relatively easy to build
and the code has to be changed often because of it. However
consensus around what people want to do (P4, P15). If
he also likes that the presets keep him updated of new rules
someone wants a new rule to be enabled, then only a few
regarding new JavaScript features.
main contributors need to say yes and it is decided upon.
A substantial portion of our survey participants also face
Other maintainers usually follow what the project leaders
these challenges: 38.3% agreed that creating or maintaining
propose (P4, P15). In a business environment, where there
configurations was a challenging part of using a linter.
may not be clear leaders, several members of the team
can have different opinions on which rules to enable and 4.4.3 False Positives
disable: “There are 60 people with their own opinion about how We also questioned our survey participants about how often
code should be written.” (P4). P4 further explains that it can they experience false positives (as, according to our inter-
be especially difficult to introduce rules on stylistic issues viewees, this does not happen often, contradicting related
since it is not considered to be important and it can be hard work [55], [8], [10], [56]).
to justify why you would inconvenience people to adhere Interestingly, although one third of the participants see
to new rules: “It may create more tension than it does actually false positives as a challenge (see Figure 9), their perception
solve problems.” (P4). on how often they observe false positives when using ESLint
Agreeing on which rules to use within a team is also seen is low, as we see in Figure 10.14 We discuss more about this
as a challenge for our survey participants, but almost half of contradiction in Section 5.3.2.
them (47.0%) reported to have experienced it as a problem.
4.4.4 Enforcing Rules
4.4.2 Creating and Maintaining Configurations As previously introduced in the Background section, there
Several interviewees mentioned that it was challenging to are three possible settings for a rule in ESLint: off, warn
create the original configuration or to keep it updated (P1,
P3, P5, P8, P12). Only two participants reported that they 14. We also asked how often users from other JavaScript linters
encounter false positives, and numbers were also similarly low. As we
had read over all available rules when they originally cre- focus on ESLint in this paper, these other numbers can be found in our
ated the configuration file (P3, P9). This involves evaluating online appendix [35].
 18

and error. The majority of the interviewees used only or In large existing projects with many collaborators it can
mostly errors and rarely warnings (which also matches with also cause conflicts and frustration when new rules are
our findings in GitHub repositories, see Table 8). These enabled (P4, P13). P4 was working at a company with
settings are used by the participants for different purposes, around 60-70 developers, all working on the same code
such as indicating the criticality of a rule (P15) or by using every day. Usually there are multiple pull requests open at
warnings as an adaptation period when enabling new rules the same time and there will be many merge conflicts when
(P4, P7, P13). Other participants liked to use warnings in a new rule is suddenly enabled, since it will likely affect
the development process so that their build would not be code everywhere in the project. In P13’s project it caused
interrupted when working on unfinished features (P9, P12, frustration with people to enable many rules all at once
P15). since the developers were not accustomed to them before.
There is however a problem with using warnings: sev- Instead, it was decided to do it slowly by enabling only one
eral interviewees claimed to use only errors since warnings rule at a time to give developers a chance to get used to the
would simply be ignored by developers (P2, P3, P4, P13). new rules.
In addition, according to these participants, when warnings
live for a long time in the codebase, people start to devalue 4.4.6 The Lack of Dynamic Analysis
them and leave them behind. Developers do not feel any JavaScript has been described as harsh terrain for static anal-
responsibility for the warnings and might think “there were ysis because of its dynamic nature [57]. Static analysis for
some warnings when I checked it out, not my job to fix it, I’m JavaScript has thus been critized and said to be limited since
going to check it back in with warnings.” (P4). P4 further ex- it can not account for runtime behavior [58]. The majority of
plains that especially if there are many warnings, developers the interview participants would indeed like ESLint to be
will not even read them and simply leave them behind. To able to do more, but they however think that the current
enforce removal of the warnings, they therefore have to be version is very acceptable since they choose themselves to
set as errors that actually prevent a build from succeeding. work with a dynamic language (P2, P5, P6, P7, P8, P9,
Indeed, making sure that the defined rules are actually P10, P11, P15). When P9 was asked whether he misses
followed by developers is a challenge that many developers dynamic analysis from the linter he replied: “Of course, but
face. In our survey, 33.6% of the participants had trouble I don’t particularly think that is ESLint’s fault, so much as a
with enforcing developers to follow the defined rules in the language defect. Due to the dynamic nature of JavaScript, static
past. analysis ranges from extremely difficult to impossible. I don’t
expect ESLint to be able to fix that.” (P9).
Regarding type checking, there are tradeoffs in choosing
4.4.5 Enabling Rules in Existing Projects
JavaScript as a programming language and there is a reason
Six participants (P1, P3, P4, P7, P10, P13) mentioned that why some languages are not strictly typed (P2, P15). If
it can be difficult to start using a linter or to enable new static typing is something that a developer wants, he or
rules in an existing codebase. Moreover, 33.2% of our survey she should rather use TypeScript or some statically typed
participants also face this challenge. language (P2, P10, P15): “Each developer has to make a decision
If the rules that are enabled cause many warnings or on if they want to work in a typed language or in an untyped
errors to occur, substantial effort is needed to go over all language and the tradeoffs of that kind of lead you to the path of
existing code to fix every reported instance. There can even what tools can provide.” (P15). The majority therefore claimed
be such a large volume of existing code that it would simply to be quite satisfied with what the linter can achieve and do
take too much time to review: “The problem with [the project] not expect it to be able to detect more of the dynamic side
is that it’s really old and big, so for that we don’t have the luxury of of the language.
turning on whatever rules we want to because we’re never going Other participants were more bothered by the fact that
to be able to update a lot of our code to support them.” (P1). a linter can not analyze the dynamic parts of the language,
Even if it is possible to fix all warnings, it can be risky where the problem was mostly centered around the lack
to change old code to conform to these rules since it is of variable types (P3, P4, P13, P14). P3 reports that he
easy to introduce new bugs in the meantime (P3, P7, P10). has spent substantial time on testing various mix-ups with
Knowing the original intent of the code can be very difficult strings and numbers, for which he would either like ESLint
and can take considerable time to try to understand (P7). to warn about types that change or would like to switch
P7 discussed an example where there were many instances from using regular JavaScript to TypeScript. Indeed, we
of two equal marks being used in the code instead of three, observe that, in order to remedy this, organizations like
which resulted in many errors from the linter. In these cases Microsoft and Facebook have exerted substantial effort on,
it could have been intentionally written so or by accident, e.g., TypeScript [59] and Flow [60].
which would need to be carefully verified and tested. The 24.5% of our survey participants see the lack of analysis
risk and effort of going back and changing the old code for the dynamic features of the JavaScript language as a
might therefore not be worth it: “Cleaning up for just clean challenge. These numbers strenghten the motivation for
up, just to enable it, I think is risky. I’ve been bitten by that a more research on the topic.
couple of times.” (P3). Enabling a linter in a project to begin
with can be a lot easier: “A linter is great if you start with that 4.4.7 Additional Challenges
and you enforce all those rules and the idea is that you will never Participants also had the opportunity to add other chal-
run into ambiguous code.” (P7). However with older projects lenges which 15 of them did. Amongst these were three chal-
it can be more difficult and even dangerous (P7). lenges mentioned by more than one participant: 1) Getting
 19

team members to use a linter (3 participants), 2) Integration These rules are shown in Table 13, in no particular order
with an IDE or other tools (3 participants), and 3) When except being grouped by category. When compared to the
ESLint, presets or plugins change (2 participants). This last ESLint recommended, Airbnb, Google, and Standard presets
challenge was described by one participant as: “I detest it (the four most popular presets in our sample, see Table 7),
if the rules in a preset change”, which is largely captured we observe that the rules we propose are only fully covered
by a previous challenge in Section 4.4.2. Other reported by the Standard preset. We suggest developers to take this
challenges included difficulty with writing new linting rules list as a starting point for teams that face the challenge of
and the fact that using the tool during development is agreeing on which rules to use, where these rules have been
“obnoxious” but is more beneficial when it comes to the time shown to be important to many other developers. While
of committing code. this is the first step towards an empirical catalogue, research
needs to be conducted so that this catalogue can go beyond
developers’ perceptions (as we also discuss in Section 5.3.4).
RQ4 : The challenges that developers face when us- Additionally, we recommend developers to use presets
ing linters are: to agree on rules in an industrial from the very beginning in case of greenfield engineering,
setting, to create and maintain the configurations, where developers still do not have to deal with large com-
the (low but existing) number of false positives, to plex codebases or legacy issues. As aforementioned, the
enforce developers to follow the rules, to enable existing presets contain the set of rules that experts from
rules in existing projects, and the lack of analysis for the JavaScript community believe to be the most important
JavaScript’s dynamic features. ones. Thus, presets can serve as an initial point for new
projects, where the number of warnings is still manageable
even with the possibly large number of enabled rules in
them. In particular, Ayewah et al. [7] found that FindBugs
5 D ISCUSSION users are more likely to fix warnings that apply to new code
The results have several implications for developers, linter than to older code. We expect the same phenomenon to
creators and researchers, which we discuss in the following. happen in greenfield JavaScript systems.
Finally we explore the relation between the results and the
experience of the survey participants.
5.2 Implications for Tool and Preset Makers
We have seen the main reasons as to why developers choose
5.1 Supporting Developers in Configuring Their Linters to use linters along with which categories they find to be the
There are mainly two ways of defining which rules can be most important ones. Tool creators should therefore place
configured by developers: 1) by using an existing preset, or emphasis on these topics that developers are most con-
2) by manually selecting rules. cerned with. Both the interviewees’ and survey respondents’
Although presets have been developed by experts from main reasons for using linters were to maintain code consis-
the JavaScript community, our study suggests that making tency and to catch possible bugs in code. While stylistic rules
use of a preset in an existing project can be challenging are used the most often in configurations, developers claim
to developers as the number of warnings can be high. that bug-catching rules are far more important. These two
Thus, in cases where greenfield engineering is not possible, topics should therefore be of main interest for tool makers
developers should enable rules carefully and incrementally. when deciding on what type of functionality to implement.
By incrementally introducing rules and continuously fixing Table 13 displays the rules that are deemed the most
the corresponding warnings, the number of warnings can important based on this study’s results. This list of rules
be kept to a minimum, making developers more likely to could be further expanded and ordered based on the rules’
examine and fix them. Previous research has also reported importance, which could be used by ESLint to assign a
that many static analysis tools output too many warnings priority to all available rules. It has been shown in this study
which makes it more difficult to use these tools [8], [61], [4]. that it can be difficult to configure a linter, especially when
We recommend developers to start by enabling the rules one needs to go through all available rules to choose from.
that they consider as the most important ones. After ana- Partly for that reason, the majority of developers choose to
lyzing the three different sources of data, we conjecture that use a preset in their configurations, while also making some
some rules are indeed more important than others (i.e., are modifications to their settings. The process of going through
used and considered important by several projects and all 236 ESLint rules could be made easier by assigning
developers). More specifically, we collected the top four15 them with an importance rating which could be based on
rules that appeared in the three data sources: they were this study’s results, i.e., on the importance reported by the
mentioned by the interviewees, appeared on the top 10 most JavaScript community and the frequency of the rules being
frequent rules in GitHub, and received a median rating of enabled in GitHub projects.
“Important” in the survey. As we did not discuss with or On a similar note, we can also identify some rules
ask any of the participants about the rules in the categories that are rarely used or commonly disabled and are not
ECMAScript 6, Node.js & CommonJS, and Strict Mode, thought to be important by the JavaScript community. These
we did not make any assumptions or recommendations rules include no-ternary (Stylistic Issues), no-underscore-dangle
regarding those. (Stylistic Issues) and vars-on-top (Best Practices). Preset cre-
ators should make sure to not enable these rules as most
15. For the Variables category, we were able to collect only three rules. developers do not want them to be used.
 20

Rule Category ESLint recommended Airbnb Google Standard

no-dupe-keys Possible Errors X X X
no-unreachable Possible Errors X X X
no-invalid-regexp Possible Errors X X X
no-irregular-whitespace Possible Errors X X X X
no-unused-vars Variables X X X X
no-undef Variables X X X
no-shadow-restricted-names Variables X
eqeqeq Best Practices X X
no-eval Best Practices X X
curly Best Practices X X
no-caller Best Practices X X
semi Stylistic Issues X X X
indent Stylistic Issues X X X
quotes Stylistic Issues X X X
comma-dangle Stylistic Issues X X X

TABLE 13: Most important rules to include in configurations and their existence in the ESLint recommended [31],
Airbnb [62], Google [63], and Standard [64] presets.

In general, preset makers should be careful to not update findings are in line with what previous literature reports on
the rule selection too often. Developers that use the presets general static analysis tool usage [55], [8], [10], [56], that the
can get frustrated when they have to do frequent changes to presence of false positives is indeed a problem. On the other
their code due to the presets being modified. hand, both our interviewees and survey respondents claim
Furthermore, as it can be difficult to enforce linting rules to not often experience false positives when using ESLint
without making them break the build, other methods would (see Figure 10). These findings are opposite to what previous
be beneficial to encourage developers to fix warnings. Sad- literature reports on general static analysis tool usage [55],
owski et al. [65] had a similar experience when introducing [8], [10], [56], where false positives are said to be frequent.
static analysis tools at Google, where developers would We conjecture this might be due to the relatively simplistic
ignore warnings that did not cause the build to fail. Tool analysis methods that are applied in linters and the non-
creators should therefore find and employ other ways to complex issues they mostly identify.16
make developers feel responsible to fix them. For example, Our current data does not show which rules provide
this could be done by linking the output of the tool to the developers with false positives. However, our intuition
project repository, to make a somewhat more personal con- suggests that researchers should separate rules that apply
nection to the developers that are responsible. A warning simplistic analysis (such as identifying the usage of tabs or
could perhaps be marked with the name of the developer spaces) from rules that apply more sophisticated analysis
who introduced it, which could motivate him or her to fix (such as identifying non used variables). We do not expect
the warning. the former to be the main cause of false positives. Thus,
more research should be conducted in order to better un-
derstand the quantity of false positives in JavaScript linters.
5.3 Implications for Researchers
Additionally, research shows that the fact that devel-
The results offer ample opportunities for further research opers tend to categorize irrelevant warnings as false pos-
into these findings. Example research directions are listed itives [11], [65] might contribute to the number of expe-
below. rienced false positives. ESLint, in particular, offers vast
functionality for configuring the tool to fit developers’ needs
5.3.1 Evolution of Linter Configurations Over Time and preferences, and we’ve seen that ESLint users often take
Beller et al. [3] studied in large scale the configurations of advantage of that. Research needs to be conducted in order
static analysis tools and how those configurations change to evaluate whether ESLint users experience fewer false pos-
over time. Since several participants claim that it is difficult itives because they have configured the tool appropriately.
to enable linters for existing code, it would be intriguing
to further research how configuration files in projects that 5.3.3 Reasons for Not Using a Linter
enable a linter early on differ from those in projects that Similarly to Johnson et al. [8], we have studied the chal-
enable a linter at a later stage. Furthermore, many partic- lenges in using a linter by interviewing active users of such
ipants explained that they try to create the configuration a tool. Furthermore, much like Christakis and Bird [6], we
so that it fits the project in the best way possible. It would surveyed linter users and asked which challenges they face.
be valuable to know how thoroughly projects follow these In our survey, most participants had used a linter but we do
configurations, providing insight into how well the config- not know how actively they used one or if they continued
uration reflects the project style and how much developers using a linter. As the main focus of our survey was to
care about upholding these code standards. explore the experiences of linter users, we did not ask many

5.3.2 False Positives 16. In our ASE paper [20], we affirmed that these results only con-
tradicted previous literature. In this paper, in the light of new data,
One third of the survey participants affirm that false posi- we observe that false positives are still an open question in JavaScript
tives are a challenge when using linters. On one hand, these linters.
 21

questions to the few participants that had never used one. It can be found in our online appendix [35]. Although we
would be informative to specifically target developers that perform multiple tests, we did not apply any correction on
do not use a linter or those that previously used a linter the significance level (e.g., Bonferroni correction). The reason
but have ceased using one. These developers might provide is that we are interested in the output of each individual test,
different insights into the challenging parts of using such a and not that that all hypotheses are true simultaneously [69].
tool. For H1 , we applied the Chi-Squared test for the combina-
tion of the four different experience groups and five options
5.3.4 Evidence-based Linting Configuration in the Likert scale (strongly disagree, disagree, neutral/NA,
Our results shed light on what rules developers see value agree, strongly agree). We also repeated the same procedure
in and what rules they do not, by means of two different for each one of the seven reasons (see Figure 5). We did not
points of view: first, the configuration they actually choose obtain a significant p-value in any of the tests, which means
to apply to their projects (more than 9,000 projects), and we are not able to accept the hypothesis that the experience
second, based on their perceptions (collected after a survey influences any of their motivations.
with more than 300 developers). For H2 , we applied the Chi-Squared test for the combi-
Currently, all the most popular presets are based either nation of the four different experience groups and whether
on the communities’ or companies’ own experiences. Our that participant has applied that configuration strategy
proposal as to which rules to enable (in Table 13) is the first (yes/no). We also repeated the same procedure for each
set of rules for ESLint that comes from concrete data. one of the nine strategies (see Figure 6). We obtained a
Although our results do not provide an explanation of significant p-value for two strategies: to fit the configuration
what makes developers decide between one rule or the to the style of the project, and to decide on rules after
other (as previously discussed), our results emerge from discussing them in pull requests or similar mechanisms. In
the experience (and choices) of thousands of projects. In both, it seems that, the more experienced a developer is, the
future research, we plan to evaluate in which scenarios more he or she makes use of these strategies.
these choices are the best ones; however, we again reiterate For H3 , we applied the Chi-Squared test for the combina-
that our results provide a very first concrete step towards tion of the four different experience groups and five options
evidence-based linting configuration. in the Likert scale (unimportant, slightly unimportant, neu-
tral/NA, important, very important). We also repeated the
5.4 The Role of Experience same procedure for each one of the seven rule categories (see
Figure 7). We did not obtain a significant p-value in any of the
Throughout the analysis of our survey, we put all partici- tests, which means we are not able to accept the hypothesis
pants in the same bucket, regardless of their experience as that the experience influences their perceptions on which
software developers. However, experience can indeed be a rule categories are most important.
factor of influence, as studies have shown that experience, Finally, for H4 , we applied the Chi-Squared test for the
among other examples, can play a role in productivity [66] combination of the four different experience groups and
and in testing [67]. whether that participant has faced that challenge (yes/no).
To investigate the role of experience in our results, we We also repeated the same procedure for each one of the
applied statistical methods that would reveal any differ- seven challenges (see Figure 9). We did not obtain a signifi-
ences between groups. To separate participants in these cant p-value in any of the tests, which means we are not able
different groups, we first measured the first, second, and to accept the hypothesis that the experience influences their
third quantiles of their years of experience in software perceptions on which challenges they face.
development (1Q = 4, M edian = 4, 3Q = 10). With Therefore, given the results we obtained, we conclude
that, we derived four groups, where each group contained that experience does not play an important role in the results
participants within the following range of experience: 1) G1 we presented throughout this paper.
= [0, 1Q] (116 participants), 2) G2 = ]1Q, M edian] (75 partic-
ipants), 3) G3 = ]M edian, Q3] (83 participants), and 4) G4 =
]Q3, ∞[ (84 participants). 6 T HREATS TO VALIDITY
We then set a number of hypotheses:
In this section, we discuss the threats to the validity of this
H1 . The experience of participants does influence their mo- study as well as the actions we took to mitigate them. We
tivations as to why they use linters (related to RQ1 ). divide this section according to the three different method-
H2 . The experience of participants does influence their con- ologies we apply in this study.
figuration strategies (related to RQ2 ).
H3 . The experience of participants does influence their
perceptions on which ESLint rule categories are most 6.1 Part I. Interviewing JavaScript Developers
important (related to RQ3 ). Transferability. The main limitation to this part of the study
H4 . The experience of participants does influence the chal- is its possible lack of generalizability. The sample size is not
lenges they face (related to RQ4 ). large and it thus may not represent all OSS development.
We applied a Chi-Squared test [68] to test each of the hy- Moreover, as we only talk to developers from OSS projects,
potheses. The test is often used to determine whether there the results may not represent industry software. We tried
is a significant difference between the expected frequencies to mitigate this fact by interviewing experienced developers
and the observed frequencies in one or more categories. We from popular and reputable projects. However, that selec-
used a significance level of α = 0.05 and all the scripts tion of the sample creates another bias in the study where
 22

the results might be different if smaller projects were ex- count as “real” software projects, i.e., not short lived coding
amined along with projects having smaller configurations. experiments or small personal projects. Although it is well
Only projects were selected that had somewhat extensive known that stars on GitHub do not necessarily represent the
configurations as we then expected to receive more input quality (nor the size) of a project [71], manually selecting
on how rules are selected for a project, thus being able to them would have been costly and possibly biased. Never-
report on what methods are used for the task. theless, as we see in Section 3.2.3, our sample contains open
As we only looked at projects that use ESLint, the results source projects that highly vary in terms of code size and
might not reflect on usage of all JavaScript linters. Also creation dates. Future work should focus on replicating this
examining other linters such as JSLint or JSHint might study in industry projects.
produce different results than presented in this study, which Reliability of the Linter Detection Strategy. A limitation
would be an interesting aspect to see in future studies. We of the tool is that it can possibly miss out on some ESLint
chose to observe the usage of only one linter to make the configuration files. The configuration file is typically located
interviews more consistent. As the available linters have in the main directory of a project (as it will then be used
different features, e.g., regarding configurability, we would for the whole project), so in order to save execution time
not have been able to ask all participants the same questions, and to simplify the tool, it is the only location where the
and thus possibly making the analysis less reliable. To tool searches for the file. It could however be the case with
minimize the effects of this possible lack of transferability, some projects that they place their configuration file in a
we chose to address the most popular and most flexible subdirectory of the project and pass it to the linter with the
linter, also to not restrict the results with more limited use command line. This limitation does however not have any
cases. Additionally, we verified that ESLint is indeed the significant effects on the results, since the only real implica-
most popular linter among the top 120 JavaScript projects tion is that the dataset for analyzing the configuration files
by manually examining the linter configurations of each is smaller than it could have been. A trivial limitation is
project. that the numbers are skewed regarding the prevalence of
Credibility. Possible variables that effect the results of using ESLint, as reported in Section 3.2.3 about the dataset’s
this study relate to the previous knowledge of the partici- characteristics. That information is however not of main
pants. It is likely that we interviewed people who already interest for this study.
feel strongly about linters as they are frequent users of Continuing on the topic of the prevalence of linter usage,
the tool. We can not know for sure if their opinions are there are limitations in acknowledging the presence of other
based on their own experience with using the tool or if it linters as well. As the tool does not check for all linters
is based on external literature that they have read. Because that exist, no assumptions can be made about the overall
of this concern, we tried to address our questions to relate prevalence of linter usage. For example, the tool does not
specifically to the participants’ own opinions and experience account for usage of JSLint, but it was decided to disregard
working on the particular project. However, in some cases, that linter since the usage is less frequent than of the other
the participants had other and even more experience in linters and it was more difficult to identify its usage.
working with a linter on other projects, for which they also Moreover, for some linters it is not necessary to have a
based their answers on. configuration file in place and it is also possible to specify
Confirmability. A possible limitation concerns the analy- a configuration file with other names than the default rec-
sis of the interviews. The interviews were conducted by the ognized formats. For these reasons the usage could as well
first author only, who also analyzed the transcripts. In order be understated. On the other hand, the usage in some cases
to reduce any possible bias that this decision could bring to can be overstated. Even though a project has a configuration
the results, the first and the second authors of this paper met file for a specific linter, it does not guarantee that the linter
after every conducted interview and discussed about what is actually used. As an example, the tool found that one
that new piece of information brought to the overall results. project has four linters enabled, having configuration files
Any existing conflict of ideas were solved in these meetings. for ESLint, JSHint and JSCS along with having Standard as
Both researchers also used these meetings to refine the final a dependency [72]. On closer inspection one can see that
codes that were used as main topics throughout the Results only one of them seems to be currently in use (and even
section of this paper. additionally uses TSLint that is intended for TypeScript).
Due to this iterative process, we did not calculate inter- Further measures could have been taken to make this
rater agreement, as commonly done in qualitative studies. analysis more accurate, such as analyzing the dependencies
Nevertheless, it is possible that someone else would have of all projects along with all its pre-running scripts. That
constructed the codes differently, perhaps resulting in differ- would however still not guarantee true results and was
ent conclusions [70]. We therefore make the derived codes considered out of scope, as this is not the main purpose
available online for inspection [35]. of the study. For these reasons, the prevalence results are
presented as estimations of the usage and not as definite
findings.
6.2 Part II. Mining Linter Configurations in Open To increase confidence in the current implementation to
Source Systems identify the usage of ESLint, JSHint, JSCS and Standard,
Sample Selection. Our sample selection procedure consisted manual and automated unit testing were applied. Actual
of selecting all JavaScript projects on GitHub with more use cases of all covered situations to detect these linters were
than 10 stars. As we explain in the Methodology section, used as input to see if the tool identified them correctly.
our goal was to only analyze repositories that can possibly Data Analysis. Our linter detection tool also took the
 23

opportunity to summarize the data on-demand (i.e., count to estimate a response rate. For JS Reddit, it is known
the number of projects that do and do not use presets, num- that the post was opened by 1,300 individuals, resulting
ber of enabled and disabled rules, as well as the arithmetic in a response rate of 20.6% for complete responses17 . It is
mean of rules per project), as we show in Table 8. Our however not known how many people saw the post on
tool did not store the raw files for future analysis. As we the front page and did not decide to take a further look,
only report the arithmetic mean, readers do not have the consequently not having access to the survey link. For the
opportunity to explore the entire distribution, which could last location, Twitter, the tweet reached a number of 4,615
lead them to different insights. In future work, we plan to users but it is difficult to predict how many of those users
better understand the distribution of the enabled, disabled, actually noticed or read the tweet.
and warned rules. Perhaps more important in this case is the representa-
tiveness of the response set. Three of the locations (JS.is,
JS Reddit and Echo JS) are communities that are specifically
6.3 Part III. Surveying Developers
created for JavaScript enthusiasts, which is exactly the target
Survey Design and Analysis. It is important to consider both group of the study. Furthermore, as the title and description
the validity and reliability of a survey [45], [46]. The validity of the survey refer to linters for JavaScript, it should attract
mainly has to do with how well surveys measure what mostly JavaScript developers that have used a linter in the
they are intend to measure, while reliability considers how past. In the case of Twitter, all retweeters except for one (who
consistent and true the responses are. To try to ensure that did not have a self-description) are either developers or
this survey was both valid and reliable, much consideration software engineering professors or researchers. These peo-
went into writing relevant questions that use direct and ple most likely have many followers of related professions
appropriate language. All questions were then reviewed where however not all might be involved with JavaScript.
and evaluated in pilot tests with members of the target We observe that JavaScript is the main language for
population. To further evaluate the reliability of the survey, 96.1% of the participants, with an average experience of
the responses were reviewed in separate batches based on 5.8 years with using JavaScript and every single respondent
the source of the participants, e.g., a separate batch for had used the language in the last 12 months. Furthermore,
all participants that entered the survey via the promotion 87.0% regularly work in industry and 50.6% with OSS,
on Reddit. We did not observe any major discrepancies which allows this analysis to cover both the commercial and
between the different groups. open source sectors. Additionally, the sample is very diverse
For the coding of the open questions, we followed a regarding the participants’ country of residence, leading to
strategy similar to Part I, where the first author performed the results not representing only one market or one part of
the initial coding, and then refined the results by means of the world.
several discussions with the second author. Similarly to the However, there is almost no gender diversity in the sam-
threats of Part I (Section 6.1), it is however possible that ple, which is unfortunately a common problem when per-
someone else would have coded the responses differently, forming surveys within the software engineering field [74].
perhaps resulting in different conclusions [70]. We therefore Even companies like Stack Overflow, with high media im-
also make the derived codes available online for inspection pact, have trouble with gender balance in their popular sur-
[35]. veys (in the 2018 survey with more than 100k respondents,
Survey Evaluation. The survey was pilot tested in order only 6.9% identified themselves as female [75]). Studies have
to identify possible problems and to make general im- shown that there exists bias against women in open source
provements. Six participants were recruited who provided communities [76] and the authors of this paper are strongly
valuable feedback for the survey where some crucial im- committed to gender diversity. Future work must explore
provements were made. After these six tests, the authors different JavaScript communities with a better gender bal-
were confident that these improvements were sufficient for ance.
the survey to be considered both valid and reliable.
It is however possible that not all problems were de- 7 R ELATED W ORK
tected in these tests and that more tests would have led
Static analysis has been a popular research topic where
to making more improvements. To make the pilot tests
many tools have been created, both for academic and in-
as efficient and effective as possible, several sources of
dustrial use. These tools can be convenient as they are
literature were examined to learn about what aspects to
used without executing the software which they are applied
focus on. This preparation played a key part in the resulting
on. A common research topic is the construction of such
confidence that the authors had about the final quality of the
tools, such as for security vulnerability, fault detection, and
survey. Additionally, when examining the final responses of
code smells [77], [78], [79], [80], [81], [82], and the various
the survey, no major problems could be identified with the
methods to do so, such as data flow, information flow,
answers, such as anything implying a wide misunderstand-
path or pointer analysis [5]. Other research topics explore
ing of any questions.
the usage of these tools, which we are more interested in
Sampling and Responses. It is important to consider
for this particular study. These topics include developers’
the response rate and the representativeness of a survey’s
perceptions of static analysis tools [6], [8], the value of
response set [73], but it is however difficult to calculate a
using these tools in real world applications [55], [5] and the
response rate for this kind of convenience sampling. For two
of the distribution places, JS.is and Echo JS, it is impossible 17. It is estimated that 378 partial responses were received from
to know how many individuals viewed the original posts Reddit, resulting in a response rate of 49.7% for all responses.
 24

challenge of false positives [10], [56]. In the following we although developers agree that Possible Errors is a highly
briefly explore these latter topics and finally discuss static important category to be enabled, in practice, only 52% of
analysis in the world of JavaScript. projects actually enable such rules. Our hypothesis is some-
what inline with previous research, where we conjecture
that the set of available Possible Errors rules is not sufficient.
7.1 Perceptions of Static Analysis Tools
Thus, more research needs to be conducted in order to
Johnson et al. [8] researched the usage of static analysis tools, provide developers with a set of rules that are able to detect
in particular why some developers do not use these tools to more complex and important bugs.
find bugs despite of their proven benefits. They conducted a
study where they interactively interviewed 20 participants 7.2 Configurations of Static Analysis Tools
while applying FindBugs on software. They found that Ayewah et al. [7], [83] studied the usage of FindBugs
the main reasons why the participants chose to use static where they saw that users are generally interested in fixing
analysis tools were to avoid the time consuming manual warnings from the tool, especially the high priority ones, but
labor to find bugs, to support team development efforts and which types of warnings depends on the user’s context. It
to enforce coding standards. Reasons to not use these tools is therefore deemed valuable to have configuration options
include poorly presented output where there are too many for these different groups of users. Similarly, Jaspan et al. [2]
false positives or too many warnings reported in general, recognized the importance of customizing and prioritizing
in addition to tools not being integrated conveniently in rules to make developers more willing to use a static analy-
the workflow. Moreover, most participants claimed that it sis tool as it reduces the number of perceived false positives
is important for these tools to be easily customizable and for a project.
that warnings should be explained properly, including how Regarding how developers configure these tools, Beller
they can be fixed. Our work is inspired by this study where et al. [3] performed a large-scale study on the prevalence
we also research why static analysis tools are used and how of static analysis tool usage along with how they are config-
they can be improved, but focusing solely on JavaScript. ured. They examined static analysis tools for four languages,
We see that the overall expectation of JavaScript developers Java, JavaScript, Python and Ruby, where for JavaScript they
with static analysis tools are similar to the ones in previous observed JSHint, ESLint, JSCS and JSL [84]. They found that
research. these tools are commonly used in OSS software (over half
A similar study was conducted by Christakis and Bird [6] of all analyzed projects) and in particular for JavaScript
who also investigate how developers perceive static analysis projects where JSHint was by far the most widely used
tools, specifically which barriers they face in the adoption of linter18 . The configurations for these projects are most often
these tools and how these tools should be created so that changed from the default settings, but typically only one
developers will take advantage of them. For this purpose rule is added, removed or modified. Furthermore, after a
they surveyed a random sample of 375 Microsoft devel- configuration file has been created, they are rarely modified.
opers. What was considered to be the largest obstacle in For the types of rules that are configured, those that are re-
using these tools was the fact that some unwanted rules are lated to maintainability were both more commonly enabled
turned on by default in the tools’ configurations. Proposed and disabled than those that have to do with functional
solutions to this problem are to have a subset of rules defects. This is in line with our findings as we have seen that
enabled instead of all rules being enabled by default, or to rules from the categories Stylistic Issues and Best Practices are
make the configuration process easy for developers. This enabled and disabled more often than rules from the Possible
idea is similar to what ESLint does; there are no default Errors category. Beller et al. predict that fewer bug-finding
configurations but one can easily enable a preset with a rules are enabled as tools are known to perform poorly in
selected subset of rules that the creators of ESLint think finding defects.
are the most important (eslint:recommended). In addition, Zampetti et al. [9] further analyzed the usage of static
in RQ2 , we explore the different strategies that JavaScript analysis tools in continuous integration (CI) pipelines in
developers have been applying in order to derive the best popular OSS projects. For the 20 analyzed projects, Check-
configuration for their projects. We conjecture our findings style was the most commonly used static analysis tool,
also make sense for static analysis tools of other languages. followed by FindBugs, where 11 projects used only one
Moreover, other frequently experienced challenges, as tool while the rest employed multiple tools. Most of these
reported by Christakis and Bird, were bad warning mes- static analysis tools were configured to break the build
sages, too many false positives and for the analysis to be in the CI pipeline, especially in the case of Checkstyle
too slow. For the functionality that these developers want (only one project solely raised warnings). Most projects
static analysis tools to detect, security issues, best practices manually configured the rules that were activated in these
and concurrency issues were the most commonly requested projects for at least one tool that was used, where Check-
ones. Issues relating to style were among the least requested style was configured in every single case. While FindBugs
features for such a tool to have. However, the majority of was typically configured to include all available rules, the
the developers reported that the issues they encounter that percentage of used rules for Checkstyle always remained
can be caught by static analysis tools are most commonly below 40%. The most commonly enabled Checkstyle rules
best practices violations and style inconsistencies, opposed were regarding indentation (Whitespace), unused imports
to more complex issues such as regarding reliability or (Imports), possible defects (Coding) and code blocks (Block
concurrency. The expectations of these developers therefore
seem to match with a tool like ESLint. Our results show that, 18. At the time of analysis in early 2015.
 25

checks). The configurations of the static analysis tools were to produce a false positive rate above 90% when applying
not changed often over the projects’ lifetime, each with less them on several software systems. The presence of false
than 10 modifications except for one project. Zampetti et positives is indeed known to be one of the main barriers of
al. recommend developers to configure static analysis tools static analysis tool adoption [8], [6], [4]. Developers will not
to their needs to benefit the most from using these tools, tolerate a high false positive rate and quickly discard a tool if
e.g., to avoid unnecessary build failures. the rate is too high [6], [65]. More specifically, Christakis and
Some of the previous literature indeed included ESLint Bird [6] found that most developers do not tolerate a false
but we suspect that these results would be somewhat dif- positive rate of over 20%. They prefer for tools to identify
ferent if these studies would be conducted again today, as fewer defects, rather than catching more bugs and having
ESLint has since then removed any default configurations a higher volume of false positives. Furthermore, Wagner et
in the tool and all rules are now off by default. A developer al. [85] experienced that when static analysis tools report a
using the tool thus now needs to create some configuration, high number of false positives, it can take even more time
which in the simplest case could just include the recom- to shift through them to find the true defects, than the time
mended settings from ESLint. that is saved on manual effort by using a static analysis tool
In addition, the related work does not perform a fine- in the first place.
grained analysis on the rules that are enabled and disabled, The term false positive can be understood in two dif-
as we proposed. Our results bring clear understanding on ferent ways, either a wrongly reported warning or a true
the prevalence of each existing rule in ESLint. We suggest warning that is not considered by a developer to improve
researchers to replicate our approach in the configuration of the software under analysis [11]. Tool makers have thus
static analysis tools in other languages. experienced that the users decide for themselves what a
false positive is and therefore how effective a tool is [65].
7.3 Effectiveness of Static Analysis Tools Additionally, when tools identify more intricate issues, the
risk increases of users not understanding the issue and
Several studies have focused on the effectiveness of static
mistaking true positives for false positives [10]. In general
analysis tools to find bugs in software systems. In a case
when there are many reported false positives, developers
study, Wagner et al. [85] found that static analysis tools
start trusting the tool less and less [10]. With less trust, even
report on different warnings than those that are found by
more true positives are thought to be false positives by the
testing a software, indicating that it is beneficial to use these
developers. Couto et al. [61] and Jaspan et al. [2] further
two methods in combination. Furthermore, static analysis
experienced that developers categorize warnings that are
tools find a subset of the issues that are found via code
not of their interest as false positives. They stressed the
review, thus if used beforehand, they can save some of
importance of customizing tools and prioritizing rules so
the time that goes into performing the manual task. Zheng
that only high-priority and relevant warnings are reported.
et al. [5] also found that static analysis tools are comple-
Without doing so, tools can not become practically usable.
mentary to other fault-detection techniques but with limited
As we discuss in Section 5.3.2, although our participants
usability where testing was found to be more effective.
see false positives as a challenge, they perceive them to
Wedyan et al. [55] further examined how effective static
occur infrequently. We raised a few hypotheses on why this
analysis tools are in detecting faults and refactoring oppor-
happens in that section. Therefore, we invite researchers to
tunities. Studying two software systems, three known static
conduct studies on the prevalence of false positives in the
analysis tools were only able to detect 3% of the defects
existing JavaScript linters.
that were actually found in the projects’ lifetime. These
tools were however much more successful in identifying
the refactorings that were performed in the project, or 7.5 Static Analysis Tools Used in Industry
around 70%. Similarly, Couto et al. [61] also found that
An abundance of static analysis tools have been developed
FindBugs was not able to identify most of the errors that
with diverse features and implementations, where software
had been fixed in some studied software systems. However
systems have different requirements for these tools. Many
they found some level of correlation between the number of
static analysis tools have been used by major software
reported warnings and the number of actual field bugs. So
companies such as Google, Facebook and Microsoft, both
despite the fact that static analysis tools may not be good at
tools that have been created internally and externally [6].
finding actual field defects, they can serve as good indicators
Bessay et al. [10], the creators of Coverity, experienced first
of the level of internal quality in a software system.
hand how difficult it can be to create a static analysis tool
In our paper, although we extensively report how devel-
that fits for commercial use of companies with large code
opers have been using linters, which rules they opt to enable
bases. The technical requirements of such companies can
and disable, as well as their perceptions on how important
vary greatly, e.g., using miscellaneous build processes and
each of them are, we do not measure how effective each
developing environments that can make running the tool
rule really is. In future work, we intend to understand and
different for each particular case. The size and nature of
measure the quality of these rules.
software projects can also entail different requirements for
static analysis tools, where Google resorted to creating their
7.4 False Positives in Static Analysis Tools own tool as no other available static analysis tool could
A common problem with static analysis tools is the high handle the size of their codebase [65].
volume of false positives [55], [8], [10], [56]. For example, Google have indeed tried and reported from using sev-
Wedyan et al. [55] found well known static analysis tools eral static analysis tools such as FindBugs and Coverity [86].
 26

Despite some success with adopting these tools, ultimately In general, while there have been several proposed static
they were not frequently used by developers due to the analysis tools and techniques for JavaScript in previous
presence of false positives, lack of scalability and incon- research, most suffer from being sound but impractical,
venient workflow integration [65]. Eventually they instead or the opposite, practical but unsound [93], [91]. Practical
created their own tool, Tricorder [65], a static analysis tool limitations include lack of scalability and inabilities to an-
with an ecosystem of developers creating and reviewing alyze library usage. More tools have been developed for
custom analyzers. They only display high priority warnings specific challenges of JavaScript, such as the prevalent use
from analyzers that have received good reviews from other of external libraries [94], [89]. Some large libraries, such as
developers, in an attempt to make the developers more the Browser API and HTML DOM, are written in other
willing to use the tool by eliminating false positives and languages and do not provide a JavaScript implementation,
low priority warnings. thus making static analysis more difficult.
EBay [2] reported on their substantial effort into choos- Other common libraries like jQuery [95] are also known
ing the most valuable and appropriate tool set for their to make extensive use of many dynamic language fea-
projects. They report on a method to evaluate these tools tures, such as use of eval and computed property names,
where eventually the tool FindBugs was integrated to their which also contributes to the difficulty of static analysis for
development process. For their software it was deemed JavaScript code. Another related challenge is the dynamic
important that the adopted tool should be extensible (to file loading of these libraries where the code is therefore not
allow for project custom rules) and customizable. They made available immediately for analysis. For this problem,
report that customization and prioritization are important researchers have proposed staged analysis [96], [97] which
steps in evaluating a tool as it can reduce the false positive is applied in two stages, first for all statically available
rate for a project and make developers more willing to use code and later for the remaining dynamically loaded code.
the tool. Interestingly, the false positives that have been identified
In our paper, although some of the interviewees were with static analysis of JavaScript code (by applying the
also developers in industry, the extensive mining analysis tool SAFE [98]), mostly had to do with API usage and
we performed was conducted solely on open source sys- asynchronous function calls, though also due to dynamic
tems. Future work should focus on how companies are file loading and dynamic code generation [99].
configuring such linters. Despite the challenges that it brings, automated static
analysis can be especially beneficial for languages that do
not have strict typing [5]. Type analysis is considered to
7.6 Static and Dynamic Analysis for JavaScript be a crucial part to catch representation errors where, for
As JavaScript has become more popular it has also attracted example, numbers can be confused with strings or func-
more attention in the research community. The need for tions confused with booleans [87], [15]. Additionally, simple
more and better tooling for the language has been empha- mistakes such as spelling mistakes can result in surprising
sized along with the challenges that JavaScript introduces consequences at run time, which is mostly avoided in other
for proper static analysis. Several studies have therefore languages with static typing [90]. Pradel et al. [15] created
focused on techniques for, and the creation of, static analysis a tool, TypeDevil, to counter against errors that have to
tools for JavaScript [87], [88], [89]. do with inconsistent types in JavaScript, using dynamic
JavaScript is a dynamic language, which e.g., involves analysis where types of variables, properties and functions
dynamic typing, dynamic file loading, first-class functions are observed at runtime. Moreover, as typing systems have
and property access, and in general code that can be gener- been developed such as TypeScript [100], Gao et al. [101]
ated during runtime. Richards et al. [57] analyzed the usage studied the effects of dynamic typing on errors in JavaScript
of dynamic features in JavaScript code in commonly used applications and examined whether typing systems could
websites and found that some dynamic features are indeed prevent them. They added type annotations to code that had
frequently used, such as adding and removing properties to caused errors in software systems and examined whether
objects after their initialization and using the eval function using TypeScript or Flow on that code would surface the
for various unpredictable purposes. As such, JavaScript has previously encountered errors. They saw that applying
been described as harsh terrain for static analysis [57], [89], these static type systems would detect many errors (15% in
[90], [91]. The usability of static analysis tools for JavaScript this study) that otherwise can go unnoticed into production
has thus also been criticized and said to be limited since code.
these tools can not account for runtime behavior [58]. For As our results show, developers acknowledge the fact
this problem, tools have been created that employ dynamic that JavaScript is a dynamic language, and thus, they accept
analysis of program executions [91], [58], [92]. A recent the fact that static analysis tools may have a hard time
example is DLint [58] that, evaluated against JSHint, de- in analyzing dynamic behavior. However, developers also
tected on average 49 new code quality issues per studied showed their interest in seeing such analysis. Our findings
system. Example issues they identified are object properties reinforce the importance of more research in the field of
shadowing prototype properties, passing too many argu- static and dynamic analysis of dynamic languages.
ments to a function, accessing the undefined property, and Lastly, code smells in JavaScript have recently been stud-
silent coercion of arithmetic operations to NaN. Some issues ied where Fard et al. [92] developed a tool, JSNose, to detect
were detected by both DLint and JSHint but with different potential smells. Saboury et al. [102] later analyzed the
methods to do so, where DLint increased the total warnings prevalence and impact of several code smells and surveyed
found by 10%, thus complementing JSHint. developers on their perceptions of those smells. The studied
 27

code smells were derived from literature and online style [6] M. Christakis and C. Bird, “What developers want and need
guides, including recommendations from ESLint and from from program analysis: an empirical study,” in Proceedings of
the 31st IEEE/ACM International Conference on Automated Software
the Airbnb preset. They found that JavaScript files that are Engineering. ACM, 2016, pp. 332–343.
affected with code smells tend to be more error-prone than [7] N. Ayewah, D. Hovemeyer, J. D. Morgenthaler, J. Penix, and
those without any smells. Moreover, Variable Re-assign and W. Pugh, “Using static analysis to find bugs,” IEEE software,
Assignment in Conditional Statements were among the most vol. 25, no. 5, 2008.
[8] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why
dangerous smells. Additionally, when surveying a large don’t software developers use static analysis tools to find bugs?”
sample of JavaScript developers, Nested Callbacks, Variable in 2013 35th International Conference on Software Engineering
Re-assign and Long Parameter List were perceived to be the (ICSE). IEEE, 2013, pp. 672–681.
most hazardous smells affecting the maintainability and [9] F. Zampetti, S. Scalabrino, R. Oliveto, G. Canfora, and
M. Di Penta, “How open source projects use static code analysis
reliability of JavaScript applications. tools in continuous integration pipelines,” in Proceedings of the
14th International Conference on Mining Software Repositories. IEEE
8 C ONCLUSION Press, 2017, pp. 334–344.
[10] A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem,
In this study we have applied three research methods where C. Henri-Gros, A. Kamsky, S. McPeak, and D. Engler, “A few
we first interviewed 15 experts on using ESLint, then an- billion lines of code later: using static analysis to find bugs in the
real world,” Communications of the ACM, vol. 53, no. 2, pp. 66–75,
alyzed over 9,500 ESLint configuration files from GitHub
2010.
projects and finally surveyed more than 300 JavaScript [11] N. Ayewah, W. Pugh, J. D. Morgenthaler, J. Penix, and Y. Zhou,
developers about their experiences with using a linter, ex- “Evaluating static analysis defect warnings on production soft-
ploring the findings from the previous two analyses. ware,” in Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop
on Program analysis for software tools and engineering. ACM, 2007,
Our findings can be summarized as follows: pp. 1–8.
• We see that most developers use a linter to maintain [12] GitHub, “Language Trends on GitHub,” https://github.com/
code consistency. Nevertheless, both interview and sur- blog/2047-language-trends-on-github, [Online; accessed 13-
vey participants claim that rules that relate to possible June-2017].
[13] F. S. Ocariza Jr, K. Pattabiraman, and B. Zorn, “JavaScript errors
errors are far more important than those that relate to in the wild: An empirical study,” in Software Reliability Engineer-
stylistic issues. ing (ISSRE), 2011 IEEE 22nd International Symposium on. IEEE,
• Regarding how developers configure linters, they gen- 2011, pp. 100–109.
erally prefer to use a preset rather than to manually [14] T. Mikkonen and A. Taivalsaari, “Using JavaScript as a real
programming language,” Sun Microsystems, Inc., 2007.
choose every rule that is used for a project. However, [15] M. Pradel, P. Schuh, and K. Sen, “Typedevil: Dynamic type
most developers additionally apply some configura- inconsistency analysis for javascript,” in Proceedings of the 37th
tions that are specific to a project or a team. International Conference on Software Engineering-Volume 1. IEEE
• While it is important to be able to identify errors early
Press, 2015, pp. 314–324.
[16] J. W. Creswell, Research design: Qualitative, quantitative, and mixed
in the development process by using a linter, even more methods approaches. Sage publications, 2013.
developers feel the need to have consistent formatting [17] “ESLint,” http://eslint.org, [Online; accessed 11-July-2017].
where everyone taking part in a project uses quotes, [18] P. Vorbach, “npm-stat, download statistics for packages eslint,
semicolons and indentation in the same way. jshint, jslint, jscs, standard,” https://npm-stat.com/charts.html?
package=eslint&package=jshint&package=jslint&package=jscs&
• Upholding these configurations can be a challenging
package=standard&from=2015-01-01&to=2017-05-31, [Online;
task where it can be difficult to agree with teammates accessed 2-June-2017].
on which rules should be used for a project. [19] B. G. Glaser and J. Holton, “Remodeling grounded theory,” in Fo-
rum Qualitative Sozialforschung/Forum: Qualitative Social Research,
The results of this study produced valuable implications vol. 5, no. 2, 2004.
for developers, tool makers and researchers, who can use the [20] K. F. Tómasdóttir, M. Aniche, and A. v. Deursen, “Why and
information to make better use of linters, to improve future how javascript developers use linters,” in Proceedings of the 32nd
versions of linters and to further research this important IEEE/ACM International Conference on Automated Software Engi-
neering. IEEE Press, 2017, pp. 578–589.
aspect of software development.
[21] “FindBugs,” http://findbugs.sourceforge.net, [Online; accessed
11-July-2017].
R EFERENCES [22] “Checkstyle,” http://checkstyle.sourceforge.net, [Online; ac-
cessed 11-July-2017].
[1] B. W. Boehm, Software engineering economics. Prentice-hall Engle- [23] “PMD,” https://pmd.github.io, [Online; accessed 11-July-2017].
wood Cliffs (NJ), 1981, vol. 197. [24] “JSHint,” http://jshint.com, [Online; accessed 11-July-2017].
[2] C. Jaspan, I. Chen, A. Sharma et al., “Understanding the value of
[25] “JSCS,” http://jscs.info, [Online; accessed 11-July-2017].
program analysis tools,” in Companion to the 22nd ACM SIGPLAN
[26] “JSLint,” http://jslint.com, [Online; accessed 11-July-2017].
conference on Object-oriented programming systems and applications
companion. ACM, 2007, pp. 963–970. [27] G. Richards, C. Hammer, B. Burg, and J. Vitek, “The eval that
[3] M. Beller, R. Bholanath, S. McIntosh, and A. Zaidman, “Analyz- men do,” in European Conference on Object-Oriented Programming
ing the state of static analysis: A large-scale evaluation in open (ECOOP). Springer, 2011, pp. 52–78.
source software,” in 2016 IEEE 23rd International Conference on [28] ESLint, “Rules,” http://eslint.org/docs/rules, [Online; accessed
Software Analysis, Evolution, and Reengineering (SANER), vol. 1. 2-June-2017].
IEEE, 2016, pp. 470–481. [29] AirBnb ESLint preset. [Online]. Available: https://github.com/
[4] N. Rutar, C. B. Almazan, and J. S. Foster, “A comparison of bug airbnb/javascript
finding tools for Java,” in Software Reliability Engineering, 2004. [30] “Standard JS,” https://standardjs.com, [Online; accessed 11-July-
ISSRE 2004. 15th International Symposium on. IEEE, 2004, pp. 2017].
245–256. [31] ESLint, “Configuring ESLint,” http://eslint.org/docs/
[5] J. Zheng, L. Williams, N. Nagappan, W. Snipes, J. P. Hudepohl, user-guide/configuring, [Online; accessed 27-May-2017].
and M. A. Vouk, “On the value of static analysis for fault [32] S. Adolph, W. Hall, and P. Kruchten, “Using grounded theory
detection in software,” IEEE transactions on software engineering, to study the experience of software development,” Empirical
vol. 32, no. 4, pp. 240–253, 2006. Software Engineering, vol. 16, no. 4, pp. 487–513, 2011.
 28

[33] K.-J. Stol, P. Ralph, and B. Fitzgerald, “Grounded theory in [61] C. Couto, J. E. Montandon, C. Silva, and M. T. Valente, “Static cor-
software engineering research: a critical review and guidelines,” respondence and correlation between field defects and warnings
in Proceedings of the 38th International Conference on Software Engi- reported by a bug finding tool,” Software Quality Journal, vol. 21,
neering. ACM, 2016, pp. 120–131. no. 2, pp. 241–257, 2013.
[34] S. E. Hove and B. Anda, “Experiences from conducting semi- [62] “AirBnb ESLint base preset,” https://github.com/airbnb/
structured interviews in empirical software engineering re- javascript/tree/master/packages/eslint-config-airbnb-base,
search,” in Software metrics, 2005. 11th ieee international symposium. [Online; accessed 15-January-2018].
IEEE, 2005, pp. 10–pp. [63] “Google ESLint preset,” https://github.com/google/
[35] K. F. Tómasdóttir, M. Aniche, and A. van Deursen, “The Adop- eslint-config-google, [Online; accessed 15-January-2018].
tion of JavaScript Linters in Practice: A Case Study on ESLint [64] “Standard ESLint preset,” https://github.com/standard/
[Data set],” https://doi.org/10.5281/zenodo.1410967, zenodo. eslint-config-standard, [Online; accessed 15-January-2018].
[36] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, [65] C. Sadowski, J. Van Gogh, C. Jaspan, E. Söderberg, and C. Winter,
and D. Damian, “The promises and perils of mining GitHub,” “Tricorder: Building a program analysis ecosystem,” in Software
in Proceedings of the 11th working conference on mining software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Con-
repositories. ACM, 2014, pp. 92–101. ference on, vol. 1. IEEE, 2015, pp. 598–608.
[37] GitHub, “About Stars,” https://help.github.com/articles/ [66] W. Fong Boh, S. A. Slaughter, and J. A. Espinosa, “Learning
about-stars, [Online; accessed 6-March-2017]. from experience in software development: A multilevel analysis,”
[38] “BigQuery,” https://cloud.google.com/bigquery, [Online; ac- Management Science, vol. 53, no. 8, pp. 1315–1331, 2007.
cessed 29-May-2017]. [67] A. Beer and R. Ramler, “The role of experience in software testing
[39] G. Gousios, “The GHTorrent dataset and tool suite,” in practice,” in Software Engineering and Advanced Applications, 2008.
Proceedings of the 10th Working Conference on Mining Software SEAA’08. 34th Euromicro Conference. IEEE, 2008, pp. 258–265.
Repositories, ser. MSR ’13. Piscataway, NJ, USA: IEEE Press, 2013, [68] P. E. Greenwood and M. S. Nikulin, A guide to chi-squared testing.
pp. 233–236. [Online]. Available: http://dl.acm.org/citation.cfm? John Wiley & Sons, 1996, vol. 280.
id=2487085.2487132
[69] T. V. Perneger, “What’s wrong with bonferroni adjustments,”
[40] “GHTorrent,” http://ghtorrent.org, [Online; accessed 29-May- BMJ: British Medical Journal, vol. 316, no. 7139, p. 1236, 1998.
2017].
[70] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and
[41] “ESLint first release (v0.0.2),” https://github.com/eslint/eslint/
A. Wesslén, Experimentation in software engineering. Springer
releases?after=v0.1.0, [Online; accessed 23-May-2017].
Science & Business Media, 2012.
[42] “ESLint first major release (v1.0.0),” https://github.com/eslint/
[71] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German,
eslint/releases?after=v1.0.0, [Online; accessed 23-May-2017].
and D. Damian, “An in-depth study of the promises and perils of
[43] “stream-handbook via GitHub (example of a repository con-
mining github,” Empirical Software Engineering, vol. 21, no. 5, pp.
taining a programming guide),” https://github.com/substack/
2035–2071, 2016.
stream-handbook, [Online; accessed 23-May-2017].
[72] “MQTT.js via GitHub (example of a repository containing four
[44] “Impress.js-Tutorial via GitHub (example of a repository con-
linters),” https://github.com/mqttjs/MQTT.js, [Online; accessed
taining code for a tutorial),” https://github.com/cubewebsites/
26-May-2017].
Impress.js-Tutorial, [Online; accessed 23-May-2017].
[45] A. Fink, The survey handbook. Sage, 2003, vol. 1. [73] B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C.
[46] D. De Vaus, Surveys in social research. Routledge, 2013. Hoaglin, K. El Emam, and J. Rosenberg, “Preliminary guidelines
for empirical research in software engineering,” IEEE Transactions
[47] B. A. Kitchenham and S. L. Pfleeger, “Principles of survey re-
on software engineering, vol. 28, no. 8, pp. 721–734, 2002.
search: part 1: turning lemons into lemonade,” ACM SIGSOFT
Software Engineering Notes, vol. 26, no. 6, pp. 16–18, 2001. [74] B. A. Kitchenham and S. L. Pfleeger, “Principles of survey re-
[48] K. F. Tómasdóttir, “Post promoting survey on JavaScript user search: part 5: populations and samples,” ACM SIGSOFT Software
group (Iceland) on Facebook,” https://facebook.com/groups/ Engineering Notes, vol. 27, no. 5, pp. 17–20, 2002.
nodejsis/permalink/1709121425832578, [Online; accessed 14- [75] S. Overflow, “Stack Overlow Developer Survey 2018.” https:
June-2017]. //insights.stackoverflow.com/survey/2018/, [Online; accessed
[49] ——, “Post promoting survey on Reddit.” https://redd.it/ April-2018].
6eumsp, [Online; accessed 14-June-2017]. [76] J. Terrell, A. Kofink, J. Middleton, C. Rainear, E. Murphy-Hill,
[50] “Echo JS,” http://echojs.com, [Online; accessed 14-June-2017]. C. Parnin, and J. Stallings, “Gender differences and bias in open
[51] K. F. Tómasdóttir, “Post promoting survey on Twitter.” https:// source: Pull request acceptance of women versus men,” PeerJ
twitter.com/kristinfjolato/status/871986432952479747, [Online; Computer Science, vol. 3, p. e111, 2017.
accessed 14-June-2017]. [77] V. B. Livshits and M. S. Lam, “Finding security vulnerabilities
[52] “eslint-config-airbnb via npm,” https://npmjs.com/package/ in Java applications with static analysis.” in USENIX Security
eslint-config-airbnb, [Online; accessed 29-May-2017]. Symposium, vol. 14, 2005, pp. 18–18.
[53] “eslint-config-airbnb-base via npm,” https://npmjs.com/ [78] M. Christodorescu and S. Jha, “Static analysis of executables
package/eslint-config-airbnb-base, [Online; accessed 29-May- to detect malicious patterns,” Wisconsin Univ-Madison Dept of
2017]. Computer Sciences, Tech. Rep., 2006.
[54] S. Heckman and L. Williams, “On establishing a benchmark [79] M. Aniche, G. Bavota, C. Treude, A. Van Deursen, and M. A.
for evaluating static analysis alert prioritization and classifica- Gerosa, “A validated set of smells in model-view-controller ar-
tion techniques,” in Proceedings of the Second ACM-IEEE interna- chitectures,” in Software Maintenance and Evolution (ICSME), 2016
tional symposium on Empirical software engineering and measurement. IEEE International Conference on. IEEE, 2016, pp. 233–243.
ACM, 2008, pp. 41–50. [80] M. Aniche, C. Treude, A. Zaidman, A. van Deursen, and M. A.
[55] F. Wedyan, D. Alrmuny, and J. M. Bieman, “The effectiveness of Gerosa, “Satt: Tailoring code metric thresholds for different
automated static analysis tools for fault detection and refactoring software architectures,” in Source Code Analysis and Manipula-
prediction,” in Software Testing Verification and Validation, 2009. tion (SCAM), 2016 IEEE 16th International Working Conference on.
ICST’09. International Conference on. IEEE, 2009, pp. 141–150. IEEE, 2016, pp. 41–50.
[56] D. Hovemeyer and W. Pugh, “Finding bugs is easy,” ACM Sigplan [81] M. Aniche, G. Bavota, C. Treude, M. A. Gerosa, and A. van
Notices, vol. 39, no. 12, pp. 92–106, 2004. Deursen, “Code smells for model-view-controller architectures,”
[57] G. Richards, S. Lebresne, B. Burg, and J. Vitek, “An analysis of Empirical Software Engineering, pp. 1–37, 2017.
the dynamic behavior of JavaScript programs,” in ACM Sigplan [82] N. Tsantalis, T. Chaikalis, and A. Chatzigeorgiou, “Jdeodorant:
Notices, vol. 45, no. 6. ACM, 2010, pp. 1–12. Identification and removal of type-checking bad smells,” in
[58] L. Gong, M. Pradel, M. Sridharan, and K. Sen, “DLint: Dynami- Software Maintenance and Reengineering, 2008. CSMR 2008. 12th
cally checking bad coding practices in JavaScript,” in Proceedings European Conference on. IEEE, 2008, pp. 329–331.
of the 2015 International Symposium on Software Testing and Analysis. [83] N. Ayewah and W. Pugh, “A report on a survey and study of
ACM, 2015, pp. 94–105. static analysis users,” in Proceedings of the 2008 workshop on Defects
[59] “TypeScript,” http://typescriptlang.org, [Online; accessed 10- in large software systems. ACM, 2008, pp. 1–5.
July-2017]. [84] “JavaScript Lint,” http://javascriptlint.com, [Online; accessed 7-
[60] “Flow,” http://flowtype.org, [Online; accessed 10-July-2017]. August-2017].
 29

[85] S. Wagner, J. Jürjens, C. Koller, and P. Trischberger, “Comparing Kristı́n Fjóla Tómasdóttir is a Software De-
bug finding tools with reviews and tests,” Lecture Notes in Com- veloper at Amazon in Cambridge, United King-
puter Science, vol. 3502, pp. 40–55, 2005. dom. Kristı́n completed her (cum laude) MSc
[86] N. Ayewah and W. Pugh, “The google findbugs fixit,” in Pro- degree at Delft University of Technology, The
ceedings of the 19th international symposium on Software testing and Netherlands. Her research focused on an empir-
analysis. ACM, 2010, pp. 241–252. ical evaluation of JavaScript linters in real-world
[87] S. H. Jensen, A. Møller, and P. Thiemann, “Type analysis for open source systems.
JavaScript,” in SAS, vol. 9. Springer, 2009, pp. 238–255.
[88] M. Sridharan, J. Dolby, S. Chandra, M. Schäfer, and F. Tip, “Cor-
relation tracking for points-to analysis of JavaScript,” European
Conference on Object-Oriented Programming (ECOOP), pp. 435–458,
2012.
[89] M. Madsen, B. Livshits, and M. Fanning, “Practical static analysis Maurı́cio Aniche is an Assistant Professor at
of JavaScript applications in the presence of frameworks and li- Delft University of Technology, The Netherlands.
braries,” in Proceedings of the 2013 9th Joint Meeting on Foundations Maurı́cio helps developers to effectively main-
of Software Engineering. ACM, 2013, pp. 499–509. tain, test, and evolve their software systems. His
[90] S. H. Jensen, M. Madsen, and A. Møller, “Modeling the HTML current research interests are systems monitor-
DOM and browser API in static analysis of JavaScript web appli- ing and DevOps, empirical software engineering,
cations,” in Proceedings of the 19th ACM SIGSOFT symposium and and software testing.
the 13th European conference on Foundations of software engineering.
ACM, 2011, pp. 59–69.
[91] M. Schäfer, M. Sridharan, J. Dolby, and F. Tip, “Dynamic determi-
nacy analysis,” in ACM SIGPLAN Notices, vol. 48, no. 6. ACM,
2013, pp. 165–174. Arie van Deursen is professor in software en-
[92] A. M. Fard and A. Mesbah, “JSNOSE: Detecting JavaScript code gineering at Delft University of Technology, The
smells,” in Source Code Analysis and Manipulation (SCAM), 2013 Netherlands, where he heads the Software En-
IEEE 13th International Working Conference on. IEEE, 2013, pp. gineering Research Group (SERG) and chairs
116–125. the Department of Software Technology. His re-
[93] Y. Ko, H. Lee, J. Dolby, and S. Ryu, “Practically tunable static search interests include empirical software en-
analysis framework for large-scale JavaScript applications (T),” gineering, software testing, and software archi-
in Automated Software Engineering (ASE), 2015 30th IEEE/ACM tecture. He aims at conducting research that will
International Conference on. IEEE, 2015, pp. 541–551. impact software engineering practice, and has
[94] E. Andreasen and A. Møller, “Determinacy in static analysis for co-founded two spin-off companies from earlier
jQuery,” in ACM SIGPLAN Notices, vol. 49, no. 10. ACM, 2014, research. He serves on the editorial boards of
pp. 17–31. Empirical Software Engineering, and the open access PeerJ/CS.
[95] “jQuery,” https://jquery.com, [Online; accessed 13-June-2017].
[96] R. Chugh, J. A. Meister, R. Jhala, and S. Lerner, “Staged informa-
tion flow for JavaScript,” ACM Sigplan Notices, vol. 44, no. 6, pp.
50–62, 2009.
[97] S. Guarnieri and B. Livshits, “GULFSTREAM: Staged Static Anal-
ysis for Streaming JavaScript Applications.” WebApps, vol. 10, pp.
6–6, 2010.
[98] H. Lee, S. Won, J. Jin, J. Cho, and S. Ryu, “SAFE: Formal specifi-
cation and implementation of a scalable analysis framework for
ECMAScript,” in International Workshop on Foundations of Object-
Oriented Languages (FOOL), vol. 10, 2012.
[99] J. Park, I. Lim, and S. Ryu, “Battles with false positives in static
analysis of JavaScript web applications in the wild,” in Proceed-
ings of the 38th International Conference on Software Engineering
Companion. ACM, 2016, pp. 61–70.
[100] G. Bierman, M. Abadi, and M. Torgersen, “Understanding Type-
Script,” in European Conference on Object-Oriented Programming.
Springer, 2014, pp. 257–281.
[101] Z. Gao, C. Bird, and E. T. Barr, “To type or not to type: quantifying
detectable bugs in JavaScript,” in Proceedings of the 39th Interna-
tional Conference on Software Engineering. IEEE Press, 2017, pp.
758–769.
[102] A. Saboury, P. Musavi, F. Khomh, and G. Antoniol, “An empirical
study of code smells in JavaScript projects,” in Software Analysis,
Evolution and Reengineering (SANER), 2017 IEEE 24th International
Conference on. IEEE, 2017, pp. 294–305.