Exploring the Relationship Between Web Application

Development Tools and Security

Matthew Finifter David Wagner

University of California, Berkeley University of California, Berkeley
[email protected] [email protected]

Abstract Security strategies of an organization often include de-

veloping processes and choosing tools that reduce the
How should software engineers choose which tools to number of vulnerabilities present in live web applica-
use to develop secure web applications? Different devel- tions. These software security measures are generally
opers have different opinions regarding which language, focused on some combination of (1) building secure soft-
framework, or vulnerability-finding tool tends to yield ware, and (2) finding and fixing security vulnerabilities
more secure software than another; some believe that in software after it has been built.
there is no difference at all between such tools. This pa- How should managers and developers in charge of
per adds quantitative data to the discussion and debate. these tasks decide which tools languages, frameworks,
We use manual source code review and an automated debuggers, etc. to use to accomplish these goals? What
black-box penetration testing tool to find security vul- basis of comparison do they have for choosing one tool
nerabilities in 9 implementations of the same web ap- over another? Common considerations for choosing
plication in 3 different programming languages. We ex- (e.g.,) one programming language over another include:
plore the relationship between programming languages How familiar staff developers are with the language.
and number of vulnerabilities, and between framework
support for security concerns and the number of vul- If new developers are going to be hired, the current
nerabilities. We also compare the vulnerabilities found state of the market for developers with knowledge
by manual source code review and automated black-box of the language.
penetration testing. Interoperability with and re-usability of existing in-
Our findings are: (1) we do not find a relationship house and externally-developed components.
between choice of programming language and applica-
Perceptions of security, scalability, reliability, and
tion security, (2) automatic framework protection mech-
maintainability of applications developed using that
anisms, such as for CSRF and session management, ap-
language.
pear to be effective at precluding vulnerabilities, while
Similar considerations exist for deciding which web ap-
manual protection mechanisms provide little value, and
plication development framework to use and which pro-
(3) manual source code review is more effective than au-
cess to use for finding vulnerabilities.
tomated black-box testing, but testing is complementary.
This work begins an inquiry into how to improve one
part of the last of these criteria: the basis for evaluating
1 Introduction a tools inclination (or disinclination) to contribute to ap-
plication security.
The web has become the dominant platform for new soft- Past research and experience reveal that different tools
ware applications. As a result, new web applications can have different effects on application security. The
are being developed all the time, causing the security software engineering and software development commu-
of such applications to become increasingly important. nities have seen that an effective way to preclude buffer
Web applications manage users personal, confidential, overflow vulnerabilities when developing a new appli-
and financial data. Vulnerabilities in web applications cation is to simply use a language that offers automatic
can prove costly for organizations; costs may include di- memory management. We have seen also that even if
rect financial losses, increases in required technical sup- other requirements dictate that the C language must be
port, and tarnished image and brand. used for development, using the safer strlcpy instead
of strcpy can preclude the introduction of many buffer security of applications developed using that language.
overflow vulnerabilities. For example, research has shown that type systems can
This research is an exploratory study into the security statically find (and therefore preclude, by halting compi-
properties of some of the tools and processes that orga- lation) certain types of vulnerabilities [21, 20]. In gen-
nizations might choose to use during and after they build eral, static typing can find bugs (any of which could be
their web applications. We seek to understand whether a vulnerability) that may not have been found until the
the choice of language, web application development time of exploitation in a dynamically-typed language.
framework, or vulnerability-finding process affects the Also, one languages standard libraries might be more
security of the applications built using these tools. usable, and therefore less prone to error, than anothers.
We study the questions by analyzing 9 independent A modern exception handling mechanism might help de-
implementations of the same web application. We col- velopers identify and recover from dangerous scenarios.
lect data on (1) the number of vulnerabilities found in But programming languages differ in many ways be-
these implementations using both a manual security re- yond the languages themselves. Each language has its
view and an automatic black-box penetration testing tool, own community, and these often differ in their philoso-
and (2) the level of security support offered by the frame- phies and values. For example, the Perl community val-
works. We look in these data sets for patterns that might ues TMTOWTDI (Theres more than one way to do
indicate differences in application security between pro- it) [4], but the Zen of Python [16] states, [t]here should
gramming languages, frameworks, or processes for find- be one and preferably, only one obvious way to do
ing vulnerabilities. These patterns allow us to generate it. Clear documentation could play a role as well.
and test hypotheses regarding the security implications Therefore, we want to test whether the choice of lan-
of the various tools we consider. guage measurably influences overall application security.
This papers main contributions are as follows: If so, it would be useful to know whether one language
We develop a methodology for studying differences fares better than another for any specific class of vulner-
in the effect on application security that differ- ability. If this is the case, developers could focus their ef-
ent web application development tools may have. forts on classes for which their language is lacking good
The tools we consider are programming languages, support, and not worry so much about those classes in
web application development frameworks, and pro- which data show their language is strong.
cesses for finding vulnerabilities.
Web application development framework. Web ap-
We generate and test hypotheses regarding the dif-
plication development frameworks provide a set of li-
ferences in security implications of these tools.
braries and tools for performing tasks common in web
We develop a taxonomy for framework-level de- application development. We want to evaluate the role
fenses that ranges from always on framework sup- that they play in the development of secure software.
port to no framework support. This can help developers make more informed decisions
We find evidence that automatic framework-level when choosing which technologies to use.
defenses work well to protect web applications, but Recently, we have seen a trend of frameworks adding
that even the best manual defenses will likely con- security features over time. Many modern frameworks
tinue to fall short of their goals. take care of creating secure session identifiers (e.g.,
We find evidence that manual source code analy- Zend, Ruby on Rails), and some have added support
sis and automated black-box penetration testing are for automatically avoiding cross-site scripting (XSS) or
complementary. cross-site request forgery (CSRF) vulnerabilities (e.g.,
Django, CodeIgniter). It is natural to wonder if frame-
works that are pro-active in developing security features
2 Goals yield software with measurably better security, but up to
this point we have no data showing whether this is so.
Programming language. We want to measure the in-
fluence that programming language choice has on the Vulnerability-finding tool. Many organizations man-
security of the software developed using that language. age security risk by assessing the security of software
If such an influence exists, software engineers (or their before they deploy or ship it. For web applications, two
managers) could take it into account when planning prominent ways to do so are (1) black-box penetration
which language to use for a given job. This informa- testing, using automated tools designed for this purpose,
tion could help reduce risk and allocate resources more and (2) manual source code analysis by an analyst knowl-
appropriately. edgeable about security risks and common vulnerabili-
We have many reasons to believe that the features of ties. The former has the advantage of being mostly au-
a programming language could cause differences in the tomated and being cheaper; the latter has a reputation as
 Team Language Frameworks used with metrics like performance, completeness, size, and
Number
usability, we re-analyze their data to evaluate the security
1 Perl DBIx::DataModel, Catalyst, Tem-
plate Toolkit
properties of these 9 programs.
2 Perl Mason, DBI Each team submitted a complete source code package
5 Perl Gantry, Bigtop, DBIx::Class and a virtual machine image. The VM image runs a web
3 Java abaXX, JBoss, Hibernate server, which hosts their implementation of People by
4 Java Spring, Spring Web Flow, Hiber-
Temperament. The source code packages were trimmed
nate, Acegi Security
9 Java Equinox, Jetty, RAP to remove any code that was not developed specifically
6 PHP Zend Framework, OXID framework for the contest, and these trimmed source code packages
7 PHP proprietary framework were released under open source licenses.1
8 PHP Zend Framework For our study, we used the set of virtual machine
Table 1: Set of frameworks used by each team. images and the trimmed source code packages. The
Plat Forms study gathered a lot of other data (e.g., sam-
ples at regular intervals of the current action of each de-
more comprehensive but more expensive. However, we veloper) that we did not need for the present study. The
are not aware of quantitative data to measure their rela- data from our study are publicly available online.2
tive effectiveness. We work toward addressing this prob-
lem by comparing the effectiveness of manual review to
that of automated black-box penetration testing. Solid 3.1 People by Temperament
data on this question may help organizations make an in-
formed choice between these assessment methods. We familiarized ourselves with the People by Tempera-
ment application before beginning our security analysis.
The application is described as follows:

3 Methodology PbT (People by Temperament) is a simple commu-

nity portal where members can find others with whom they
might like to get in contact: people register to become
In order to address these questions, we analyze several members, take a personality test, and then search for others
independent implementations of the same web applica- based on criteria such as personality types, likes/dislikes,
tion specification, written using different programming etc. Members can then get in contact with one another if
both choose to do so. [18]
languages and different frameworks. We find vulnerabil-
ities in these applications using both manual source code People by Temperament is a small but realistic web ap-
review and automated black-box penetration testing, and plication with a non-trivial attack surface. It has security
we determine the level of framework support each im- goals that are common amongst many web applications.
plementation has at its disposal to help it contend with We list them here:
various classes of vulnerabilities. We look for associa- Isolation between users. No user should be able
tions between: (1) programming language and number to gain access to another users account; that is,
of vulnerabilities, (2) framework support and number of all information input by a user should be integrity-
vulnerabilities, and (3) number of vulnerabilities found protected with respect to other users. No user
by manual source code analysis and by automated black- should be able to view another users confidential
box penetration testing. information without approval. Confidential infor-
We analyze data collected in a previous study called mation includes a users password, full name, email
Plat Forms [19]. In that work, the researchers devised address, answers to personality test questions, and
and executed a controlled experiment that gave 9 profes- list of contacts. Two users are allowed to view each
sional programming teams the same programming task others full name and email address once they have
for a programming contest. Three of the teams used Perl, agreed to be in contact with one another.
three used PHP, and the remaining three used Java.
Database confidentiality and integrity. No user
The contest rules stated that each team had 30 hours to
should be able to directly access the database, since
implement the specification of a web application called
it contains other users information and it may con-
People By Temperament [18]. Each team chose which
tain confidential web site usage information.
frameworks they were going to use. There was little
overlap in the set of frameworks used by teams using Web site integrity. No user should be able to van-
the same programming language. Table 1 lists the set of dalize or otherwise modify the web site contents.
frameworks used by each team. 1 http://www.plat-forms.org/sites/default/files/
The researchers collected the 9 programs and analyzed platforms2007solutions.zip
their properties. While they were primarily concerned 2 http://www.cs.berkeley.edu/ finifter/datasets/

 Stored XSS 3.2.1 Manual source code review
Reflected XSS
Integer-valued
SQL injection One reviewer (Finifter) reviewed all implementations.
Authentication or authorization bypass
CSRF
This reviewer is knowledgeable about security and had
Binary Broken session management previously been involved in security reviews.
Insecure password storage Using one reviewer for all implementations avoids the
Table 2: The types of vulnerabilities we looked for. We distin- problem of subjectivity between different reviewers that
guish binary and integer-valued vulnerability classes. Integer- would arise if the reviewers were to examine disjoint sets
valued classes may occur more than once in an implementa- of implementations. We note that having multiple re-
tion. For example, an application may have several reflected viewers would be beneficial if each reviewer was able
XSS vulnerabilities. The binary classes represent presence or to review all implementations independently of all other
absence of an application-wide vulnerability. For example, in reviewers.
all implementations in this study, CSRF protection was either
The reviewer followed the Flaw Hypothesis Method-
present throughout the application or not present at all.
ology [6] for conducting the source code reviews. He
Review Dev. Lang. SLOC Review Review
used the People by Temperament specification and
Number Team Time Rate knowledge of flaws common to web applications to de-
Number (min.) (SLOC/hr) velop a list of types of vulnerabilities to look for. He
1 6 PHP 2,764 256 648 performed two phases of review, first looking for spe-
2 3 Java 3,686 229 966 cific types of flaws from the list, then comprehensively
3 1 Perl 1,057 210 302
reviewing the implementation. He confirmed each sus-
4 4 Java 2,021 154 787
5 2 Perl 1,259 180 420 pected vulnerability by developing an exploit.
6 8 PHP 2,029 174 700 We randomly generated the order in which to perform
7 9 Java 2,301 100 1,381 the manual source code reviews in order to mitigate any
8 7 PHP 2,649 99 1,605 biases that may have resulted from choosing any partic-
9 5 Perl 3,403 161 1,268
ular review order. Table 3 presents the order in which
Table 3: Time taken for manual source code reviews, and num- the reviewer reviewed the implementations as well as the
ber of source lines of code for each implementation.
amount of time spent on each review.
The reviewer spent as much time as he felt necessary
System confidentiality and integrity. No user to perform a complete review. As shown in Table 3, the
should be able to gain access to anything on the number of source lines of code reviewed per hour varies
web application server outside of the scope of the widely across implementations; the minimum is 302 and
web application. No user should be able to execute the maximum is 1,605. Cohen [7] states that [a]n ex-
additional code on the server. pected value for a meticulous inspection would be 100-
The classes of vulnerabilities that we consider are pre- 200 LOC/hour; a normal inspection might be 200-500.
sented in Table 2. A vulnerability in any of these classes It is unclear upon what data or experience these numbers
violates at least one of the applications security goals. are based, but we expect the notion of normal to vary
across different types of software. For example, we ex-
pect a review of a kernel module to proceed much more
3.2 Vulnerability data slowly than that of a web application. Additionally, we
note that the number of source lines of code includes both
We gathered vulnerability data for each implementation static HTML content and auto-generated code, neither of
in two distinct phases. In the first phase, a reviewer per- which tends to require rigorous security review.
formed a manual source code review of the implementa- To help gauge the validity of our data for manual
tion. In the second phase, we subjected the implemen- source code review, we test the following hypotheses:
tation to the attacks from an automated black-box web Later reviews take less time than earlier reviews.
penetration testing tool called Burp Suite Pro [17].
We used both methods because we want to find as More vulnerabilities were found in later reviews.
many vulnerabilities as we can. We hope that any failings Slower reviews find more vulnerabilities.
of one method will be at least partially compensated by If we find evidence in support of either of the first two
the other. Although we have many static analysis tools at hypotheses, this may indicate that the reviewer gained
our disposal, we chose not to include them in this study experience over the course of the reviews, which may
because we are not aware of any that work equally well have biased the manual review data. A more experi-
for all language platforms. Using a static analysis tool enced reviewer can be expected to find a larger fraction
that performs better for one language than another would of the vulnerabilities in the code, and if this fraction in-
have introduced systematic bias into our experiment. creases with each review, we expect our data to be biased
in showing those implementations reviewed earlier to be abled by the developer if he so desires. Team 2s custom
more secure. Spearmans rho for these two hypotheses is ORM framework provides opt-out support for SQL in-
= 0.633 (p = 0.0671) and = 0.0502 (p = 0.8979), jection. If the framework is used, SQL injection cannot
respectively, which means that we do not find evidence occur, but a developer can opt out by going around the
in support of either of these hypotheses. framework to directly issue SQL queries.
If we find evidence in support of the third of these Opt-in support refers to a defense that is disabled by
hypotheses, this may indicate that the reviewer did not default, but can be enabled by the developer to provide
allow adequate time to complete one or more of the re- protection throughout the application. Enabling the pro-
views. This would bias the data to make it appear that tection may involve changing a configuration variable
those implementations reviewed more quickly are more or calling into the framework code at initialization time.
secure than those for which the review proceeded more Once enabled, opt-in support defends against all subse-
slowly. The correlation coefficient between review rate quent instances of that vulnerability class. Acegi Secu-
and number of vulnerabilities found using manual anal- rity, used by Team 4, provides a PasswordEncoder
ysis is r = 0.0676 (p = 0.8627), which means we do interface with several different implementations. We
not find evidence in support of this hypothesis. The lack consider this opt-in support because a developer can se-
of support for these hypotheses modestly increases our lect an implementation that provides secure password
confidence in the validity of our manual analysis data. storage for his application.
Manual support is the weakest level of framework
support. This term applies if the framework provides
3.2.2 Black-box testing a useful routine to help protect against a vulnerability
class, but that routine must be utilized by the devel-
We used Portswiggers Burp Suite Professional version
oper each time protection is desired. For example, many
1.3.08 [17] for black box testing of the implementations.
frameworks provide XSS filters that can be applied to
We chose this tool because a previous study has shown it
untrusted data before it is included in the HTML page.
to be among the best of the black box testing tools [11]
These filters spare a developer the burden of writing a
and because it has a relatively low cost.
correct filter, but the developer must still remember to
We manually spidered each implementation before
invoke the filter every time untrusted data is output to
running Burp Suites automated attack mode (called
a user. Manual support is weak because a developer has
scanner). All vulnerabilities found by Burp Suite were
many opportunities to make an error of omission. Forget-
manually verified and de-duplicated (when necessary).
ting to call a routine (such as an XSS filter) even once is
enough to introduce a vulnerability. We use the term au-
tomatic support to contrast with manual support; it refers
3.3 Framework support data
to any level of support stronger than manual support.
We devised a taxonomy to categorize the level of support For each implementation, we looked at the source
a framework provides for protecting against various vul- code to discover which frameworks were used. We read
nerability classes. We distinguish levels of framework through the documentation for each of the frameworks
support as follows. to find out which protection mechanisms were offered
The strongest level of framework support is always on. for each vulnerability class we consider. We defined the
Once a developer decides to use a framework that offers implementations level of support for a particular vulner-
always-on protection for some vulnerability class, a vul- ability class to be the highest level of support offered by
nerability in that class cannot be introduced unless the any framework used by the implementation.
developer stops using the framework. An example of this
is the CSRF protection provided automatically by Spring
Web Flow [10], which Team 4 used in its implementa- 3.4 Individual vulnerability data
tion. Spring Web Flow introduces the notion of tokens,
which define flows through the UI of an application, and We gather data about each individual vulnerability to
these tokens double as CSRF tokens, a well-known pro- deepen our understanding of the current framework
tection mechanism for defending against CSRF vulner- ecosystem, the reasons that developers introduce vul-
abilities. Since they are integral to the functionality the nerabilities, and the limitations of manual review. For
framework provides, they cannot be removed or disabled each vulnerability, we determine how far the developers
without ceasing to use the framework entirely. would have had to stray from their chosen frameworks in
The next strongest level of framework support is opt- order to find manual framework support that could have
out support. This level of support provides protection prevented the vulnerability. Specifically, we label each
against a vulnerability class by default, but it can be dis- vulnerability with one of the following classifications:
 Java Perl PHP rity practices because their jobs required them to be [23].
Number of programmers 9 9 9
Mean age (years) 32 32 32
It may be that explicitly mentioning security or speci-
Mean experience (years) 7.1 8.7 9.8 fying security requirements would have changed the de-
Table 4: Statistics of the programmers. velopers focus and therefore the security of the imple-
mentations, but we believe that the lack of special men-
1. Framework used. Framework support that could tion is realistic and representative of many programming
have prevented this vulnerability exists in at least projects. In the worst case, this limits the external valid-
one of the frameworks used by the implementation. ity of our results to software projects in which security is
not explicitly highlighted as a requirement.
2. Newer version of framework used. Framework
support exists in a newer version of one of the Small sample size. Due to the cost of gathering data of
frameworks used by the implementation. this nature, the sample size is necessarily small. This is
a threat to external validity because it makes it difficult
3. Another framework for language used. Frame- to find statistically significant results. In the worst case,
work support exists in a different framework for the we can consider this a case study that lets us generate
same language used by the implementation. hypotheses to test in future research.
4. Some framework for some language. Framework Generalization to other applications. People by
support exists in some framework for some lan- Temperament is one web application, and any findings
guage other than the one used by the implementa- with respect to it may not hold true with respect to other
tion. web applications, especially those with vastly different
5. No known support. We cannot find framework requirements or of much larger scale. The teams had
support in any framework for any language that only 30 hours to complete their implementation, which
would have stopped the vulnerability. is not representative of most real software development
We label each vulnerability with the lowest level at which projects. Despite these facts, the application does have
we are able to find framework support that could have a significant amount of functionality and a large enough
prevented the vulnerability. We do so using our aware- attack surface to be worth examining.
ness and knowledge of state-of-the-art frameworks as Number of vulnerabilities. We would like to find the
well as the documentation frameworks provide. total number of vulnerabilities present in each implemen-
Similarly, for each vulnerability, we determine the tation, but each analysis (manual and black-box) finds
level at which the developers could have found automatic only some fraction of them. If the detection rate of our
(i.e., opt-in or better) framework support. We evaluate manual analysis is better for one language or one imple-
this in the same manner as we did for manual support, but mentation than it is for another, this is a possible threat
with a focus only on automatic protection mechanisms. to validity. However, we have no reason to expect a sys-
tematic bias of this nature, as the reviewers level of ex-
3.5 Threats to validity perience in manual source code review is approximately
equivalent for all three languages. At no time did the
Experimental design. The Plat Forms data were gath-
reviewer feel that any one review was easier or more dif-
ered in a non-randomized experiment. This means that
ficult than any other.
the programmers chose which language to use; the lan-
Similarly, if the detection rate of our black-box tool
guage was not randomly assigned to them by the re-
is better for one language or implementation than it is
searchers. This leaves the experiment open to selection
for another, this could pose a threat to validity. We have
bias; it could be the case that more skilled programmers
no reason to believe this is the case. Because black-box
tend to choose one language instead of another. As a re-
testing examines only the input-output behavior of a web
sult, any results we find represent what one might expect
application and not its implementation, it is inherently
when hiring new programmers who choose which lan-
language- and implementation-agnostic, which leads us
guage to use, rather than having developers on staff and
to expect that it has no bias for any one implementation
telling them which language to use.
over any other.
Programmer skill level. If the skill level of the pro- Vulnerability severity. Our analysis does not take into
grammers varies from team to team, then the results rep- account any differences in vulnerability severity. Using
resent the skills of the programmers, not inherent proper- our analysis, an implementation with many low-severity
ties of the technologies they use for development. Fortu- vulnerabilities would appear less secure than an imple-
nately, the teams had similar skills, as shown in Table 4. mentation with only a few very high-severity vulnerabil-
Security awareness. Security was not explicitly men- ities, though in fact the latter system may be less secure
tioned to the developers, but all were familiar with secu- overall (e.g., expose more confidential customer data).
We have no reason to believe that average vulnerability Total Number of Vulnerabilities
severity varies widely between implementations, but we

40
did not study this in detail.

30
Vulnerabilities introduced later in the product cycle.
Our study considers only those vulnerabilities introduced

20
during initial product development. Continued develop-
ment brings new challenges for developers that simply

10
were not present in this experiment. Our results do not
answer any questions about vulnerabilities introduced
during code maintenance or when features are added af-

0
Java 3 Java 4 Java 9 PHP 6 PHP 7 PHP 8 Perl 1 Perl 2 Perl 5
ter initial product development. Manual Both
Black-box

Framework documentation. If a frameworks docu-

mentation is incomplete or incorrect, or if we misread Figure 1: The total number of vulnerabilities found in the 9
implementations of People by Temperament. The x-axis is la-
or misunderstood the documentation, we may have mis-
beled with the language and team number.
labeled the level of support offered by the framework.
However, the documentation represents the level of sup-
port a developer could reasonably be expected to know One of the Perl implementations has by far the most
about. If we were unable to find documentation for pro- vulnerabilities, primarily due to its complete lack of XSS
tection against a class of vulnerabilities, we expect that protection.3 This does not seem to be related to the fact
developers would struggle as well. that Perl is the language used, however, since the other
two Perl implementations have only a handful of vulner-
Awareness of frameworks and levels of support. abilities, and few XSS vulnerabilities.
There may exist frameworks that we are not aware of The Java implementations have fewer vulnerabilities
that provide strong framework support for a vulnerabil- than the PHP implementations. In fact, every Java imple-
ity class. If this is the case, our labeling of vulnerabilities mentation contains fewer vulnerabilities than each PHP
with the nearest level at which framework support exists implementation.
(Section 3.4) may be incorrect. We have made every ef- A one-way ANOVA test reveals that the overall re-
fort to consider all frameworks with a significant user lationship in our data set between language and total
base in order to mitigate this problem, and we have con- number of vulnerabilities is not statistically significant
sulted several lists of frameworks (e.g., [14]) in order to (F = 0.57, p = 0.592). Because this appears to be
make our search as thorough as reasonably possible. largely due to the obvious lack of a difference between
Perl and each of the other two languages, we also per-
form a Students t-test for each pair of languages, using
the Bonferroni correction to correct for the fact that we
test 3 separate hypotheses. As expected, we do not find a
4 Results
significant difference between PHP and Perl or between
We look for patterns in the data and analyze it using sta- Perl and Java. We find a statistically significant differ-
tistical techniques. We note that we do not find many ence between PHP and Java (p = 0.033).
statistically significant results due to the limited size of
our data set. 4.2 Vulnerability classes
Figure 2 breaks down the total number of vulnerabilities
4.1 Total number of vulnerabilities into the separate integer-valued vulnerability classes, and
the shaded rows in Table 5 present the data for the binary
Figure 1 displays the total number of vulnerabilities vulnerability classes.
found in each implementation, including both integer-
valued and binary vulnerability classes (we count a bi- XSS. A one-way ANOVA test reveals that the relation-
nary vulnerability as one vulnerability in these aggregate ship between language and number of stored XSS vul-
counts). nerabilities is not statistically significant (F = 0.92, p =
Every implementation had at least one vulnerability. 0.4492). The same is true for reflected XSS (F = 0.43,
This suggests that building secure web applications is p = 0.6689).
difficult, even with a well-defined specification, and even 3 None of our conclusions would differ if we were to exclude this
for a relatively small application. apparent outlier.
 CSRF Session Management Password Storage
Team Language Vulnerable? Framework Vulnerable? Framework Vulnerable? Framework
Number Support Support Support
1 Perl none opt-in opt-in
2 Perl none none none
5 Perl none none opt-out
3 Java manual opt-out none
4 Java always on opt-in opt-in
9 Java none opt-in none
6 PHP none opt-out opt-in
7 PHP none opt-out none
8 PHP none opt-out opt-in
Table 5: Presence or absence of binary vulnerability classes, and framework support for preventing them.

Stored XSS Reflected XSS

10

20
8

15
6

10
4

5
2
0

Java 3 Java 4 Java 9 PHP 6 PHP 7 PHP 8 Perl 1 Perl 2 Perl 5 Java 3 Java 4 Java 9 PHP 6 PHP 7 PHP 8 Perl 1 Perl 2 Perl 5

Manual Both Manual Both

Black-box Black-box

SQL Injection Authentication/Authorization Bypass

3

2
2

1
1
0

Java 3 Java 4 Java 9 PHP 6 PHP 7 PHP 8 Perl 1 Perl 2 Perl 5 Java 3 Java 4 Java 9 PHP 6 PHP 7 PHP 8 Perl 1 Perl 2 Perl 5

Manual Both Manual Both

Black-box Black-box

Figure 2: Vulnerabilities by vulnerability class.

SQL injection. Very few SQL injection vulnerabilities CSRF. As seen in Table 5, all of the PHP and Perl
were found. Only two implementations had any such implementations, and 1 of 3 Java implementations were
vulnerabilities, and only 4 were found in total. The dif- vulnerable to CSRF attacks. Fishers exact test reveals
ference between languages is not statistically significant that the difference between languages is not statistically
(F = 0.70, p = 0.5330). significant (p = 0.25).
Session management. All implementations other than
Authentication and authorization bypass. No such 2 of the 3 Perl implementations were found to implement
vulnerabilities were found in 5 of the 9 implementations. secure session management. That is, the Perl implemen-
Each of the other 4 had only 1 or 2 such vulnerabilities. tations were the only ones with vulnerable session man-
The difference between languages is not statistically sig- agement. Fishers exact test reveals that the difference is
nificant (F = 0.17, p = 0.8503). not statistically significant (p = 0.25).
 Vulnerabilities found by
Team Language Manual Black- Both Total
Number only box only
1 Perl 4 1 0 5
2 Perl 3 1 0 4
5 Perl 12 3 18 33
3 Java 1 7 0 8 20 19 52
4 Java 2 2 0 4
9 Java 5 0 0 5
6 PHP 7 3 0 10
7 PHP 7 3 0 10
8 PHP 11 0 1 12
Table 6: Number of vulnerabilities found in the implementa- Black-box Manual
tions of People by Temperament. The Vulnerabilities found Figure 3: Vulnerabilities found by manual analysis and black-
by columns display the number of vulnerabilities found only box penetration testing.
by manual analysis, only by black-box testing, and by both
techniques, respectively. The final column displays the total
number of vulnerabilities found in each implementation. Manual review is the clear winner for authentication
and authorization bypass and stored XSS vulnerabilities,
while black-box testing finds more reflected XSS and
Insecure password storage. Most of the implementa- SQL injection vulnerabilities. This motivates the need
tions used some form of insecure password storage, rang- for further research and development of better black-box
ing from storing passwords in plaintext to not using a salt penetration testing techniques for stored XSS and au-
before hashing the passwords. One Perl and one Java thentication and authorization bypass vulnerabilities. We
implementation did not violate current best practices for note that recent research has made progress toward find-
password storage. There does not, however, appear to be ing authentication and authorization bypass vulnerabili-
any association between programming language and in- ties [9, 13], but these are not black-box techniques.
secure password storage. Fishers exact test does not find
a statistically significant difference (p = 0.999). Reviewer ability. We now discuss the 20 vulnerabil-
ities that were not found manually. Our analysis of
these vulnerabilities further supports our conclusion that
4.3 Manual review vs. black-box testing black-box testing complements manual review.
Table 6 and Figure 1 list how many vulnerabilities were For 40% (8) of these, the reviewer found at least one
found only by manual analysis, only by black-box test- similar vulnerability in the same implementation. That
ing, and by both techniques. All vulnerabilities in the bi- is, there is evidence that the reviewer had the skills and
nary vulnerability classes were found by manual review, knowledge required to identify these vulnerabilities, but
and none were found by black-box testing. overlooked them. This suggests that we cannot expect a
We observe that manual analysis fared better overall, reviewer to have the consistency of an automated tool.
finding 71 vulnerabilities (including the binary vulner- For another 40%, the vulnerability detected by the tool
ability classes), while black-box testing found only 39. was in framework code, which was not analyzed by the
We also observe that there is very little overlap between reviewer. An automated tool may find vulnerabilities that
the two techniques; the two techniques find different vul- reviewers are not even looking for.
nerabilities. Out of a total of 91 vulnerabilities found by The remaining 20% (4) represent vulnerabilities for
either technique, only 19 were found by both techniques which no similar vulnerabilities were found by the re-
(see Figure 3). This suggests that they are complemen- viewer in the same implementation. It is possible that
tary, and that it may make sense for organizations to use the reviewer lacked the necessary skills or knowledge to
both. find these vulnerabilities.
Organizations commonly use only black-box testing.
These results suggest that on a smaller budget, this prac-
4.4 Framework support
tice makes sense because either technique will find some
vulnerabilities that the other will miss. If, however, an or- We examine whether stronger framework support is as-
ganization can afford the cost of manual review, it should sociated with fewer vulnerabilities. Figure 4 displays the
supplement this with black-box testing. The cost is small relationship for each integer-valued vulnerability class
relative to that of review, and our results suggest that between the level of framework support for that class and
black-box testing will find additional vulnerabilities. the number of vulnerabilities in that class. If for some
Figure 2 reveals that the effectiveness of the two tech- vulnerability class there were an association between the
niques differs depending upon the vulnerability class. level of framework support and the number of vulnerabil-
 Number of Vulnerabilities vs. Framework Support management have vulnerable session management. Sim-
30
ilarly, only the two implementations that have framework
support for CSRF were not found to be vulnerable to
CSRF attacks. Both results were found to be statistically
Number of Vulnerabilities
20

significant using Fishers exact test (p = 0.028 for each).

The difference in results between the integer-valued

10

and binary vulnerability classes suggests that manual

support does not provide much protection, while more
automated support is effective at preventing vulnerabil-
0

no support manual
Framework Support
opt in opt out
ities. During our manual source code review, we fre-
XSS SQL Injection
quently observed that developers were able to correctly
Auth. Bypass use manual support mechanisms in some places, but they
forgot or neglected to do so in other places.
Figure 4: Level of framework support vs. number of vulnera-
bilities for integer-valued vulnerability classes. The area of a Figure 5 presents the results from our identification of
mark scales with the number of observations at its center. the lowest level at which framework support exists that
could have prevented each individual vulnerability (as
ities, we would expect most of the points to be clustered described in Section 3.4).
around (or below) a line with a negative slope.
For each of the three4 classes, we performed a one- It is rare for developers not to use available automatic
way ANOVA test between framework support for the support (the darkest bars in Figure 5b show only 2 such
vulnerability class and number of vulnerabilities in the vulnerabilities), but they commonly fail to use existing
class. None of these results are statistically significant. manual support (the darkest bars in Figure 5a, 37 vul-
Our data set allows us to compare only frameworks nerabilities). In many cases (30 of the 91 vulnerabilities
with no support to frameworks with manual support be- found), the existing manual support was correctly used
cause the implementations in our data set do not use elsewhere. This suggests that no matter how good man-
frameworks with stronger support (with one exception). ual defenses are, they will never be good enough; devel-
We found no significant difference between these levels opers can forget to use even the best manual framework
of support. However, this data set does not allow us to support, even when it is evident that they are aware of it
examine the effect of opt-in, opt-out, or always-on sup- and know how to use it correctly.
port on vulnerability rates. In future work, we would like
to analyze implementations that use frameworks with For both manual and automatic support, the major-
stronger support for these vulnerability classes. Exam- ity of vulnerabilities could have been prevented by sup-
ple frameworks include CodeIgniters xss clean [1], port from another framework for the same language that
Google Ctemplate [3], and Djangos autoescape [2], the implementation used. That is, it appears that strong
all of which provide opt-out support for preventing XSS framework support exists for most vulnerability classes
vulnerabilities. A more diverse data set might reveal re- for each language in this study.
lationships that cannot be gleaned from our current data.
The annotations in Figure 5 point out particular
Table 5 displays the relationship between framework
shortcomings of frameworks for different vulnerability
support and vulnerability status for each of the binary
classes. We did not find any framework that provides
vulnerability classes.
any level of support for sanitizing untrusted output in a
There does not appear to be any relationship for pass-
JavaScript context, which Team 3 failed to do repeatedly,
word storage. Many of the implementations use frame-
leading to 3 reflected XSS vulnerabilities. We were also
works that provide opt-in support for secure password
unable to find a PHP framework that offers automatic
storage, but they do not use this support and are therefore
support for secure password storage, though we were
vulnerable anyway. This highlights the fact that manual
able to find many tutorials on how to correctly (but man-
framework support is only as good as developers aware-
ually) salt and hash passwords in PHP. Finally, we are not
ness of its existence.
aware of any automatic framework support for prevent-
Session management and CSRF do, on the other hand,
ing authorization bypass vulnerabilities. Unlike the other
appear to be in such a relationship. Only the two im-
vulnerability classes we consider, these require correct
plementations that lack framework support for session
policies; in this sense, this vulnerability class is funda-
4 The
level of framework support for stored XSS and reflected XSS mentally different, and harder to tackle, as acknowledged
is identical in each implementation, so we combined these two classes. by recent work [9, 13].
 Where manual support exists to prevent vulnerabilities Where automatic support exists to prevent vulnerabilities
35 35

30 30 Authorization
No known framework No known framework
bypass
Some fwk. for some language Some fwk. for some language
25 Diff. fwk. for language used 25 Diff. fwk. for language used
Newer version of fwk. used Newer version of fwk. used
Framework used Framework used
20 20

15 Reflected XSS in 15 Reflected XSS in Secure password storage

JavaScript context JavaScript context
10 10 Authorization
bypass
5 5

0 0
Java3 Java4 Java9 PHP6 PHP7 PHP8 Perl1 Perl2 Perl5 Java3 Java4 Java9 PHP6 PHP7 PHP8 Perl1 Perl2 Perl5
(a) Manual framework support (b) Automatic framework support
Figure 5: For each vulnerability found, how far developers would have to stray from the technologies they used in order to find
framework support that could have prevented each vulnerability, either manually (left) or automatically (right).

4.5 Limitations of statistical analysis the average number of serious vulnerabilities over the
lifetime of the applications. For example, in their sample
We caution the reader against drawing strong, general- of applications, 57% of the vulnerabilities in JSP appli-
izable conclusions from our statistical analysis, and we cations were XSS vulnerabilities, while only 52% of the
view even our strongest results as merely suggestive but vulnerabilities in Perl applications were XSS vulnerabil-
not conclusive. Although we entered this study with spe- ities. Another finding was that PHP applications were
cific goals and hypotheses (as described in Section 2), found to have an average of 26.6 vulnerabilities over their
results that appear statistically significant may not in fact lifetime, while Perl applications had 44.8 and JSP appli-
be valid they could be due to random chance. cations had 25.8. The report makes no mention of statis-
When testing 20 hypotheses at a 0.05 significance tical significance, but given the size of their data set, one
level, we expect one of them to appear significant purely can expect all of their findings to be statistically signifi-
by chance. We tested 19 hypotheses in this study, and cant (though not necessarily practically significant).
3 of them appeared to be significant. Therefore, we
Walden et al. [25] measured the vulnerability density
should not be surprised if one or two of these seemingly-
of the source code of 14 PHP and 11 Java applications,
significant associations are in fact spurious and due
using different static analysis tools for each set. They
solely to chance. We believe more powerful studies with
found that the Java applications had lower vulnerability
larger data sets are needed to convincingly confirm the
density than the PHP applications, but the result was not
apparent associations we have found.
statistically significant.
While these analyses sample across distinct applica-
5 Related work tions, ours samples across implementations of the same
application. Our data set is smaller, but its collection was
In this section, we survey related work, which falls more controlled. The first study focused on fixed combi-
into 3 categories: (1) studies of the relationship be- nations of programming language and framework (e.g.,
tween programming languages and application secu- Java JSP), and the second did not include a framework
rity, (2) comparisons of the effectiveness of different comparison. Our study focuses separately on language
automated black-box web application penetration test- and framework.
ing tools, and (3) comparisons of different bug- and Dwarampudi et al. [12] compiled a fairly comprehen-
vulnerability-finding techniques. sive list of pros and cons of the offerings of several dif-
Programming languages and security. The 9th edi- ferent programming languages with respect to many lan-
tion of the WhiteHat Website Security Statistic Re- guage features, including security. No experiment or data
port [26] offers what we believe is the best insight to analysis were performed as a part of this effort.
date regarding the relationship between programming Finally, the Plat Forms [19] study (from which the
language and application security. Their data set, which present study acquired its data) performed a shallow se-
includes over 1,500 web applications and over 20,000 curity analysis of the data set. They ran simple black-box
vulnerabilities, was gathered from the penetration-testing tests against the implementations in order to find indica-
service WhiteHat performs for its clients. tions of errors or vulnerabilities, and they found minor
Their report found differences between languages in differences. We greatly extended their study using both
the prevalence of different vulnerability classes as well as white- and black-box techniques to find vulnerabilities.
Automated black-box penetration testing. We are 6 Conclusion and future work
aware of three separate efforts to compare the effective-
ness of different automated black-box web application We have analyzed a data set of 9 implementations of the
security scanners. Suto [22] tested each scanner against same web application to look for security differences as-
the demonstration site of each other scanner and found sociated with programming language, framework, and
differences in the effectiveness of the different tools. His method of finding vulnerabilities. Each implementation
report lists detailed pros and cons of using each tool had at least one vulnerability, which indicates that it is
based on his experience. Bau et al. [5] tested 8 differ- difficult to build a secure web application even a small,
ent scanners in an effort to identify ways in which the well-defined one.
state of the art of black box scanning could be improved. Our results provide little evidence that programming
They found that the scanners tended to perform well on language plays a role in application security, but they
reflected XSS and (first-order) SQL injection vulnera- do suggest that the level of framework support for secu-
bilities, but poorly on second-order vulnerabilities (e.g., rity may influence application security, at least for some
stored XSS). We augment this finding with the result that classes of vulnerabilities. Even the best manual support
manual analysis performs better for stored XSS, authen- is likely not good enough; frameworks should provide
tication and authorization bypass, CSRF, insecure ses- automatic defenses if possible.
sion management, and insecure password storage, and In future work, we would like to evaluate more mod-
black-box testing performs better for reflected XSS and ern frameworks that offer stronger support for prevent-
SQL injection. ing vulnerabilities. We are aware of several frameworks
Doupe et al. [11] evaluated 11 scanners against a that provide automatic support for avoiding many types
web application custom-designed to have many different of XSS vulnerabilities.
crawling challenges and types of vulnerabilities. They We have found evidence that manual code review is
found that the scanners were generally poor at crawling more effective than black-box testing, but combining the
the site, they performed poorly against logic vulner- two techniques is more effective than using either one by
abilities (e.g., application-specific vulnerabilities, which itself. We found that the two techniques fared differently
often include authorization bypass vulnerabilities), and for different classes of vulnerabilities. Black-box testing
that they required their operators to have a lot of knowl- performed better for reflected XSS and SQL injection,
edge and training to be able to use them effectively. while manual review performed better for stored XSS,
authentication and authorization bypass, session man-
While these studies compare several black-box tools agement, CSRF, and insecure password storage. We be-
to one another, we compare the effectiveness of a sin- lieve these findings warrant future research with a larger
gle black-box tool to that of manual source code anal- data set, more reviewers, and more black-box tools.
ysis. Our choice regarding which black-box scanner to
We believe it will be valuable for future research to test
use was based in part on these studies.
the following hypotheses, which were generated from
this exploratory study.
Bug- and vulnerability-finding techniques. Wagner
H1: The practical significance of the difference in
et al. [24] performed a case study against 5 applications
security between applications that use different pro-
in which they analyzed the true- and false-positive rates
gramming languages is negligible. If true, pro-
of three static bug-finding tools and compared manual
grammers need not concern themselves with secu-
source code review to static analysis for one of the 5 ap-
rity when choosing which language to use (subject
plications. This study focused on defects of any type,
to the support offered by frameworks available for
making no specific mention of security vulnerabilities.
that language).
They found that all defects the static analysis tools dis-
covered were also found by the manual review. Our study H2: Stronger, more automatic, framework support
focuses specifically on security vulnerabilities in web ap- for vulnerabilities is associated with fewer vulnera-
plications, and we use a different type of tool in our study bilities. If true, recent advances in framework sup-
than they use in theirs. port for security have been beneficial, and research
into more framework-provided protections should
Two short articles [8, 15] discuss differences between
be pursued.
various tools one might consider using to find vulnera-
bilities in an application. The first lists constraints, pros, H3: Black-box penetration testing tools and manual
and cons of several tools, including source code analysis, source code review tend to find different sets of vul-
dynamic analysis, and black-box scanners. The second nerabilities. If true, organizations can make more
article discusses differences between white- and black- informed decisions regarding their strategy for vul-
box approaches to finding vulnerabilities. nerability remediation.
We see no reason to limit ourselves to exploring these [10] D ONALD , K., V ERVAET, E., AND S TOYANCHEV, R. Spring
hypotheses in the context of web applications; they are Web Flow: Reference Documentation, October 2007. http:
//static.springsource.org/spring-webflow/
equally interesting in the context of mobile applications, docs/1.0.x/reference/index.html.
desktop applications, and network services.
[11] D OUP E , A., C OVA , M., AND V IGNA , G. Why Johnny Cant
Finally, we note that future work in this area may ben- Pentest: An Analysis of Black-box Web Vulnerability Scanners.
efit from additional data sources, such as source code In Proceedings of the Conference on Detection of Intrusions and
repositories. These rich data sets may help us answer Malware and Vulnerability Assessment (DIMVA) (Bonn, Ger-
many, July 2010).
questions about (e.g.,) developers intentions or misun-
derstandings when introducing vulnerabilities and how [12] DWARAMPUDI , V., D HILLON , S. S., S HAH , J., S EBASTIAN ,
N. J., AND K ANIGICHARLA , N. S. Comparative study of the
vulnerabilities are introduced into applications over time. Pros and Cons of Programming languages: Java, Scala, C++,
A deeper understanding of such issues will aid us in de- Haskell, VB.NET, AspectJ, Perl, Ruby, PHP & Scheme. http:
signing new tools and processes that will help developers //arxiv.org/pdf/1008.3431.
write more secure software. [13] F ELMETSGER , V., C AVEDON , L., K RUEGEL , C., AND V IGNA ,
G. Toward Automated Detection of Logic Vulnerabilities in Web
Applications. In Proceedings of the USENIX Security Symposium
Acknowledgments (Washington, DC, August 2010).
[14] JASPAL. The best web development frameworks,
We thank Adrienne Felt, Erika Chin, and the anony- June 2010. http://www.webdesignish.com/
the-best-web-development-frameworks.html.
mous reviewers for their thoughtful comments on earlier
drafts of this paper. We also thank the Plat Forms team [15] M C G RAW, G., AND P OTTER , B. Software security testing. IEEE
Security and Privacy 2 (2004), 8185.
for their hard work in putting together the Plat Forms
contest. This research was partially supported by Na- [16] P ETERS , T. PEP 20 The Zen of Python. http://www.
python.org/dev/peps/pep-0020/.
tional Science Foundation grants CNS-1018924 and
[17] P ORT S WIGGER LTD . Burp Suite Professional. http://www.
CCF-0424422. Matthew Finifter was also supported by portswigger.net/burp/editions.html.
a National Science Graduate Research Fellowship. Any
[18] P RECHELT, L. Plat Forms 2007 task: PbT. Tech. Rep. TR-B-
opinions, findings, conclusions or recommendations ex- 07-10, Freie Universitat Berlin, Institut fur Informatik, Germany,
pressed in this publication are those of the author(s) and January 2007.
do not necessarily reflect the views of the National Sci- [19] P RECHELT, L. Plat Forms: A Web Development Platform Com-
ence Foundation. parison by an Exploratory Experiment Searching for Emergent
Platform Properties. IEEE Transactions on Software Engineer-
ing 99 (2010).
References [20] ROBERTSON , W., AND V IGNA , G. Static Enforcement of Web
Application Integrity Through Strong Typing. In Proceedings
[1] CodeIgniter User Guide Version 1.7.3: Input Class. http: of the USENIX Security Symposium (Montreal, Canada, August
//codeigniter.com/user_guide/libraries/ 2009).
input.html. [21] S HANKAR , U., TALWAR , K., F OSTER , J. S., AND WAGNER ,
[2] django: Built-in template tags and filters. http: D. Detecting Format String Vulnerabilities with Type Qualifiers.
//docs.djangoproject.com/en/dev/ref/ In Proceedings of the 10th USENIX Security Symposium (2001),
templates/builtins. pp. 201220.

[3] google-ctemplate. http://code.google.com/p/ [22] S UTO , L. Analyzing the Accuracy and Time Costs
google-ctemplate/. of Web Application Security Scanners, February 2010.
http://www.ntobjectives.com/files/Accuracy_
[4] perl.org glossary. http://faq.perl.org/glossary. and_Time_Costs_of_Web_App_Scanners.pdf.
html#TMTOWTDI.
[23] T HIEL , F. Personal Communication, November 2009.
[5] BAU , J., B URSZTEIN , E., G UPTA , D., AND M ITCHELL , J.
[24] WAGNER , S., J RJENS , J., KOLLER , C., T RISCHBERGER , P.,
State of the art: Automated black-box web application vulnera-
AND M NCHEN , T. U. Comparing bug finding tools with reviews
bility testing. In 2010 IEEE Symposium on Security and Privacy
and tests. In In Proc. 17th International Conference on Testing
(2010), IEEE, pp. 332345.
of Communicating Systems (TestCom05), volume 3502 of LNCS
[6] B ISHOP, M. Computer Security: Art and Science. Addison- (2005), Springer, pp. 4055.
Wesley Professional, Boston, MA, 2003.
[25] WALDEN , J., D OYLE , M., L ENHOF, R., AND M URRAY, J. Java
[7] C OHEN , J. Best Kept Secrets of Peer Code Review. Smart Bear, vs. PHP: Security Implications of Language Choice for Web Ap-
Inc., Austin, TX, 2006, p. 117. plications. In International Symposium on Engineering Secure
Software and Systems (ESSoS) (February 2010).
[8] C URPHEY, M., AND A RAUJO , R. Web application security as-
sessment tools. IEEE Security and Privacy 4 (2006), 3241. [26] W HITE H AT S ECURITY. WhiteHat Website Security Statistic Re-
port: 9th Edition, May 2010. http://www.whitehatsec.
[9] DALTON , M., KOZYRAKIS , C., AND Z ELDOVICH , N. Nemesis: com/home/resource/stats.html.
Preventing authentication & access control vulnerabilities in web
applications. In USENIX Security Symposium (2009), USENIX
Association, pp. 267282.