Linköping University | Department of Computer and Information Science

Master’s thesis, 30 ECTS | Datateknik

2022 | LIU-IDA/LITH-EX-A--2022/016--SE

Study of the techniques used by

OWASP ZAP for analysis of vul-
nerabilities in web applications
En studie av de tekniker OWASP ZAP använder för att analysera
sårbarheter i webbapplikationer

Adam Jakobsson
Isak Häggström

Supervisor : Sahand Sadjadee

Examiner : Eva Blomqvist

External supervisor : Otto Bergdahl

Linköpings universitet
SE–581 83 Linköping
+46 13 28 10 00 , www.liu.se
Upphovsrätt
Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer-
ingsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko-
pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis-
ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan
användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker-
heten och tillgängligheten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som
god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet
ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman-
nens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se förlagets hemsida
http://www.ep.liu.se/.

Copyright
The publishers will keep this document online on the Internet - or its possible replacement - for a
period of 25 years starting from the date of publication barring exceptional circumstances.
The online availability of the document implies permanent permission for anyone to read, to down-
load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial
research and educational purpose. Subsequent transfers of copyright cannot revoke this permission.
All other uses of the document are conditional upon the consent of the copyright owner. The publisher
has taken technical and administrative measures to assure authenticity, security and accessibility.
According to intellectual property law the author has the right to be mentioned when his/her work
is accessed as described above and to be protected against infringement.
For additional information about the Linköping University Electronic Press and its procedures
for publication and for assurance of document integrity, please refer to its www home page:
http://www.ep.liu.se/.

Adam Jakobsson
©
Isak Häggström
 Abstract

Today, new web applications are made every single day with increasingly more sensitive
data to manage. To ensure that no security vulnerabilities such as data leakage in web
applications exist, developers are using tools such as a web vulnerability scanner. This
type of tool can detect vulnerabilities by automatically finding input fields where data can
be injected and performing different attacks on these fields. One of the most common
web vulnerability scanners is OWASP ZAP. Web vulnerability scanners were first devel-
oped during a time when traditional multi-page applications were prominent. Nowadays,
when modern single-page applications have become the de facto standard, new challenges
for web vulnerability scanners have arisen. These problems include identifying dynam-
ically updated web pages. This thesis aims to evaluate the techniques used by OWASP
ZAP and several other web vulnerability scanners for identifying two of the most common
vulnerabilities, SQL injections and cross-site scripting. This issue is approached by testing
the selected web vulnerability scanners on deliberately vulnerable web applications, to as-
sess the performance and techniques used, and to determine if the performance of OWASP
ZAP could be improved. If an identified technique in another web vulnerability scanner
performed better than the counterpart in OWASP ZAP, it will be implemented in OWASP
ZAP and evaluated. From the tests performed, it could be concluded that the performance
of OWASP ZAP was lacking in the search for input fields, where a depth-first search al-
gorithm was used. The breadth-first search algorithm used by other scanners was shown
to be more effective in specific cases and was therefore implemented in OWASP ZAP. The
result shows that the use case for the two algorithms differs between web applications and
by using both of the algorithms to find vulnerabilities, better performance is achieved.
Acknowledgments

We would like to thank both our external supervisors, Villiam Rydfalk and Otto Bergdahl,
for the help along our last months as students at Linköping University. A thanks also goes
to our supervisor and examiner at the university, Sahand Sadjadee and Eva Blomqvist, for
great feedback to help make this master thesis. Lastly, we would like to say thank you to
MindRoad for letting us do the thesis in their facilities and to play pinball and card games
during the breaks.

iv
Contents

Abstract iii

Acknowledgments iv

Contents v

List of Figures vii

List of Tables viii

1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Theory 4
2.1 Network communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 SQL injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Tautology based attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 Malformed queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.3 Union query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.4 Piggy-backed queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.5 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.6 Stored procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.7 Alternate encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Cross-Site Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.1 Stored XSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.2 Reflected XSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.3 DOM-based XSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 AJAX web application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 Web Vulnerability Scanners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.5.1 The crawler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.5.2 The Fuzzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5.3 The Analyser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.5.4 Work process of a WVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5.5 General limitations of WVSs . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5.6 Issues posed on WVSs when using AJAX . . . . . . . . . . . . . . . . . . 18
2.5.7 Open-source WVSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.5.8 Important techniques in three WVSs . . . . . . . . . . . . . . . . . . . . . 20
2.6 Web applications for testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6.1 Examples of applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

v
3 Related work 25
3.1 Developing WVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.1 Focusing on the fuzzer module . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.2 Focusing on the web crawler . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Performance evaluation of different WVSs . . . . . . . . . . . . . . . . . . . . . . 28

4 Methodology 29
4.1 Literature study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2 Selections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2.1 Selected WVSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2.2 Selected deliberately vulnerable web applications . . . . . . . . . . . . . 30
4.3 Initial test session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4.1 Performance analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4.2 Code analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.4.4 Control test session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5 Results 34
5.1 Initial test session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.1.1 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.2 Techniques identified during code analysis . . . . . . . . . . . . . . . . . . . . . 38
5.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.4 Control test session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6 Discussion 44
6.1 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.1.1 Initial test session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.1.2 Implementation and control test session . . . . . . . . . . . . . . . . . . . 47
6.2 Method discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2.1 Literature study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2.2 Selection of WVSs and applications . . . . . . . . . . . . . . . . . . . . . 49
6.2.3 Initial test session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.2.4 Code analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.3 The work in a wider context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.3.1 Societal aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.3.2 Ethical aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7 Conclusion 52
7.1 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Bibliography 55

A Appendix A 59
A.1 Configurations for scanners during testing . . . . . . . . . . . . . . . . . . . . . 59

vi
List of Figures

2.1 The layers of the OSI reference model. . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 An example of a typical SQLi. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Code for malicious link for tricking user to click on. . . . . . . . . . . . . . . . . . . 9
2.4 Example of a stored reflected XSS attack. . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5 An example of a typical reflected XSS attack. . . . . . . . . . . . . . . . . . . . . . . 10
2.6 An example of a URL context DOM-based XSS attack. . . . . . . . . . . . . . . . . . 11
2.7 The model of a classic web application compared to an AJAX web application. . . 12
2.8 Flow graph for a general Web Crawler. . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.9 Flow graph for a general fuzzing process. . . . . . . . . . . . . . . . . . . . . . . . . 15
2.10 A flow graph example of how the analyser handles requests during a tautology
based attack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.11 Flowchart of a traditional WVS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.12 The graphical user interface for OWASP ZAP. . . . . . . . . . . . . . . . . . . . . . . 19
2.13 The web-based graphical user interface for parts of the landing page in Hackazon. 23
2.14 The web-based graphical user interface of the landing page in Security Shepherd. . 24

4.1 Flowchart of the methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5.1 Crawljax with DFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.2 Crawljax with BFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

6.1 Example of a web application to illustrate Arachni’s issue for identifying XSS vul-
nerabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.2 Juice Shop’s pop-up window seen at first visit. . . . . . . . . . . . . . . . . . . . . . 47

vii
List of Tables

2.1 Example of the different types of SQL injections. . . . . . . . . . . . . . . . . . . . . 8

4.1 Research question mapped to the corresponding part of the method. . . . . . . . . 29

4.2 Selected WVSs for the purpose of performance analysis. . . . . . . . . . . . . . . . . 30
4.3 Selected deliberately vulnerable web applications used for testing. . . . . . . . . . 31

5.1 The total number of found vulnerabilities for each WVS on each deliberately vul-
nerable web application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.2 The total number of found false positive vulnerabilities compared with the num-
ber of reported vulnerabilities for each WVS on each deliberately vulnerable web
application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.3 The total number of found unique vulnerabilities for each WVS on each deliber-
ately vulnerable web application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.4 The number of found real vulnerabilities for each WVS which was not found by
OWASP ZAP on all deliberately vulnerable web application. . . . . . . . . . . . . . 36
5.5 Changes in settings that made OWASP ZAP find vulnerabilities it could not find
in the initial test session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.6 The total number of found vulnerabilities for OWASP ZAP AJAX Spider with BFS
on all deliberately vulnerable web application. . . . . . . . . . . . . . . . . . . . . . 42
5.7 The crawling results for DFS and BFS running for 30 minutes on Hackazon. . . . . 42

viii
 1 Introduction

In the introduction chapter, the reader will be introduced to the motivation and the aim of
this thesis. Then the research questions and the delimitations follow.

1.1 Motivation
The importance of secure web applications has been increasing ever since the birth of the
World Wide Web. One reason for this is that the information sent and stored nowadays is
significantly more sensitive compared to when the World Wide Web was in its early stages.
Furthermore, there is also a greater throughput of data being sent. Parts of this data can be
stored in databases, such as a Structured Query Language (SQL) database. This, and the fact
that the number of connected users has been increasing [53], have made web applications
vulnerable to information leaks.

Open Web Application Security Project (OWASP), a non-profit foundation, lists the ten
most common vulnerabilities for web applications [52]. Third on the list in 2021 and first in
2017 is injection. An injection vulnerability occurs when users are allowed to send untrusted
data to gain or modify sensitive information. There are several types of injections, such as
SQL injection (SQLi) and Cross-Site Scripting (XSS). By doing a SQLi, an attacker can use
SQL queries to gather or manipulate sensitive information in a database. One example of
a real-life SQLi attack is the data breach on Gab (social network), where 70 GB of sensitive
user data was exposed [40]. If an attacker is instead doing an XSS injection, the attacker
injects malicious JavaScript code into a vulnerable input of a web application that a user can
be affected by. A real-life example of an XSS attack occurred back in 2019 when a flaw, in
the popular battle royale game Fortnite [31], allowed an attacker to get access to all the user
accounts on Fortnite. Multiple mitigation techniques can be implemented to minimize these
vulnerabilities, which during development require testing.

There are multiple ways to test if a web application does not contain vulnerabilities. These
test methods can either be manual or automated. In an industrial context, the use of auto-
mated test methods is important, because writing manual tests is a time-consuming task,
meaning it will be expensive. Another aspect which is critical to have in mind is the human
factor. Depending on the skill level of the developer writing the tests, there is a risk that

1
 1.2. Aim

vulnerabilities such as SQLi will be missed. This is where an automated test process can be
used, to minimise the risk. One such tool used for automated tests is a Web Vulnerability
Scanner (WVS). A WVS scans a web application for certain vulnerabilities. It consists of three
modules: a web crawler, a fuzzer and, an analyser. The modules have different responsibili-
ties for finding and analysing possible vulnerabilities. This tool is a type of black-box testing,
meaning the application will be tested without knowledge of the structure and code base. An
alternative test method to use is white-box testing, this is, as the name suggests, the opposite
of black-box testing. The tester knows the code base and structure. White-box testing is great
for code coverage, but there is at least one disadvantage. [56] The tests done by white-box
testing are sensitive to changes in the code base, which requires expensive maintenance.

There are several challenges for a WVS to circumvent. Eriksson et al. [18] described that the
challenges are: handling JavaScript events, the state of the application, the ability to reverse
engineer the execution flow of the application and handling different network requests.
When a JavaScript event is fired, new code can dynamically be inserted into the HTML file of
a single-page web application, this is all done with the help of an Asynchronous JavaScript
and XML (AJAX) engine. This means that the WVS requires having the knowledge to identify
that such events have occurred. It also needs to associate that the newly inserted code should
be rescanned due to new possible injection points. This type of dynamic code insertion is
often used in modern frameworks such as React and Angular, which are frameworks most
used by web developers. Traditionally, the way web applications have been made is by
creating multi-page applications, in which a whole HTML file needs to be downloaded for
each request made by the user. The crawling process in a WVS for a multi-page application
does not require the same amount of knowledge since a new HTML document is loaded to
the web application for each request. This file can easily be added to the list of documents
not scanned.

There exist multiple different WVSs, both commercial and open-source, where OWASP
ZAP is an open-source scanner used by companies such as MindRoad AB. According to
Amankwah et al. [3], both classes of WVSs achieve similar results when trying to detect
different vulnerabilities. According to Matti [37], even if the different WVSs achieve similar
results, different techniques are used to detect the vulnerabilities. Matti concludes that all
the different techniques used have both advantages and disadvantages. What are those
techniques and is it possible to implement a new technique or combine techniques in an
existing WVS to identify more possible vulnerabilities?

1.2 Aim
The goal of the project is to evaluate the current status of WVSs, investigate their perfor-
mance and examine the possibility to implement one of the differences found when compar-
ing WVSs with OWASP ZAP. By following this procedure, the thesis aims to get an insight
into which techniques and modules in OWASP ZAP have the biggest effect on the number of
detected XSS and SQLi vulnerabilities and why.

1.3 Research questions

The thesis aims to answer these research questions:

1. Which different techniques exist to detect SQL injections and XSS in web vulnerabilities
scanners, and how do they theoretically compare to the one existing in OWASP ZAP?

2. How does the detection of vulnerabilities for these techniques differ between single-
page applications and multi-page applications?

2
 1.4. Delimitations

3. How does the performance of some web vulnerability scanners implementing these
techniques compare to the performance of OWASP ZAP?

4. Is it possible to implement these techniques in OWASP ZAP? If so, which modifications

are needed?

1.4 Delimitations
This thesis will focus on comparing only open-source WVSs such as OWASP ZAP, since a
code analysis is going to be performed, access to the source code is a criterion to be met. The
parameters in the performance analysis the thesis focuses on, are the number of the detected
vulnerabilities, the corresponding false positives and the uniquely found vulnerabilities. Fur-
thermore, the thesis focuses on two types of vulnerabilities for web applications, SQLi and
XSS.

3
 2 Theory

The theory chapter aims to present information on the main concepts used in the thesis.

2.1 Network communication

The standardisation of protocols for communication on the internet was first defined by Zim-
mermann [63] in 1980. The paper presents the Open Systems Interconnection (OSI) reference
model. This is a seven-layer model, in which each layer has a specified assignment and there
is only a need for communication between the layer above and below. This communication
is all done using a specified protocol. Each layer is presented in Figure 2.1. However, the
layer in which WVSs operates is the application layer. This layer is used for communication
between user software such as web browsers. There is a multitude of protocols used in the
application layer, although the protocols which are used in web browsers and relevant for
WVSs are the Hypertext Transport Protocol (HTTP) and the HTTP Secure (HTTPS).

Figure 2.1: The layers of the OSI reference model.

4
 2.2. SQL injection

HTTP version 1.0 was defined by Berners-Lee et al. [6]. This protocol works by using a re-
quest/response architecture, in which the client establishes a connection with a server and
sends a request. The request sent consists of a method, such as a POST or a GET, a URI
and the protocol version. The last parameter in an HTTP request is a message in the format
of a Multi-purpose Internet Mail Extension (MIME) which contains request modifiers, client
information and body content if possible. The server, when receiving the request sends a re-
sponse to the client. The response sent from the server consists of a status line, which includes
the version of the protocol and a success/error code. Lastly, the response contains a MIME
message with server information, entity meta-information and body content. As earlier men-
tioned, there are several types of methods that can be included in the request, however, in
this thesis, the relevant methods are the POST and GET requests. The POST request is used
when the user wants to add a resource to the database in the web application, this includes
adding a user and posting a comment. The GET request intends to retrieve resources stored
in the database, such as user information or information about posted comments. HTTPS
is the encrypted version of HTTP, where both requests and responses are encrypted. In this
thesis, all requests will be referred to as HTTP requests, for simplicity.

2.2 SQL injection

SQLi is classified as a code injection attack, where the attacker injects malicious SQL queries
into an input field. An example of a typical SQLi can be seen in Figure 2.2. All types of
SQLi are performed as this example, the difference is the payload used. By doing this, an
attacker can either manipulate the database or show the content of the database according
to Halfond et al. [22]. A database is described by Jatana et al. [27] as a fast application for
storing and gathering data. There are two types of databases, relational and non-relational.
The one relevant to this thesis is the relational database, in which the data is stored in tables.
Each row in the table is called a tuple and each column is called an attribute. In this type of
database, the data can be accessed in multiple ways because of pre-defined relations to each
other. An example could be a database with two tables, one with user information such as an
attribute with the account number. This account number can be used to access the account
information for that specific account number in another table.

Figure 2.2: An example of a typical SQLi.

There exist several types of SQLi attacks, which are described in the following sections.

2.2.1 Tautology based attack

A tautology based attack is described in the paper by Hlaing and Khaing [23] as a SQL query
that is created “cleverly”, so it can be evaluated as true based on its conditional query state-
ment. A typical example of how this SQLi attack looks like when it is injected into an input
field of a web application can be seen in Table 2.1. The example query, from Table 2.1, is the

5
 2.2. SQL injection

tautology "a"="a. This tautology will always be evaluated as true and because of this will get
a match for all tuples in the database. Then the database returns these tuples to the user. In
most cases, a database will return one tuple when it is a subject of a SQLi attack, but in a
tautology based attack, it is, as mentioned before, possible to get all the tuples returned from
the database.

2.2.2 Malformed queries

A malformed query is a syntactically wrong structure of a SQL query, according to Hlaing
and Khaing [23]. By injecting a malformed query, an attacker can get valuable information
from the returned error message. Halfond et al. [22] state that there are three types of errors
that occur in the database if a malformed query is injected. The first one is a syntax error.
This type of error returns valuable information on which parameters are injectable to the
attacker. The second one is a type error. The error message returned by a type error gives
the information, with the help of some conclusions, of which the data type is for a column.
It is also possible to extract data from this error message. The third one is a logical error,
which most of the time shows the name of the tables and the attributes in the current targeted
database from the error message. With this information, the attacker could choose to create
a syntactically correct SQL query to either collect and read confidential information or delete
data from the database. An example of this SQL injection attack can be seen in Table 2.1.

2.2.3 Union query

In a union query attack, the attacker injects the query UNION SELECT to the input field
stated by Hlaing and Khaing [23]. By injecting a UNION SELECT, the goal of the attacker is
to trick the database to return data from other tables or attributes which is not intended by
the original query. An example of this attack can be seen in Table 2.1.

2.2.4 Piggy-backed queries

Hlaing and Khaing [23] describe a piggy-backed query attack, as an attack where more SQL
queries are added to the existing one in the back end. The intention of the attacker is not
to modify the SQL query as in the previous SQL injection attacks, but instead to finish the
original query and execute its own queries. As claimed by Halfond et al. [22], the attacker
can abuse the delimiter “;”. The “;” delimiter is an indication that the SQL query is done,
and because of this, the attacker can add additional SQL queries. The added SQL queries are
constructed to either fetch, add, modify or delete data. A deletion example of a piggy-backed
query can be seen in Table 2.1.

2.2.5 Inference
An inference is a technique which changes the behaviour of the database based on the query
an attacker is injecting, according to Hlaing and Khaing [23]. There exist two widespread
types of inference techniques. The first one is called blind injection, which is described by
Halfond et al. [22]. The way it works is that the developers have secured the error message,
provided by the database, by hiding the valuable information from the attacker. The attacker
might still be able to exploit the database by injecting Boolean statements and studying how
the response from the database is affecting the web application. Based on which Boolean
statement was used and how the response from the database affected the web application,
the attacker can make analyses of which kinds of queries can be used to exploit the database.

The second inference technique is a timing attack. Halfond et al. [22] stated that the at-
tacker can gather information by monitoring the time delay of the database responses.
Depending on what kind of logical data is injected by the attacker, timing attacks can either

6
 2.2. SQL injection

use “if-then” statements to make the database execute the query for a longer time or use a
time delay statement. For a time-based attack, there is a SLEEP statement injected to cause a
delay. The attacker can then monitor if the web page has a delay or not when it is updating.
For both of these two inference techniques, there is an example shown in Table 2.1.

2.2.6 Stored procedures

This type of SQLi, stored procedure, is presented by Hlaing and Khaing [23] as the attacker
attempts to execute existing stored procedures in the database. This is achieved by injecting
queries to trigger the execution. According to Halfond et al. [22] it is possible to exploit this if
the attacker knows which database management system (DBMS) is used since the DBMS has
several predefined stored procedures generated by the database vendor. Such types of DBMS
could be MySQL and PostgreSQL (if the database is a relational database) or MongoDB and
DynamoDB (if the database is a non-relational database). In Table 2.1 an example of a stored
procedure attack can be seen.

2.2.7 Alternate encoding

Halfond et al. [22] describes alternate encoding as not a specific SQLi, they describe it rather
as several possibilities to avoid the detection of a SQLi. An alternate encoding attack is when
the injected text is modified to bypass the different detection techniques of a SQLi attack.
This is often used as an evasion technique since the most common detection technique is to
inspect the input after known “evil characters” that cause a database to be exploited. The
attacker could modify the injected text to hexadecimal, ASCII or Unicode characters. For this
to work the detection techniques do not inspect all kinds of encoded text and thereby allow
some alternate encoding attacks to bypass. In Table 2.1 can an example of alternate encoding
attack be seen.

7
 2.2. SQL injection

Table 2.1: Example of the different types of SQL injections.

Type of SQLi Input for the SQL query executed Result of the
attack input field at server side SQLi attack
Tautology " OR "a"="a SELECT * FROM user All usernames and
based attack " OR "a"="a WHERE username="" passwords are
OR "a"="a" AND fetched.
password="" OR "a"="a"
Malformed aaaa" SELECT * FROM user The error message
queries WHERE username= sent back from the
"aaaa"" AND server is showing
password= the password.
Union query UNION SELECT SELECT * FROM user The password for
password FROM WHERE username="" the admin is
user WHERE UNION SELECT fetched.
username= password FROM
"admin" -- user WHERE
username= "admin"
-- AND password=""
Piggy-backed ; DROP TABLE SELECT * FROM user All tuples of the table
queries user -- WHERE username="" user will be deleted.
AND password="";
DROP TABLE user --
Inference - "abc" OR "a"="a" -- SELECT * FROM user Two different SQL
Blind injection "abc" AND "a"="b" -- WHERE username= queries to analyse
"abc" OR "a"="a"-- the different
AND password="" responses.
SELECT * FROM user
WHERE username=
"abs" AND "a"="b" --
AND password=""
Inference - IF(((SELECT SELECT * FROM user This query has a
Timing attack UPPER(MID( WHERE username= delay when the first
password,1,1)) IF(((SELECT UPPER token in the admins
FROM user (MID(password,1,1)) password is A. Else,
WHERE FROM user WHERE 1 is returned.
username="admin" username="admin" (OBS, SLEEP only
="A"), SLEEP(5), 1) ="A"), SLEEP(5), 1) works in MySQL.)
Stored ; SHUTDOWN; -- SELECT * FROM user The query should
procedures WHERE username=""; shutdown the
SHUTDOWN; -- database if the query
password="" is calling a stored
procedure to check
the credentials.
Alternate ; exec(0x7368757 SELECT * FROM user The hexadecimal
encoding 4646f776e) -- WHERE username=""; value being executed
exec(char( is a SHUTDOWN
0x73687574646f776e command, which is
)); -- password="" encoded back to its
plaintext character.

8
 2.3. Cross-Site Scripting

2.3 Cross-Site Scripting

XSS is just like SQLi a type of injection attack but instead based on a scripting language such
as JavaScript. According to Cook, [11], an XSS attack is not an attack on the web application
that a SQLi attack is. An XSS attack is an attack on the user and indirectly endangers the
web application. Usually, an XSS attack is performed by injecting a script to an input field on
the client-side of a web application. The most common example of trying to identify an XSS
vulnerability is to inject the following:

ă script ą alert(”XSS”) ă /script ą

The result of a successful injection of this type is that an alert box with the XSS string is
shown to the user on the web application.

There exists three different types of XSS attacks: stored, reflected and Document Object
Model-based (DOM-based). In the following sections, a description of how these XSS attacks
are executed, and their characteristics are presented.

2.3.1 Stored XSS

According to Kirda et al. [32], a stored XSS attack occurs when a malicious script (most of the
time JavaScript) is stored on the server of the targeted web application. By storing the script,
it will not be executed or affect the web application until a user triggers it. An example of this
could be when a user is visiting a popular web application called www.safecite.com which is a
bank that handles sensitive information. This web application uses a cookie to store sensitive
information about the session in the user’s browser. The web application is using the same-
origin policy, which implies that the cookie is only accessible to JavaScript downloaded from
a safesite.com server. Nonetheless, the user could also browse on a malicious web application,
like www.evil.com. When browsing this web application, the user could be tricked to click on
a link, shown in Figure 2.3, that sends an HTTP request to www.safesite.com server requesting
the web application. The web application safesite.com handles this request by checking if the
request contains the right resources. The host for safesite.com will not find the requested page
and henceforth return an error message. Furthermore, the server for safesite.com may include
the requested file name in the error message to point out which file the host could not find.
 
<a href="http://www.safesite.com/
<script>
document.location=’http://www.evil.com/cookie-stealer.php?’+document.
cookie
</script>">
Click here to withdrawn money from your account.
</a>
 

Figure 2.3: Code for malicious link for tricking user to click on.

If this situation occurs, the file name (which in this case is a script) is sent from
www.safesite.com to the user’s browser and the script executes in the cohesion to safesite.com
origin. The cookie is saving the user’s bank credentials and its authenticated session when
the script executes and then sending this data back in a parameter to evil.com. The cookie can
then be abused by the owner of evil.com by collecting money from the user without the user’s
awareness. A flow graph of this example can be seen in Figure 2.4.

9
 2.3. Cross-Site Scripting

Figure 2.4: Example of a stored reflected XSS attack.

2.3.2 Reflected XSS

In a reflected XSS attack, the injected payload (the data given by the user/attacker) is included
in the immediate response when sending an HTTP request according to Kirda et al. [32]. An
example, shown in Figure 2.5, could be an attacker sending a phishing email, which is an
email designed by the attacker to trick the victim into either downloading certain files or
giving out sensitive information.

Figure 2.5: An example of a typical reflected XSS attack.

When the victim clicks the URL it will be sent to the server and if the website is vulnerable to
reflected XSS the code will be sent back to the victim. Meaning, that the website displayed to
the victim will contain the JavaScript code, and it will be executed when the browser parses
the webpage. A result of this execution could be user information being sent to the attacker.

2.3.3 DOM-based XSS

DOM-based XSS attack was introduced by Klein [33] in 2005, where the fundamental charac-
teristics of a DOM-based XSS attack were described. The DOM is, according to Wood et al.
[61], an application for defining the logical structure of HTML and XML documents repre-
senting a web application. The DOM also defines how a document is accessed and manipu-
lated. The structure of the DOM is like a tree, where a node can contain an HTML element.
Continuing with the attack, according to Klein and the paper by Lekies et al. [35], a DOM-
based XSS attack is caused by the client-side code being improperly designed. In comparison
to the previous two XSS attacks where it rather depends on improper server-side code. An
attacker can exploit a DOM-based XSS vulnerability by injecting an enormous collection of
different attack vectors, which includes the malicious payload. The web application then

10
 2.4. AJAX web application

processes the inputs from the attacker if it has such workflow and could manage one of the
inputs to be converted from string to code and place the code in the dynamic DOM. An ex-
ample of a URL context DOM-based XSS attack (described below) can be seen in Figure 2.6.
From the empirical study Lekies et al. made, they concluded that if the web application does
not sanitise the inputs, it could be possible that an attacker gets its malicious script inserted
into the web application. Henceforth, Lekies et al. suggest that there exist different types of
DOM-based XSS attacks since there are different ways to convert the injected string to code.

Figure 2.6: An example of a URL context DOM-based XSS attack.

The first type is the HTML context. One of the most common ways to insert HTML code in a
web application is through functions as document.write or innerHTML. When these functions
are executed by the web application, the browser will interpret the input string into HTML
code and place the code into a predetermined position in the dynamic DOM. If there is a lack
of sanitation of the input, an attacker has the possibility to inject HTML tags such as <script>,
which allows the attacker’s JavaScript code to be executed.

The second type is the JavaScript context. Another function that is vulnerable to DOM-
based XSS attacks is the eval function. The eval function works by taking the string from
an input and interprets the string into JavaScript code which will be executed. It is not
only eval which does this conversion of string to JavaScript, script.innerText, script.text and
script.textContent are also included.

The third type is the URL context. When an attacker tries to inject a string into the URL
field, the interpretation from string to code does not happen. Despite this, it is still possible
for an attacker to abuse the URL field. If the attacker can control the URL field, the attacker
could use schemes such as JavaScript and data to, for example, execute JavaScript code. If
the attacker only controls a part of the URL field, the attacker could use approaches such as
redirecting the user to a malicious web application, using social engineering attacks such as
phishing and rarely executing JavaScript code.

The last type is the Other contexts. The previous types have been about the possibility
for the attacker to execute code. There are other contexts that are vulnerable for DOM-based
XSS attacks such as the document.cookie, the Web Storage API, the setAttribute function and the
postMessage function. The challenge with DOM-based XSS attacks and their different types
is the difficulty to detect the injected script since the attacks abuse the dynamic DOM on the
client-side of a web application.

2.4 AJAX web application

AJAX, short for Asynchronous JavaScript And XML, is a set of technologies to let a user inter-
act with a web application asynchronously according to Garret [20]. The difference between
a typical web application and an AJAX web application is that an AJAX engine is introduced

11
 2.5. Web Vulnerability Scanners

between the user interface and the server. The main task for the engine is to render the user
interface and communicate with the server. As an example, if a user interacts with the user
interface which requires an HTTP request to be sent, the AJAX engine first gets a JavaScript
call from the user interface. It handles this call, letting the user continue to interact with the
user interface. If the request does not require a call to the server, the AJAX engine handles
the request by itself. Otherwise, it sends an HTTP request to the server, which sends back an
XML response. The AJAX engine handles the XML data and puts the HTML or CSS proper-
ties into the web application. See Figure 2.7 for a model comparing a classic web application
to an AJAX web application.

Figure 2.7: The model of a classic web application compared to an AJAX web application.

2.5 Web Vulnerability Scanners

As described earlier in Section 1.1, a WVS is a tool used to scan for vulnerabilities in a web
application. It consists of three modules: a web crawler, a fuzzer and an analyser [25]. On a
higher level, the web crawler is used to analyse the web application to identify where HTTP
requests can be sent. When the places for inputs and HTTP requests have been identified, the
fuzzer is used to send and generate possible attack patterns. Lastly, the result is evaluated by
the analyser to determine whether the pattern is a successful attack or not.

12
 2.5. Web Vulnerability Scanners

2.5.1 The crawler

As described in the overview above, the web crawler automatically searches the web applica-
tion for URLs when receiving what is known as seed URLs. A web crawler has traditionally
been used in search engines such as Google and Bing. The work process of a crawler starts
by downloading the web page with the seed URL and searching the HTML DOM to find
associated URLs. These modules are usually called the Downloader and Parser [2]. Before
the web page is downloaded, a check needs to be done to determine if the page is allowed to
be downloaded or not. During this process, a control is performed to see if there is a robot
exclusion file or any other type of exclusion instructions provided in the header. A robots.txt
file is used to off-load the web crawler by notifying which URLs the crawler is allowed or
disallowed to access on the website. Another type of file which could be analysed during
this process is the XML sitemap. This file is a list of important web pages, and the file could
also be used to determine the structure of the website.

The URLs found in the HTML DOM from the seed URL are put into a list, called crawl
frontier, of URLs to visit. The order in which the links are put into is dependent on a set of
rules and policies. Each URL is then visited individually, where the same process is done
by parsing the HTML DOM and URLs are extracted and put in the crawl frontier. The flow
graph for a general web crawler can be seen in Figure 2.8.

Figure 2.8: Flow graph for a general Web Crawler.

13
 2.5. Web Vulnerability Scanners

There are multiple types of web crawlers available, a few are described below.

The Breadth-First Search (BFS) Crawler is described by Shkapenyuk and Suel [49] as using
the traditional breadth-first algorithm when picking URLs from the frontier list. This means
the crawler will go through each HTML DOM of the first seed URL, before starting to search
the URLs found during parsing. In reality, a strict BFS is not the actual outcome when
implementing this type of crawler. The traversing often differs dependent on policies, such
as what web pages are deemed more important.

Another type of algorithm which can be used during crawling is The Depth First Search
(DFS) Algorithm. It is described by Linxuan et al. [62] to be executed by first parsing the seed
URL and follow one of the child URLs, if no other specifications are provided it will follow
the left most child. The crawler will then traverse deep until no child URLs can be found.
This will be done for all child URLs of the initial seed URL.

According to Shrivastava and Sahai [50] there are advantages of using a BFS when crawling
an application with one or two-level depth. A BFS crawler finds the minimal path if several
paths are directed to the same internal page within an application. The disadvantages are
few, it takes more time to crawl an application if it consists of a deep level, and it becomes
more inefficient if the number of branches is large. Shrivastava and Sahai continue with the
advantages of a DFS crawler. An advantage is that it is better suited for crawling an applica-
tion with a high depth level, which ensures that all pages are visited. The disadvantage is if
there are many branches, it could enter an infinite loop.

The Incremental Web Crawler is a crawler designed according to Cho and Garcia-Molina
[9] to determine if the web page already has been downloaded. If so, the crawler will update
the page rather than download it again. This crawler type needs to have a way of determin-
ing if the page has been changed since it was downloaded.

The Focused Web Crawler has three important components. The first component is the
classifier, which determines the importance of the web page currently being searched to
decide if hyperlinks should be followed. The next component is a distiller, which prioritises
the found URLs. The last component is the actual crawler, which has controls for configuring
the priority. With the help of these components, a Focused Crawler aims to only download
pages within a specified topic or with similarities.

Parallel Crawlers are defined as multiple processes running the crawling task. This is done to
accelerate the speed at which the crawling task is done. The use of parallel crawlers helps
to reduce the amount of time for the crawling process, but to decrease the time, even more,
Distributed Web Crawler has been introduced [2]. As per the definition of distributed systems,
a URL server administers URLs to multiple crawlers running simultaneously. Each crawler
downloads the page in parallel, the pages get sent to a central indexer which extracts the
URLs and sends the result back to each crawler via the URL server.

2.5.2 The Fuzzer

The definition of fuzzing is an automated testing technique that randomizes the input to pro-
grams and focuses on boundary cases using invalid data [41]. Fuzzing was firstly described
by Miller et al. [38] for a program called The Fuzz. This program generated random characters
as input to test UNIX utilities. There are two traditional types of fuzzing techniques, which
are listed below:

14
 2.5. Web Vulnerability Scanners

Mutation based fuzzing is defined to be the process of collecting valid data, this could be files,
input strings etc. and randomly or heuristically changing this data. The data is then used
as input to the System Under Test (SUT) and the behaviour is observed [39]. An example of
mutation-based fuzzing with a heuristic could be to change the length of an input string.

The other traditional type of fuzzing is Generation based fuzzing which uses a specifica-
tion, describing the composition of the input such as the file format, to automatically create
semi-valid inputs. This is often combined with different types of fuzzing heuristics, such as
very long empty strings. Generally, mutation-based fuzzing is seen to be easier to get started
with, since not a lot of information is needed to know about the SUT. However, the result of a
mutation-based fuzzing tends to have a lower code coverage when compared to generation
based fuzzing [39]. Figure 2.9 shows the flow graph for a general fuzzing process in which
the execution is stopped when an error occurs and a bug is found.

Figure 2.9: Flow graph for a general fuzzing process.

It is also possible to categorise the fuzzer in the same way as Pham et al. [47] do, which is a
black-box-, white-box- and grey-box fuzzing.

Black-box fuzzing is a type of black-box testing. The definition of black-box testing is a

testing technique which has no acquaintanceship with the code base, according to Ehmer
Khan and Khan [30]. Henceforth, describe Verma et al. [55] that black-box testing decides the
functionality of a software application by using valid and invalid inputs to get expected out-
puts. The advantages of using black-box testing are: the efficiency of testing large code bases
such as the fast development of test cases, the tester’s knowledge of the software application
does not need to be large and the tester and the developer are independent of each other.
The disadvantages of using black-box testing are the following: there is a low code cover-
age since the test cases do not cover all test scenarios, the testing sessions are inefficient, and
it is difficult to create test cases without comprehensible specifications from the test scenarios.

White-box fuzzing is a type of white-box testing. The definition of white-box testing is to

examine the code base of a software application, according to Ehmer Khan and Khan [30].
By knowing the application, test cases can be derived to test the control flow and the design.
In comparison to black-box testing, the tester knows the code base in white-box testing.The
advantages of using white-box testing are that it guarantees high code coverage, it can find
errors in hidden code, and it can be benefited from side effects. The disadvantages of white-
box testing are that since the tester has knowledge of the code, it requires high skills and is
expensive. Even if there is high code coverage, not all execution paths are tested, and it could
miss errors in the code. There is a possibility that code parts, that are unconnected to the
main code, are not tested.

Grey-box fuzzing is a type of grey-box testing. According to Ehmer Khan and Khan [30],
the definition is a combination of white-box and black-box testing. This means the tester has
partial knowledge about the internal data structures and algorithms used through documen-
tation. Examples of such documentation are an Architectural Model, a Unified Modelling
Language (UML) or a State Model. When designing and writing the test cases for the SUT,

15
 2.5. Web Vulnerability Scanners

white-box testing methods are used to analyse and write the tests. However, the tests per-
formed need to be run through the interfaces exposed to the user (black-box testing). As
grey-box testing is a combination of black- and white-box testing, the advantage of using it
is a merge of the two. However, it still exists disadvantages to using grey-box testing such as
the test coverage, which is limited due to the source code not being available. The same goes
for program paths, and it might lead to redundant test cases.

2.5.3 The Analyser

In a WVS, the task for the analyser is to decide if there exists a vulnerability in the web
application. This is decided based on the response given back from the attack the fuzzer
performed on the web application, referring to the paper of Doupé et al. [15]. An example
is that if the fuzzer performs a tautology based attack (described in Section 2.2.1 and shown
in Figure 2.10) on a web application, the response is a database error message. The analyser
then tries to find common patterns in the error messages that indicate that this type of SQLi
attack is exploitable in the web application.

Figure 2.10: A flow graph example of how the analyser handles requests during a tautology
based attack.

Another aspect to take into consideration about the analyser is the possibility to evaluate
the response as a vulnerability, even if it is not, or not evaluate it as a vulnerability, even if
it is. This is called false positive and false negative. Both false positive and false negative
can be a problem for developers using a WVS. In the false negative case, it is obvious that
vulnerabilities that are not found by the WVS can be used by an attacker to exploit the web
application. In the false positive case, consider that a developer gets 100 hits by the WVS. If
90 of them are classed as false positives, it is a time-consuming task for the developer to go
through all the found vulnerabilities and decide which of them that are real or not.

16
 2.5. Web Vulnerability Scanners

2.5.4 Work process of a WVS

A summarisation of the earlier description of the working process of a WVS starts with the
web crawler. The crawler browses a web application to gather all possible entry points by
starting with a seed, according to Kals et al. [29]. The seed is the landing page for a web
application. From the seed, the crawler can search through the HTML DOM tree for HTTP
requests and internal page URLs, as described by Koswara and Asnar [34]. The URLs are
placed in two different lists, the first list containing all URLs which have not been searched.
Each URL in this list is then put through the same procedure as with the seed URL until the
list is empty. The second list, which is being added to continually during the search, contains
all URLs parsed by the crawler.

The next step in the working process is the fuzzer. In a WVS, the fuzzer is using the
URL list, returned from the crawler, to send an attack pattern [25]. The intention of the attack
pattern is to trigger the vulnerability, for example, a SQLi attack.

The last step is the analyser. After an attack pattern has been performed, the analyser
needs to parse the response message from the application [29]. After the parsing, the anal-
yser determines if the attack was successfully performed based on different keywords and
criteria. A successfully performed attack depends on a confidence value because detection
of false positive attacks is possible. An example of a response is a database error message,
which is an indication that a SQLi vulnerability is present in the web application. See Figure
2.11 for a flowchart of the above described work process.

Figure 2.11: Flowchart of a traditional WVS.

2.5.5 General limitations of WVSs

As can be concluded from Section 2.5.4, the result from a WVS is completely dependent on
the crawling process. If a URL is not found during this process, the vulnerabilities associated
with that specific URL will never be able to be detected. A problem with all type of automated
black-box testing, including WVSs, is how to test parts of an application which needs some
type of user authentication. WVSs try to solve this by giving the option to configure for
logins, which except for making the testing process longer and in need of a manual hand,
also is not always possible. Such a case could be when two-factor authentication is used.
Blindly following URLs might make the crawling process perform a logout, which also needs
to be configured for. One way to configure for logouts is to add URLs that the crawler can
not visit. There is also a possibility of adding URLs that are deemed important to visit, which
would make the crawling process prioritise these URLs. Lastly, the number and the types of
vulnerabilities found can differ a lot between different WVSs, meaning it can be a good idea

17
 2.5. Web Vulnerability Scanners

to run multiple scans using different WVSs [48]. This just as with all other options presented
in this section makes the scanning process more difficult.

2.5.6 Issues posed on WVSs when using AJAX

As mentioned in Section 2.4, the response from the server in an AJAX application returns an
XML response. Koswara and Asnar [34] state that this is a problem for the traditional crawler.
The crawler expects a complete web page to be returned from the server, to extract the URLs
from tags as <a> and <form> located in the HTML document. In an AJAX web application,
these kinds of tags are embedded in scripts that the crawler can not collect and extract the
URLs. By not being able to collect these URLs, important parts of the web application could
be missed, which possible contain vulnerabilities. This is one of two issues regarding WVSs
and AJAX web applications.

The other issue according to Koswara and Asnar [34] is that in an AJAX web applica-
tion, the client can send multiple requests to the server asynchronously. The response from
one request can then be used to update a part of the web page. This broadens the possibility
of developing single-page web applications, where the DOM is dynamically updated. How-
ever, it also creates problems for traditional crawlers to extract URLs in the updated parts of
the web application. Hence, updated parts execute first at the client-side and cause AJAX
requests to appear.

2.5.7 Open-source WVSs

There exists a lot of open-source WVSs. In the following section, a selection of a few of these
WVSs is briefly described. These have been picked from earlier work in the field based on
the performance, and a few of these will be used during the rest of the thesis. The selection
used in this thesis is shown and motivated in Section 4.2.1.

Arachni
Arachni is a tool to help penetration testers to find vulnerabilities in their web application
[4]. It was first released in 2015 and was continuously updated until 2017, however, the
development started again in 2022. Arachni is built on the Ruby framework and can learn
the behaviour of the web application while crawling. It can perform a meta-analysis based
on several factors to minimise the false positive rate of the found vulnerabilities. Arachni
supports the crawling of AJAX web applications by detecting changes in the DOM and
therefore increasing the code coverage. Arachni offers both a command-line interface and a
graphical user interface.

Black Widow
Black Widow [7] is an academic WVS, meaning it was created to use as a proof-of-concept
WVS for a research paper [18]. This tool is written in Python and based on the Selenium
tool, where the crawling process is run with help of a Chromedriver. Black Widow was
created to improve the crawling process of a WVS. This was done by creating a graph for
each found URL. There the crawler easier could traverse back and forward to revisit different
URLs, where it could be more URLs to visit. Further reading of the crawling process in
Black Widow can be found in Section 3.1.2. As mentioned before, Black Widow is a proof-of-
concept WWS for a research paper and therefore, it is only auditing for XSS vulnerabilities.
Black Widow is offering a command-line interface.

Enemy-of-the-state
Enemy-of-the-state [17] is like Black Widow, an academic WVS created by Doupé et al. [14]
in 2012. It was created with the intention to deduce the internal state machine of a web

18
 2.5. Web Vulnerability Scanners

application. Doupé et al. achieved to deduce the internal state during the crawling process,
acknowledge the different outputs of the web application and create a model where the
different states of the web application are represented. To invoke JavaScript code, Doupé
et al. are using htmlunit in their crawling process to support updates in the dynamic DOM.
However, htmlunit is not supporting updates in the DOM generated by the AJAX technology.
Enemy-of-the-state offers, just like Black Widow, only a command-line interface.

IronWASP
IronWASP [26], short for Iron Web Application Advanced Security testing Platform, is a WVS
written in Python and Ruby [13]. IronWASP was designed to be highly customisable, which
enables the possibility for developers to customise the scanner to their desire. IronWASP also
supports the evaluation of false positive and false negative vulnerabilities. IronWASP offers
a graphical user interface.

OWASP ZAP
OWASP ZAP [46] is created by the OWASP foundation and is the most used and maintained
open-source WVS. OWASP ZAP offers to crawl a web application with both a traditional
spider and an AJAX spider and perform attacks to find vulnerabilities listed in the OWASP
top ten most common vulnerabilities 2021 [52]. There is also a script feature in OWASP ZAP,
allowing the user to script the authentication phase among other things. OWASP ZAP offers
a user interface, which can be seen in Figure 2.12. It also exists a command-line interface.

Figure 2.12: The graphical user interface for OWASP ZAP.

Skipfish
Skipfish [51] is an open-source WVS created and maintained by Zalewski et al. The WVS
performs its scans by providing an interactive sitemap of the scanning web application by
performing a recursive crawl. The resulting sitemap is then attacked to find possible vulner-
abilities. Skipfish offers a command-line interface.

Vega
Vega [54] is an open-source WVS entirely written in Java and was created in 2010. Vega
offers two ways to scan a web application. It can do it both manually and automated. For
the manual scan, Vega is using a proxy to connect against the browser, which allows the user
to browse the web application itself. The automated scan works as it is using a traditional
crawler to find URLs it later can attack and hopefully find vulnerabilities. Vega offers both a
command-line interface and a graphical user interface.

19
 2.5. Web Vulnerability Scanners

VulScan
VulScan is an academic WVS created by Huang et al. [24] in 2017. VulScan was developed to
use an evasion technique to generate different attacks to find vulnerabilities in web applica-
tions. Huang et al. goals were to find more vulnerabilities and increase the test coverage. A
more detailed description can be seen in Section 3.1.1.

Wapiti
Wapiti [59] is a WVS written in Python. Wapiti works by crawling the targeted web applica-
tion and extracting URLs and forms. Then, it sends different payloads to find error messages
or odd behaviour of the web application, which possibly is an indication that a vulnerability
exists. Wapiti only offers a command-line interface.

W3AF
W3AF [57] is a WVS written in Python and was first launched in 2011. W3AF offers a number
of different plugins for the user. This allows the user to enable its own desired plugins and
create a profile to save its customised plugins for later scans. In the plugin settings, the user
needs to enable the traditional spider for crawling the web application. The user then needs
to enable the audit plugin to identify if any vulnerabilities exist in the web application, and
with this setting, the user can specify which vulnerabilities W3AF will audit. W3AF offers
both a command-line interface and a graphical user interface.

2.5.8 Important techniques in three WVSs

In this section, the important techniques used in three WVSs are stated. These tools were
chosen based on a code analysis of a selected number of WVSs done in this thesis. This
is presented in this chapter to describe some findings of the code analysis, in order to give
the reader a better understanding of how the techniques work in each WVS. This is done by
listing the different parts of a WVS for each of the WVSs which performed the best (according
to the criteria stated in Section 1.4) during the test session and these are Black Widow, Arachni
and OWASP ZAP’s AJAX Spider. The three WVSs will be split up between the modules:
crawler, fuzzer and analyser. An aspect which unifies all the different crawlers implemented
in the different scanners is the fact that they all try to build a graph. Each node in the graph
is a new state and the edges are the action fired to achieve this state. All scanners also uses
XML Path language [10]. This language is used to express where a specified HTML element
can be found in the DOM.

• Crawler

– Arachni: The crawler which is used in Arachni, is a BFS crawler. This crawler
classifies a new state as a new URL visited, or after the crawler has clicked on an
action element. Arachni uses the URL or the page (DOM) to put in a queue. URLs
and pages placed in this queue are hashed and compared each time action has
happened to identify if the new state is a clone and should not be added to the
queue. Arachni is trying to avoid infinite loops by checking and extracting the
URI. From the URI, a regex pattern is created and placed in a dictionary as a key
and a counter integer is set as a value. Then, if the same regex pattern is created
several times, the counter is decreased and when it reaches zero, Arachni defines
the URI as an infinite loop and stops to visit the URL. One thing to note is that this
crawler is scanning for all types of "Input" elements.

– Black Widow: This proof of concept WVS is using a BFS crawler. Black Widow
defines a new state as a new URL visited or when an event is executed. When
the seed URL first is analysed, Black Widow extract all URLs, forms, events, and

20
 2.5. Web Vulnerability Scanners

iframes and puts them in a queue to later be executed. When they are executed,
the new node is connected with the parent node via an edge. This edge consists as
earlier mentioned of the action which leads to the new state. If the new node is a
clone, all events extracted from the DOM that has already been “visited” will not
be added to the new state. Since Black Widow is a BFS crawler, when the clone is
visited again all the events will already have been visited and no events will be
fired. To avoid getting stuck in an infinite loop, Black Widow has implemented
a technique where two pages are defined to be similar if the query parameters
in the URI are the only difference. If this is identified, a counter is increased and
checked to see if a threshold has been reached. This threshold can be changed.
Black Widow just like Arachni, is scanning for all types of "Input" elements.

– OWASP ZAP: The AJAX crawler used in OWASP ZAP is a DFS crawler. A new
state is defined as a change in the DOM. This change is calculated when an action
is fired, the new DOM will be converted to a string and compared with the DOM
before the action was fired. The comparison is done using the Levenshtein dis-
tance. This method calculates the amount of change needed to get from one string
to the other [36]. The default threshold used in OWASP ZAP for what is consid-
ered to be a new state is 0, this means any change to the DOM is considered to
be a new state. There is also a depth limit, meaning the maximum steps from the
index state which is allowed to be taken. This number is set to 10 by default, but
can be changed by the user. OWASP ZAP handles "Input" elements different by
default, and is only scanning for "Input" elements of type "Submit" and "Button".
This can just be like the threshold being changed by the user. The anchor elements
with a hyperlink (href) attribute are prioritised to be handled by the crawler when
scanning a new state, followed by the "Input" elements.

• Fuzzer

– Arachni: The fuzzing in Arachni is done during the crawling phase. This applies
to both SQLi and XSS vulnerabilities. An aspect to note is that each insertion of
a script to identify XSS vulnerabilities has an ID. There is a multitude of different
XSS scripts injected depending on the tag of the HTML element, but they all have
an ID. There are multiple different types of SQLi, which are all described in Section
2.2, but Arachni only uses two different payloads during the fuzzing phase. The
two payloads tests for Piggy-backed query and Malformed query. These payloads
are the following: ’"\’‘–’ and ’)’.

– Black Widow: Just as with Arachni, the fuzzing is done during the crawling phase.
It inserts scripts with a corresponding ID to identify XSS vulnerabilities and there
is a multitude of different XSS scripts injected. An important note described in
Section 3.1.2, is that Black Widow only fuzzes for XSS vulnerabilities. Another
element which will be described in Section 3.1.2 is the fact that the crawler is
using inter-state dependencies to make it possible to identify multi-step stored XSS
vulnerabilities.

– OWASP ZAP: As OWASP ZAP has favoured more of a modular design pattern,
the fuzzing is done after the crawling. All requests gathered during the crawling
will be saved, and it is these requests which are later fuzzed. When analysing
the payloads used for SQLi and comparing them to the different types of attacks
that have been described in Section 2.2, these are the following types which are de-
fined to be tested for: Tautology based queries, Malformed Queries, Union queries,

21
 2.6. Web applications for testing

Piggy-backed queries and Blind injection queries. When analysing the XSS injec-
tions there are just as with both Arachni and Black Widow a multitude of different
XSS payloads injected.

• Analyser

– Arachni: The analyser checks the DOM to identify possible stored/reflected XSS
vulnerabilities and error messages for possible SQLi. As a default setting, the anal-
yser scans for error messages on different DBMSs such as MySQL and SQLite.
– Black Widow: The analyser used in Black Widow only scans for two of the three
types of XSS vulnerabilities, these are stored and reflected. Since each request is
fuzzed with a script using a unique ID, it is possible to identify where the script
was injected.
– OWASP ZAP: Just like the analyser in Arachni, the DOM is checked to identify
possible stored/reflected XSS vulnerabilities and error messages are scanned for
possible SQLi. A difference though is that, OWASP ZAP by default does not take
into consideration different types of DBMSs such as MySQL and SQLite.

2.6 Web applications for testing

It is immoral and will probably get the tester into legal issues to try out WVSs, or do any
type of penetration testing on web applications without permission. This problem needs
to be overcome since the tools developed need to be tested and compared, the solution is
deliberately insecure web applications. These web applications are, just as the name suggests,
designed to be insecure and a safe place for professionals to test their tools and knowledge.

2.6.1 Examples of applications

There exist multiple different deliberately insecure applications, both traditional multi-page
applications and more modern single-page applications. A few of them will be presented
in this section. The presented deliberately vulnerable web applications are selected from
the OWASP Vulnerable Web Application Directory [45]. A further explanation of why these
deliberately vulnerable web applications are selected is presented in Section 4.2.2.

Broken Crystals
Broken Crystals [8] is a web application developed using Node.js and React, an open-source
JavaScript library used to build the front end. The application is a marketplace for rare
stones. The web application contains multiple types of known vulnerabilities, including XSS
and SQLi.

Damn Vulnerable Web Application

Damn Vulnerable Web Application (DVWA) [12] is a web application developed in Personal
Home Page (PHP) together with MySQL. It is what has earlier been described in this paper, a
multi-page application and has extensively been used as a tool to educate people on potential
security risks.

Hackazon
Hackazon [21] is an online shopfront built with an AJAX interface. It allows for users to con-
figure each part of the application with their own choice of vulnerabilities, which includes
SQLi, XSS and Cross-Site Request Forgery. This is done to minimize the risk of any form of
“cheating”, such as “known vulnerability testing” where the scanner recognizes the SUT and
is preconfigured to know where the vulnerabilities are located.

22
 2.6. Web applications for testing

OWASP Juice Shop

OWASP Juice Shop [43] is developed using Node.js, an open-source runtime environment
used by developers when building server-side applications. Express, a framework used
together with Node.js and Angular, is a typescript-based framework used to build user
interfaces in web applications. OWASP Juice Shop contains all types of vulnerabilities on the
OWASP top ten list [52].

OWASP Security Shepherd

Developed by OWASP, Security Shepherd is a web and mobile application used for security
training. Security Shepherd is written in Java using Jakarta Server Pages, which is a technol-
ogy allowing to writing dynamic (single-page application) data-driven pages. Just as with
OWASP Juice Shop, all types of vulnerabilities in the OWASP top ten [52] are present.

OWASP WebGoat
OWASP WebGoat [60] allows for users to test vulnerabilities which are frequently found
in Java-based applications. It is a single-page application consisting of multiple lessons
spanning over the OWASP top ten list [52].

WackoPicko
WackoPicko [58] was built by Doupé et al. [15] to be used in a research paper first published
in the 7th International Conference, "Detection of Intrusions and Malware, and Vulnerability
Assessment", 2010. The application has since been used as a benchmark for WVSs extensively
in multiple research papers. The web application is written in PHP together with MySQL
and has features such as photo sharing and photo-purchasing.

Categorisation of web applications

All these applications can be categorised by the web-based graphical user interface (GUI).
Where Broken Crystals, Hackazon and OWASP Juice Shop as earlier described are web ap-
plications representing online shops, which all have a similar interface. Figure 2.13 displays
parts of the landing page for Hackazon.

Figure 2.13: The web-based graphical user interface for parts of the landing page in Hacka-
zon.

23
 2.6. Web applications for testing

The second category when grouping the applications described is the security training ap-
plications. This includes DVWA, OWASP Security Shepherd and OWASP WebGoat. These
all have a similar GUI with a menu bar on the left side, this can be seen in Figure 2.14. The
outlier is Wackopicko, which is earlier described as a social media application.

Figure 2.14: The web-based graphical user interface of the landing page in Security Shepherd.

24
 3 Related work

In this chapter, related works are presented and followed by a discussion on how the research
papers will be used in this thesis.

3.1 Developing WVS

For developing a WVS, there are three modules to be considered. The following sections
present how previous research papers implement new techniques into two of those modules
in a WVS.

3.1.1 Focusing on the fuzzer module

Huang et al. [24] developed a WVS called VulScan. To develop this WVS, Huang et al. first
explored the different countermeasures for SQLi and XSS vulnerabilities and their draw-
backs. The countermeasures were categorised into three categories: secure implementation,
defence mechanism deployment and penetration testing.

Secure implementation includes input validation, input sanitation and parameterised queries
and is used to prevent SQLi. It is also possible to design a web application that considers the
least privilege principle. The drawback according to Huang et al. [24] of secure implemen-
tation is the difficulty for a developer to write secure code. The second category is defence
mechanism deployment, which can be tools such as a Web Application Firewall (WAF). One
drawback of a WAF is that it is often configured wrong, allowing possible vulnerable traffic
to bypass the firewall. The last category is penetration testing. This type of testing allows
developers to locate vulnerable sections in the web application. Automated penetration
testing is preferred by companies and allows for less expensive tests but generally has loose
code coverage.

By using this information, Huang et al. [24] propose VulScan which automatically creates
test data with the help of evasion techniques to bypass WAFs and input sanitation. VulScan,
compared to an ordinary WVS, contains four modules: web crawler, scanner knowledge
base, evasion techniques knowledge base and scanner engine (analyser). The difference to
other WVSs lies in the fuzzer where VulScan uses the evasion technique knowledge base to

25
 3.1. Developing WVS

generate test data. As concluded by Huang et al., this will cause more tests to be executed,
which will lead to longer and more exhaustive test sessions. To minimize inefficient test
cases, different techniques are applied, such as combining evasion techniques.

Huang et al. [24] evaluated the number of vulnerabilities found in the vulnerable web
application OWASP WebGoat and compared it to OWASP ZAP. The result is a higher num-
ber of vulnerabilities found using VulScan. The goal was simply to look at the fuzzer in a
WVS and see what could make it more efficient. This paper will be used in this thesis as a
basis for which changes are possible in the fuzzer of a WWS such as OWASP ZAP.

3.1.2 Focusing on the web crawler

Koswara and Asnar [34] developed a WVS called W3AF+, which is an extension of W3AF
that uses another web crawler for AJAX web applications. To develop W3AF+, Koswara and
Asnar first investigated the difference between a web crawler for ordinary web applications
and a web crawler for AJAX web applications.

According to Koswara and Asnar [34] the difference is that an AJAX application can send
multiple HTTP requests asynchronously to the server, which the ordinary web crawler can
not detect. Therefore, Koswara and Asnar suggest that an AJAX web crawler needs to use
two new sources to detect the asynchronous HTTP requests, the updated DOM and the AJAX
requests. Furthermore, since a new kind of request (AJAX request) is gathered from the web
crawler, the analyser can use the same algorithm to analyse the response because the error
message is not changed, only the format of the HTTP response.

The developed WVS by Koswara and Asnar’s [34] detects more vulnerabilities, in an AJAX
web application called Gregarius than W3AF. Just as with the paper by Huang et al. [24] the
authors Koswara and Asnar only focus on one module in a WVS to improve.

Another paper, which focus was to improve the crawler part of a WVS, is the paper by
Rennhard et al. [48]. In this paper, a tool called JARVIS was developed and aimed to improve
the code coverage when crawling a web application.

The first thought, according to Rennhard et al. [48], of how to improve a crawler is to
replace it with something better. However, Rennhard et al. stated that it could be harmful
to WVSs that are using a well-adapted crawler to do this. Consequently, they developed a
process which has two different approaches to help the crawler increase the coverage. The
first approach is what Rennhard et al. call endpoint extraction. This approach searches
the source code after URLs and parameters to be used as seeds. Since endpoint extraction
requires access to the source code, it is not always possible to use. Therefore, a second
approach is used. The purpose of this approach is to use a crawler with already known high
code coverage and use the found URLs as seeds.

When the seeds are collected, they need to be inserted into the crawler. There are four
different procedures to insert the seeds. The first two procedures are to place the seeds in
either the robot.txt, sitemap.xml or generate a new file if the two first files do not exist in the
web application. The third procedure, called landing-page based, is to create a new web
page, where all seeds are inserted into forms or links. This web page is then configured to be
the seed URL for the crawler, meaning it replaces the actual web application under test. Then,
when the crawler starts its process, it will find all seeds for the actual web application. One
limitation with this procedure is that some crawlers in the WVSs are interpreting the seed
URL (the landing page) as a directory. This means that the crawler retrieves the resources
below the landing page instead of requesting the landing page. Due to this limitation,

26
 3.1. Developing WVS

Rennhard et al. [48] created the last and fourth procedure, called index page-based. In this
procedure, the seeds are inserted into the original web application rather than a new creation
of a web page. The seeds are inserted directly into the first web page of the web application
to be found by the crawler.

The result of developing JARVIS was that it increased the detection of unique SQLi and
XSS vulnerabilities up to 60 % on average when using the seed approach. The performance
of detecting false positive vulnerabilities was on the same level as before when JARVIS was
not used.

A recent paper from 2021, made by Eriksson et al. [18], implemented a new strategy for
crawling a web application. Their WVS, called Black Widow, has a crawling process which
consists of three different parts. The first part is called Navigation Modeling. This part creates
a model of the web application to make it easy for the crawler to explore and detect vulnera-
bilities. The model tracks both server- and client-side of the web application by tracking the
server’s workflow, inter-state dependencies and capturing the client’s elements. The crawler
starts to extract all JavaScript events. The events are added to the model before they are fired.
When they are fired, the internal state can be changed and needs, therefore, to be stored in
the model. The model is a directed graph, where the node is the current state and the edges
are the action taken from an event. The actions that the model supports are GET requests,
form submissions, iframes, and JavaScript events. When an event is fired, the crawler rescans
the web page to see if new dynamic content is added to the DOM, since new injection points
could have been inserted.

The second part is called Traversal. This part traverses the Navigation Modeling by se-
lecting the unvisited edges in the order of when they were added to the model. An important
aspect of this part is that it takes care of authenticated sessions. Such sessions are noticed
when the crawler finds a form with a password field. When authentication is performed, the
crawler can dive deeper into the web application. When traversing the model, an approach
is needed to handle the inter-state dependencies. This approach considers an edge in the
model as safe before executing the next coming edge. A safe edge is when an action has
performed a GET request. If the edge is not safe, it will recursively continue to the previous
edge and check if that edge is safe. When a safe edge is found, the edges which connect to
the safe edge will be executed.

The last part is called Inter-state Dependencies. The main task for this part is to identify
and map if the state of the web application and the user inputs are connected. This is
achieved by the crawler when an input field (source) is identified, and a unique token is
submitted. Then, when the crawler visits other web pages (sinks), it looks after this token. A
token is a lowercase eight character long randomised string.

By testing Black Widow against other WVSs for both stored and reflected XSS, the per-
formance of Black Widow was seen to be superior. The crawling process in Black Widow
made a code coverage improvement by 63 % to 280 % compared to the other tested WVSs.
The number of unique detected vulnerabilities by Black Widow was 25, compared to the best
scanner, which detected 7 unique vulnerabilities. An addition is that Black Widow did not
report any false positives as the other WVSs did.

The three papers present three different approaches for improving the crawling process
in a WVS. This paper aims to take into consideration if any of these approaches are possible
to implement in OWASP ZAP and if it is, compare the result of the performance between
these WVSs.

27
 3.2. Performance evaluation of different WVSs

3.2 Performance evaluation of different WVSs

In the paper by Amankwah et al. [3], eight open-source and commercial WVSs (Acunetix,
AppAcan, Arachni, IronWASP, OWASP ZAP, Skipfish, Vega and WebInspect) are compared
on the effectiveness to find vulnerabilities. All WVSs were tested on two different vulnerable
web applications, Damn Vulnerable Web Application (DVWA) and WebGoat. Lastly, all
WVSs were evaluated with different performance metrics.

The results show that the detection rates between the different WVSs were diverse.
Amankwah et al. [3] described this variation as a result of each WVS unique way to de-
tect vulnerabilities. The WVSs used in the experiment found all real vulnerabilities (true
positive). However, the WVSs also found some vulnerabilities which after a manual check
were defined as not a real vulnerability (false positive). Amankwah et al. suggested that if
the WVSs ability to detect vulnerabilities improves, the false positive rate will decrease.

Another study, done on the performance of WVSs by Foster [19] from 2018, focuses on
the behaviour of WVSs when analysing modern web applications using for example AJAX,
Google Web Toolkit and JSON. The study tested three WVSs: OWASP ZAP, BurpSuite Pro
and Acunetix Vulnerability Scanner. The chosen vulnerable web application for the study
was Hackazon, which when used can be configured with customisable vulnerabilities, to
minimise the risk for scanners to have prior knowledge about specific vulnerabilities in the
application. As previously mentioned, this vulnerable application is using more modern
technologies and frameworks, unlike more traditional options such as DVWA, which was
used by Amankwah et al. in their study. Foster brings up that a study comparing the per-
formance of WVSs on AJAX and modern web applications are important, since they were
built around traditional HTML and not newer formats, instead, they have had to adapt to
the newer technologies.

The study was conducted by running the three WVSs with minimal configurations on
Hackazon with customised vulnerabilities. The only configurations set were user credentials
to enable access to all parts of the web application, and AJAX options were selected. The
number and the nature of found vulnerabilities were recorded and categorised into three
categories: high, medium, and low. This is based on the severity of the detected vulner-
ability. The result was filtered on the number of unique vulnerabilities, which concluded
that a high number of duplicated vulnerabilities was detected. It was found that 25% of the
preconfigured vulnerabilities were detected. Most of the undetected vulnerabilities were OS
Command Injection, Remote File Inclusion, Stored XSS and Integer Overflow. Foster [19]
points to multiple reasons why vulnerabilities were missed during the scanning process,
such as the crawling process missing URLs, leading to parts of the application never being
searched. The study, therefore, concludes that scanners should be used together in conjunc-
tion and with manual testing.

The last paper, by Matti [37], is an evaluation and analysis of three different open-source
WVSs. The thesis was based on analysing the detection of SQL injection and XSS vulnerabili-
ties, using 22 already known insecure web applications in OWASP Broken Web Applications
(OWASP BWA). The result is a discussion on the different approaches by the WVSs to find the
specified vulnerabilities. When looking at the SQL injection, Matti concludes that there are
both similarities and differences between the scanners. One considerable difference, which
had a big impact on the result, was the payload used by the scanners.

These three presented papers are used by this thesis to create a methodology for how
performance testing and analysis on WVSs are assembled. It is also used as a decision basis
for selecting WVS and vulnerable web applications used in performance testing and analysis.

28
 4 Methodology

In this chapter, the chosen methodology is presented together with a motivation for the dif-
ferent choices taken during the thesis time lapse. Each part of the method has been selected
with the research questions in mind. In Table 4.1 it is displayed which research question is
corresponding to what part of the method.

Table 4.1: Research question mapped to the corresponding part of the method.
Part in method Research question
Literature study RQ1
Initial test session RQ3
Code analysis RQ1, RQ2, RQ4
Implementation RQ4
Control test session RQ3, RQ4

4.1 Literature study

To gather knowledge and a better understanding of the different aspects of WVSs, SQLi and
XSS, the first phase of this thesis is to do a literature study. The literature study is conducted
by using Google Scholar and LiU library as sources when searching for peer-reviewed pa-
pers. By using different combinations of keywords for SQLi, XSS and the three components
in a WVS, a list of research papers is gathered. The list is then filtered based on the credibility
of the journal/publisher in which the research paper is presented. One way to control if the
papers are not credible is to investigate if the selected papers are published in a journal/pub-
lisher listed as predatory by Beall’s list [5]. One reason a predatory journal is considered to be
predatory is that the published papers have little or no peer-review. Another way to control
the credibility is to check the number of citations and the H-index of the author, which is
presented by Google Scholar. A further step in collecting papers is to control the publication
date and to read the abstract to determine if the paper is relevant for the thesis or not. These
steps should provide the final list of papers used in this thesis. The literature study is done to
answer the first part of research question 1.

29
 4.2. Selections

4.2 Selections
To be able to identify the flaws OWASP ZAP potentially contains, it has to be compared to
other WVSs. The collections of WVSs are then tested with as much of a default setting as
possible on different deliberately vulnerable web applications to be able to find scenarios
where the detection of vulnerabilities differs. For making this possible, the selected WVSs
and deliberately web applications are presented below with a motivation.

4.2.1 Selected WVSs

The selected WVSs are presented in Table 4.2 and all of them are open-source projects de-
scribed in Section 2.5.7. This is an important criterion in the choice of WVSs since it enables
the possibility to analyse the code base. The selected WVSs are divided into two categories.
The first category is the WVSs being able to crawl AJAX web applications, and the second
category is WVSs without an AJAX crawler. Black Widow [7], which belongs to the first
category, is a recently created WVS, where the focus was to get a high code coverage of the
scanned application. By including this in the performance analysis for this thesis, it would
increase the possibility to find differences in the crawler. W3AF [57] and Arachni [4] are
two other WVSs that belong to the first category. Both W3AF [18, 34, 48] and Arachni [3,
18] are used in earlier performance analysis and performed better than OWASP ZAP [46] in
finding SQLis. The last three, Wapiti [59], Skipfish [51] and Vega [54] belong to the second
category. All of them can be considered to be traditional WVSs and are selected since they
are used in older papers doing performance analysis [3, 18, 37, 48]. The remaining open-
source WVSs presented in Section 2.5.7, Enemy-of-the-state, IronWASP and VulScan are not
selected. Enemy-of-the-state could not crawl AJAX applications and seems to perform worse
than Black Widow in earlier research papers. The crawler of Black Widow is also a state aware
crawler with additional features and has the ability to crawl AJAX applications. Due to these
reasons, Enemy-of-the-state is not selected. IronWASP is a highly customisable WVS and can
be constructed after a developer’s desire. In this thesis, each WVS is run with as much of
default settings as possible. According to Amankwah et al. [3] the default settings for Iron-
WASP will not perform well and therefore IronWASP is not selected for this thesis. VulScan
is not selected due to the source code not being available.

Table 4.2: Selected WVSs for the purpose of performance analysis.

WVS Version Active Supporting AJAX
Arachni 2.0 dev Yes Yes
Black Widow 1.0 No Yes
OWASP ZAP 2.11.1 Yes Yes
Skipfish 2.10b No No
Vega 1.0b No No
Wapiti 3.0.9 Yes No
W3AF 1.6.49 No Yes

4.2.2 Selected deliberately vulnerable web applications

The selected deliberately vulnerable web applications are presented in Table 4.3 and are de-
scribed in 2.6.1. There are four AJAX web applications selected, to be able to answer research
question 2, and they are: Broken Crystals [8], Hackazon [21], OWASP Juice Shop [43] and
OWASP Security Shepard [44]. The remaining two web applications, DVWA [12] and Wack-
oPicko [58] are traditional multi-page web applications. These deliberately vulnerable web
applications have been used in earlier research such as [3, 18, 19, 24, 37, 48]. The one appli-
cation left from Section 2.6.1 is OWASP WebGoat. This vulnerable web application is not se-

30
 4.3. Initial test session

lected due to its similarities with OWASP Security Shepherd and DVWA, since all of them are
security training applications. Both WebGoat and Security Shepherd are single-page applica-
tions and DVWA is a multi-page application. By removing WebGoat there is one single- and
one multi-page application left in the security training web application category. WebGoat,
compared to Security Shepherd, is made with more intention to teach about the understand-
ing of different vulnerabilities and is not made to be scanned for security issues.

Table 4.3: Selected deliberately vulnerable web applications used for testing.
Application Version AJAX application # of vulnerabilities
XSS SQLi
Broken Crystals 1.0 Yes 3 1
DVWA 2.0.1 No 3 2
Hackazon 1.0 Yes 8 23
OWASP Juice Shop 13.2.1 Yes 9 4
Security Shepard 3.1 Yes 7 10
WackoPicko 1.0 No 5 2

4.3 Initial test session

When the literature study is completed, a testing environment is created. The testing environ-
ment is a virtual machine and is created with help of Oracle VM VirtualBox [42]. The selected
operating system is Kali Linux 2021.4a [28]. Kali Linux is a Debian-based Linux distribution
specifically made for digital forensics and penetration testing. It also has both Skipfish and
Wapiti pre-installed. The virtual machine is installed on a laptop with the following specifica-
tion: AMD Ryzen 5 Mobile 2500U, 8 GB of RAM and Windows 10 Home 64-bit. The selected
WVSs from Table 4.2 and the selected deliberately web applications from Table 4.3 are then
installed on the virtual machine. The configurations for the WVSs have been specified in
Appendix A. If possible, the tests are done as an authenticated scan. This means, the scan-
ners would automatically log in and be able to access restricted parts of the web applications,
which would make it possible to get higher code coverage. This part of the methodology is a
necessary step for Section 4.4.1 to be able to answer research questions 2 and 3.

4.4 Analysis
The analysis in this thesis consists of three major parts: performance analysis, code analysis
and control test session. After the initial test session, all the found vulnerabilities are analysed
in the performance analysis Section 4.4.1. For all the vulnerabilities not found by OWASP
ZAP, a code analysis is done in Section 4.4.2 to determine if either the code needs to be mod-
ified or the settings in OWASP ZAP. This is followed by a control test session, described in
Section 4.4.4. This flow can be seen in Figure 4.1.

31
 4.4. Analysis

Figure 4.1: Flowchart of the methodology.

4.4.1 Performance analysis

The performance analysis is conducted from the test result. The inspected performance pa-
rameters are the number of found vulnerabilities, the number of false positives and the num-
ber of uniquely found vulnerabilities. To analyse the three parameters, each scanner produces
a report for each test on the vulnerable web applications. From the reports, all potential vul-
nerabilities are collected and compared to the known actual vulnerabilities that exist in the
vulnerable web application. As a result of this workflow, each vulnerability found by the
WVS could be determined to be either true positive or false positive. When all true positive
and false positive vulnerabilities are determined, the unique vulnerabilities could be identi-
fied for each application. This is performed by comparing where the different vulnerabilities
are found in the application. This and the previous part of the methodology are done to
answer research question 3.

4.4.2 Code analysis

From the performance analysis in Section 4.4.1, the WVSs with the largest difference in found
vulnerabilities compared to OWASP ZAP are selected to do a code analysis. For each WVS,
the source code is retrieved to perform the code analysis. The procedure of performing the
code analysis is first to identify which part of the WVS is the reason for the missed vulnera-
bilities. This step aims to reduce the amount of code to be searched. The next step is to read
and understand the different techniques in the code of the identified part of the WVS. The last
step is to compare it to OWASP ZAP to identify the similarities and the differences between
the techniques, which could be helpful with the determination if any changes are possible to
be performed in OWASP ZAP. During the process of reading the code of each WVS, this the-
sis aims to follow one of the three code reading techniques called systematic techniques and
presented in the paper by Dunsmore et al. [16]. The systematic technique approach to code
reading is to first analyse the interdependencies in each WVS, where the classes with the least
dependencies are read first. The same follows with the class methods. During the inspection
of the class methods, all used functions inside a method are traced and read to be understood.
This often involves reading functions from other classes. By using this technique, this thesis
should understand the different code bases. This part of the methodology is done to answer
the second part of research question 1, research question 2, and research question 4.

32
 4.4. Analysis

4.4.3 Implementation
Based on the result of section 4.4.2, the identified differences are further analysed and eval-
uated if possible to implement in OWASP ZAP considering multiple factors. These factors
are time frame, effect on result and possibility of the implementation being production-ready.
This is done by making a draft of the changes needed to implement the technique and listing
which vulnerabilities would be possible to find using the specified technique. During the im-
plementation process, the code guideline and syntax used in OWASP ZAP will be followed.
The aim of this part of the methodology is a necessary step to be able to answer the second
part of research question 4.

4.4.4 Control test session

This part of the methodology follows the same approach as in Section 4.3. The only difference
is that the only tests to be run are OWASP ZAP with the new configuration or implemented
part on all the vulnerable web applications. This part of the methodology is a necessary
step to verify that the changes made from Section 4.4.2 are increasing the number of found
vulnerabilities by OWASP ZAP.

33
 5 Results

In this section, the data gathered is presented by following the methodology in Section 4.

5.1 Initial test session

The results from the initial test session are presented below. The initial test session has been
performed using the selected WVSs in Table 4.2. It should be noted that the tests on OWASP
ZAP have been performed using both the traditional spider and the AJAX spider (in the ta-
bles named ZAP Spider and ZAP AJAX). This is done because it is suggested by the founders
of OWASP ZAP to run both the traditional spider and AJAX spider to get the optimal result.
Table 5.1 lists all the vulnerabilities found by each scanner, divided into XSS and SQLi. The
number above XSS and SQLi is the number of the predetermined vulnerabilities that exist
in the web application. As earlier mentioned in Section 2.5.7, Black Widow is only capable
of finding XSS vulnerabilities and because of that are the cells for SQLi filled with dashes.
The setup was done before each run is described in Appendix A. Based on the selected ap-
plications in Table 4.3 and an analysis of the reports produced during scanning, it can be
concluded that only Black Widow found all predetermined vulnerabilities during one of the
scans. This was seen in the scan of DVWA. During the scan of DVWA, Arachni and Black
Widow did discover more XSS vulnerabilities than what was predetermined. These vulnera-
bilities were manually examined on the application and determined to be real vulnerabilities.
Arachni however did miss one of the predetermined XSS vulnerabilities. An important note
to make is the total number of vulnerabilities found by Wapiti and Skipfish is three.

34
 5.1. Initial test session

Table 5.1: The total number of found vulnerabilities for each WVS on each deliberately vul-
nerable web application.
WVS/ Broken DVWA Hackazon Juice Security WackoPicko Total
Application Crystals Shop Shepherd
3 1 3 2 8 23 9 4 7 10 5 2
XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi
Arachni 0 0 8 2 8 0 0 1 0 0 7 1 27
Black Widow 1 - 7 - 2 - 0 - 1 - 3 - 14
Skipfish 0 0 0 0 1 0 0 0 0 0 1 0 2
Vega 0 0 0 0 1 6 0 0 0 0 1 2 10
Wapiti 0 0 0 0 0 0 0 0 0 0 1 0 1
W3AF 0 0 3 3 1 0 0 0 0 0 2 0 9
ZAP AJAX 2 0 3 1 1 1 0 1 0 0 3 1 13
ZAP Spider 1 0 2 2 2 1 0 1 0 0 3 1 13

Table 5.2, lists the number of false positive vulnerabilities, compared with the total number of
reported vulnerabilities listed in Table 5.1. What can be noted from this table is the number of
false positives produced by Arachni during the scan of Hackazon and WackoPicko. Another
result which should be noted is the number of false positive SQLi vulnerabilities by Vega in
Hackazon. Otherwise, it seems that most scanners are quite conservative when determining
if it is a real vulnerability or not. Black Widow did have the best result, with only one false
positive, when comparing the best performing WVSs. The false positive vulnerability found
by Black Widow was a reflected XSS.

Table 5.2: The total number of found false positive vulnerabilities compared with the number
of reported vulnerabilities for each WVS on each deliberately vulnerable web application.
WVS/ Broken DVWA Hackazon Juice Security WackoPicko Total
Application Crystals Shop Shepherd
3 1 3 2 8 23 9 4 7 10 5 2
XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi
Arachni 0/0 0/0 0/8 0/2 6/8 0/0 0/0 0/1 0/0 0/0 3/7 0/1 9/27
Black Widow 0/1 - 0/7 - 1/8 - 0/0 - 0/1 - 0/3 - 1/14
Skipfish 0/0 0/0 0/0 0/0 0/1 0/0 0/0 0/0 0/0 0/0 0/1 0/0 0/2
Vega 0/0 0/0 0/0 0/0 0/1 4/6 0/0 0/0 0/0 0/0 0/1 2/2 6/10
Wapiti 0/0 0/0 0/0 0/0 0/0 0/0 0/0 0/0 0/0 0/0 0/1 0/0 0/1
W3AF 0/0 0/0 1/3 1/3 0/1 0/0 0/0 0/0 0/0 0/0 0/2 0/0 2/9
ZAP AJAX 0/2 0/0 1/3 1/1 1/1 0/1 0/0 0/1 0/0 0/0 2/3 0/1 5/13
ZAP Spider 0/1 0/0 1/2 1/2 1/2 1/1 0/0 0/1 0/0 0/0 2/3 0/1 6/13

The reports produced during scanning made it possible to list the unique number of vulner-
abilities found per web application and WVS. This is summarised in Table 5.3. By reviewing
this table, it can be seen that the WVS which found the most amount of unique vulnerabil-
ities is Black Widow, followed by Arachni. Another interesting thing to note is the result
of Vega and W3AF, which did respectively find a unique SQLi in Hackazon and an XSS in
WackoPicko. The last of these vulnerabilities can be seen as out of scope because it can only
be executed with Flash, which is deprecated [1] and not supported by any of the other WVSs.

Table 5.4 displays the number of vulnerabilities each WVS found that OWASP ZAP did
not. This includes the scanning made with both the AJAX spider and traditional spider in
OWASP ZAP. The WVSs which detected the most amount of vulnerabilities that OWASP
ZAP did not are Arachni and Black Widow. Another interesting thing to note is that the only
two vulnerabilities found by Skipfish were not detected by OWASP ZAP. From this and the
earlier tables described, a code analysis was done on the two most interesting WVSs, Arachni
and Black Widow.

35
 5.1. Initial test session

Table 5.3: The total number of found unique vulnerabilities for each WVS on each deliberately
vulnerable web application.
WVS/ Broken DVWA Hackazon Juice Security WackoPicko Total
Application Crystals Shop Shepherd
3 1 3 2 8 23 9 4 7 10 5 2
XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi
Arachni 0 0 0 0 0 0 0 1 0 0 1 0 2
Black Widow 1 - 2 - 0 - 0 - 1 - 1 - 5
Skipfish 0 0 0 0 0 0 0 0 0 0 0 0 0
Vega 0 0 0 0 0 1 0 0 0 0 0 0 1
Wapiti 0 0 0 0 0 0 0 0 0 0 0 0 0
W3AF 0 0 0 0 0 0 0 0 0 0 1 0 1
ZAP AJAX 1 0 0 0 0 0 0 0 0 0 0 0 1
ZAP Spider 0 0 0 0 0 0 0 0 0 0 0 0 0

Table 5.4: The number of found real vulnerabilities for each WVS which was not found by
OWASP ZAP on all deliberately vulnerable web application.
WVS/ Broken DVWA Hackazon Juice Security WackoPicko Total
Application Crystals Shop Shepherd
3 1 3 2 8 23 9 4 7 10 5 2
XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi
Arachni 0 0 4 1 1 0 0 1 0 0 3 0 10
Black Widow 1 - 5 - 1 - 0 - 1 - 3 - 11
Skipfish 0 0 0 0 1 0 0 0 0 0 1 0 2
Vega 0 0 0 0 1 1 0 0 0 0 1 0 3
Wapiti 0 0 0 0 0 0 0 0 0 0 1 0 1
W3AF 0 0 0 1 1 0 0 0 0 0 2 0 4

5.1.1 Settings
After the initial test session, options were found in OWASP ZAP for how many payloads are
used and how sensitive the analyser should be to determine if an attack was successful or
not. These two options could be changed in the policy for the attack session in OWASP ZAP,
where the number of payloads is determined by the strength parameter and the sensitivity
is determined by the threshold parameter. What is changed by the sensitivity setting differs
depending on which vulnerability is being searched for. In the case of SQLi, changing the
setting to low enables the check for generic error messages from a specific database. A specific
database error message could be any error message that a database can return. This includes
messages which might not be error messages prone to a SQLi vulnerability, such as:

ER_CONNECTION_KILLED 70100 ”Connection was killed”

For reflected, and stored XSS, it changes the sensitivity level on where the inserted payload
should be reflected. Using a high level means that the payload needs to be reflected at the
same place in the DOM as was detected during the initial check.

During the code analysis, several new scan rules were found for detecting SQLi. This is
referred to as beta add-on in Table 5.5. The beta add-on was by default not activated in
OWASP ZAP, which made OWASP ZAP miss detecting some SQLi vulnerabilities. The
beta add-on contains different types of time-based SQLi payloads, which code syntax is
based on the DBMS the web application is using. The add-on supports code syntax for
the DBMSs: MySQL, MsSQL, PostgreSQL, SQLite, Hypersonic and Oracle. An example is
MySQL time-based SQLi, where the payload consists of the function sleep. For PostgreSQL,
the corresponding function code syntax looks like pq_sleep and for SQLite, the code syntax
is randomblob. By enabling this add-on in the control test session three more vulnerabilities
could be found, where two of them were found in DVWA and one in Juice Shop. DVWA is

36
 5.1. Initial test session

using MySQL as the DBMS, while Juice Shop is using SQLite as the DBMS.

It is also a possibility to find the SQLi in Juice Shop by changing the strength and threshold
parameter to insane and low. The SQLi was found when this payload: ’( was injected. This
produced a SQLite error message that OWASP ZAP detected with the standard scan rule for
SQLi. With the time-based SQLi, described in the paragraph before, this is the payload that
was injected:
1
| case randomblob(1000000) when not null then ”” else ”” end | 1

Since it is a time-based SQLi attack, OWASP ZAP detects a delay to get back the response
from the database compared to when an empty string is injected. Based on this knowledge,
OWASP ZAP decided it is a possibility that there exists a SQLi.

Table 5.5: Changes in settings that made OWASP ZAP find vulnerabilities it could not find in
the initial test session.
Application Vulnerable URL and Found by WVS Vulnerability Change of setting
element in application type
Broken Crystals Input element subject Black Widow DOM-based Policy changes
at landing page XSS Strength = high
Threshold = low
DVWA Input element Arachni, W3AF SQLi Add beta add-
id at /sqli on for policy
DVWA Input element Arachni, W3AF SQLi Add beta add-
username at /brute on for policy
DVWA Input element Arachni, Reflected XSS Policy changes
username at /brute Black Widow Strength = high
Threshold = low
DVWA Input element Black Widow Reflected XSS Policy changes
uploaded at /upload Strength = high
Threshold = low
DVWA Input element Black Widow DOM-based Add beta add-on
txtName at /xss_d XSS and for policy set
Strength = high
Threshold = low
Hackazon Input element id Vega SQLi Add beta add-on
at /product/view and for policy set
Strength = high
Threshold = low
Juice Shop Input element q Arachni SQLi Add beta add-
at /product/search on for policy
WackoPicko Input element Arachni Reflected XSS Policy changes
username at /login Strength = high
Threshold = low
WackoPicko Input element query Arachni, Reflected XSS Option –> AJAX
at /pictures/search Black Widow, Spider –> Disable
Skipfish, Vega, "click default
Wapiti, W3AF elements only"

The last setting that was changed based on the result of the code analysis is what type of
input elements OWASP ZAP interacts with by default. When using the default setting,
OWASP ZAP only interacts with input elements that are of type "button" and "submit". This
approach for what types of elements in the HTML-file that are considered to be potentially
clickable elements are a part that differs between OWASP ZAP and the other two scanners
(Black Widow and Arachni) used in the code analysis. OWASP ZAP does state that it con-
siders the "Input" type elements as potentially clickable. This was as earlier shown to not
be entirely true. It is possible to change this to make OWASP ZAP interact with all types of

37
 5.2. Techniques identified during code analysis

input elements, which made it possible to find a vulnerability in WackoPicko that OWASP
ZAP missed from the initial test session.

To summarise, Table 5.5 shows the result of the setting changes presented. By doing these
changes, ten vulnerabilities were found by OWASP ZAP that was not found in the initial test
session. Four of these ten changes consisted of changing the policy setting in OWASP ZAP,
where the strength and threshold parameters were changed from default to high and low.
Three of these ten changes of settings needed the beta add-on to be enabled for finding the
missing vulnerabilities. By combining the two changes of settings just presented, two more
vulnerabilities were found. The last setting which was changed was the "click only at default
elements" setting. This needed to be disabled to click on any type of input element to find
the vulnerability.

5.2 Techniques identified during code analysis

A few concepts and problems were identified by doing the code analysis based on the result
in Table 5.4. The WVSs selected from this table and compared with OWASP ZAP are Arachni
and Black Widow since they found more true positive vulnerabilities during the initial test
session. The major differences in techniques between the selected scanners are the type of
crawler, the technique for finding stored XSS and the technique for getting access to restricted
areas by using an authentication procedure.

Crawler
The code analysis concludes that OWASP ZAP is using an open-source web crawler for
crawling AJAX applications, named Crawljax. This is done by implementing an API for
communicating between OWASP ZAP and the crawler. Crawljax is a DFS crawler, this is
in contrast to both Arachni and Black Widow which are using a BFS crawler. The advan-
tages and disadvantages of DFS and BFS were earlier described in Section 2.5.1 and can be
summarised by concluding that BFS crawlers are better suited to use when the landing page
has a high number of branches. However, DFS crawlers are better to use when crawling an
application with a high depth level.

Based on the advantages and disadvantages, it was concluded that the XSS vulnerability
which was not found in Hackazon by OWASP ZAP but found by both Arachni and Black
Widow is due to the DFS crawler implemented in Crawljax. As earlier described in Section
2.6.1, Hackazon is an online shopfront built with an AJAX interface. This means there are a
lot of possible branches and paths on the landing page, which as described earlier means a
bigger risk for a DFS algorithm to enter an infinite loop. As described in Section 2.5.8, Crawl-
jax prioritises Anchor elements with a hyperlink attribute. The problem that occurred in this
specific case was that when running a scan with OWASP ZAP on Hackazon, the request and
the parameter vulnerable for the XSS were never found before entering an infinite loop. This
is due to the vulnerable action being an "Input" element which would have been scanned last
by Crawljax.

Stored XSS Vulnerabilities

It was earlier described in section 3.1.2 that Black Widow is using inter-state dependencies. By
doing so, it allows for finding action sequences which enable certain vulnerabilities. Another
way Black Widow differs from OWASP ZAP is it allows for analysing stored XSS vulnerabili-
ties while crawling. This is done by inserting scripts with a unique ID and searching for those
IDs while crawling, which when combined with the inter-state dependencies makes it possible
to find multi-step stored XSS vulnerabilities which no other WVSs can find. This was shown
when scanning WackoPicko, where Black Widow was the only scanner to find the multi-step

38
 5.2. Techniques identified during code analysis

XSS vulnerability. OWASP ZAP is doing the crawling first and gathers all requests, but there
is no regard to in what order the requests are made except in a depth-first manner. Later, the
user of OWASP ZAP can decide what type of vulnerabilities to search for. If the user decides
to search for stored XSS, OWASP ZAP will inject a safe value in all POST requests that
exist in the tree of the application after the crawling phase. This is followed by making all
the found GET requests to search for the injected safe value and identify which GET request
is the sink to the POST request that injected the value. The POST request is seen as the source.

A problem with the technique used by OWASP ZAP could be identified while scanning
Hackazon. During the scan for stored XSS vulnerabilities, OWASP ZAP injected a safe value
using a POST request but identified the wrong sink node. This is due to the safe value
occurring twice in the application, but only one of these occurrences was vulnerable to stored
XSS. To access the vulnerable request, a button, which was created when doing the POST
request, was needed to be pressed. This button was connected to a GET request with the ID
parameter from the response in the POST request. Due to OWASP ZAP not crawling while
searching for stored XSS, the specific button and GET request connected to this vulnerability
were never found.

Arachni just like Black Widow injects scripts with a specified ID during the crawling/at-
tack phase. A problem which was identified during the code analysis and further described
in Section 6.1.1 is the fact that Arachni is using the same ID during the whole test session.
A further issue with this can be identified in the report created by Arachni. By using this
technique, it does not allow Arachni to categorise the detected vulnerabilities in stored XSS
and reflected XSS. This is due to Arachni not being able to determine if the found script was
injected in the most recent request and reflected in the response.

Authentication techniques
The identification and implementation of the authentication needed during a scan, to give
entry to what are otherwise hidden parts of a web application, differs between the three
WVSs selected for the code analysis. To be able to access these otherwise hidden parts gives
the WVSs a greater chance of finding vulnerabilities. In most cases, both for the tested appli-
cations in this thesis and real-world web applications, there is a need for a user to be logged
in to post comments or do similar tasks. OWASP ZAP gives the user multiple options on how
to do an authenticated scan. The used option in this thesis was to manually explore the login
steps, this is described in Appendix A. The user then has to configure the request manually
to log in during the automated scan, this includes changing the type of request such as either
FORM-based or JSON-based request. The other option for an authenticated scan is to use
a Zest script, this could be used for example when a randomised authentication token is
needed during the request. This authentication token is normally implemented to minimise
the risk of a replay attack, where the attacker intercepts the request and re-transmits it at a
later time.

Compare this to the approach of Black Widow, where the user changes the username
and password in the code (no user interface, since the WVS is only a proof-of-concept). The
login form is then automatically found in the scan during the early states of the crawling
process, when all the GET requests are made, using a function called find_login_form. This
function extracts all forms in the current state and looks for keywords such as "password".
If a login form is found, the manually supplied data is injected and the form is submitted.
This approach was concluded to not be as effective as manually supplying the request, due
to Black Widow having a problem successfully authenticating in certain applications such as
Hackazon. Arachni did also have some issues doing the authentication during this thesis,
as stated in Appendix A, a decision was made to use the autologin plugin. Another plugin
which exists in Arachni is the login_script plugin, this was deemed to be out of scope for this

39
 5.3. Implementation

thesis and not used. The autologin plugin, works by the user supplying the URL where the
login form is located, the parameters, and a string. The string is used to check if the login
was successful by comparing the response body in the request to see if the string is present.
The code in Arachni is implemented similar to Black Widow, where a GET request with the
user-supplied URL is made to obtain the DOM for which the login form is located. This
DOM is then parsed, and the login form is extracted. The user-supplied data is then given,
and the request is sent. As earlier stated, both these techniques had problems during the
authentication, and it was therefore concluded that changing the technique for authentication
in OWASP ZAP would not have a positive effect on the result.

5.3 Implementation
The result from Section 5.1 shows that Arachni and Black Widow performed better than
OWASP ZAP. Based on this and the differences found during the code analysis, and also
considering the time limit and effect on the result, a BFS crawler was implemented. The
result of the implemented BFS crawler in Crawljax can be seen in Algorithms 1 and 2. In
Algorithm 1, an action on row number one is derived from a cache with all extracted actions
that exist in the current state. A state in Crawljax is a unique DOM on the web application
under crawling. Each state is then connected with an event and when the event is fired the
crawling process goes from the current state to a possibly new state. The connected states are
the structure of a state graph. Continuously, at row three, the associate element of the action
is derived. From the element and the action, an event can be created. The event is checked
if it is possible to be fired. If it succeeded to be fired, the crawler inspects the newly entered
state.

Algorithm 1 Crawl through actions algorithm.

1: action Ð actionCache.poll ()
2: while action != null do
3: element Ð action.getElement()
4: if element.satis f yAllBrowserConditions() then
5: event Ð getEvent(element, action)
6: if f ireEvent(event) then
7: inspectNewState(event)
8: end if
9: end if
10: action Ð actionCache.poll ()
11: while action != null and unvisitedStates.size() ą 0 do
12: reset()
13: stateToInspectList Ð unvisitedStates.getFirstAndRemove()
14: stateToInspect Ð stateToInspectList.get(0)
15: pathtoState Ð stateToInspectList.get(1)
16: f ollow( pathToState, stateToInspect)
17: parseCurrentPageForCandidateElements()
18: action Ð actionCache.poll ()
19: end while
20: end while

The inspection of the newly entered state happens in Algorithm 2. First is a check to see if the
crawler found a state that is not belonging to the web application under crawl. If the crawler
is out of the scope, it is going back to the previous state to try to find a new state. Otherwise,
the newly entered state’s DOM is checked for changes. If the DOM is not changed, the crawler
is just like in the out-of-scope case, going back to the previous state to try to find a new state
instead. Else, the newly entered state is checked if it is a clone. i.e., the crawler has already

40
 5.3. Implementation

visited the newly entered state, and it is already present in the state graph. If the newly
entered state is not a clone, the crawler confirms that it has not reached the maximum depth.
If the crawling process is still within the crawl depth limit, the path from the initial state to
the previous page is updated with the event that connects the path to traverse to the new
state. Then an array list is created that contains the new state and the path from the initial
state to the newly entered state. The array list is added as a value in an ordered map, where
the key is the identifier of the new state. Lastly, the crawler is going back to the previous state
to try to find a new state.

Algorithm 2 Inspect new state algorithm.

Require: event
1: if crawlerNotInScope () then
2: goBackOneState()
3: else
4: newState Ð stateGraph.getNewState()
5: if domChanged(event, newState) then
6: isNewState Ð stateGraph.switchToStateAndCheckI f Clone()
7: if isNewState && (maxDepth ą ++crawlDepth) then
8: crawl path.add(event)
9: stateList.add(newState)
10: stateList.add(crawlPath)
11: unvisitedStates.put(newState.getId(), stateList)
12: end if
13: goBackOneState()
14: end if
15: end if

Going back to row seven in Algorithm 1, the crawler is back to the previous state and at row
ten is the next action polled for that state. If the crawler already has gone through all actions
from the cache and the unvisited states, the crawling process is finished. Otherwise, the
crawler enters the while-loop at row eleven. Then a reset function is triggered to initialize
the path, the crawl depth and the state graph to its original values. The crawler continues
to get the first key in the ordered map unvisitedStates to extract the state to inspect and the
state’s path. The function follow is then used to adjust both the state graph and the browser.
In the state graph is the inspected state set as the current state and in the browser is the cor-
responding page of the state set as the current page. This makes it is possible for the function
parseCurrentPageForCandidateElements to do what the name suggests. When the current page
is parsed, the cache has been filled with action of the current state and the crawler can poll
an action from the cache.

To summarise the workflow of the two algorithms, firstly all actions are gathered from
the current state. One of the actions is used together with its corresponding element to create
an event and fire it. The result is inspected to identify if a new state has been entered. To
achieve the inspection, the DOM of the newly entered state is retrieved and compared with
the current state. If the two DOMs are equal according to the Levenshtein distance method
the newly entered state is a clone, otherwise a new state is identified. The new state is saved
in the map together with the path from the initial state to the new state. The next step is to
use the upcoming action gathered from the current state and do the same workflow. If no
actions are left in the current state, a state is taken from the map and that state becomes the
current state.

The implementation is using mostly already implemented functions and variables, but
they are used to achieve a BFS instead of a DFS. The BFS algorithm is implemented to crawl

41
 5.4. Control test session

the leftmost branch first. In this case, the crawler is searching for elements to extract from top
to bottom in the DOM of the current page. The ordered map chosen to be used in these two
algorithms is convenient with a BFS algorithm. The key of the ordered map is the identifier
of a state. The identifier is a number that is increased every time a new state is created,
hence the map is in the order as newer found states are added last to the map. When the
value of the first key in the map is obtained and removed, this value consists of a new state
that is examined. When the examination is finished, it continues to take a new state from
the ordered map. As a result, the ordered map is taking one level of states at a time, which
creates the BFS behaviour.

5.4 Control test session

The BFS crawler, implemented in Section 5.3 is tested on the same deliberately vulnerable web
applications as in Section 5.1. The result of these tests can be seen in Table 5.6. By analysing
Table 5.6 both OWASP ZAP AJAX BFS and OWASP ZAP AJAX DFS perform equally on
the applications Broken Crystals, Juice Shop and WackoPicko. DVWA and Security Shep-
herd could also be considered as performing equally since in DVWA did OWASP ZAP AJAX
DFS find a false positive SQLi that OWASP ZAP AJAX BFS did not find. The same could
be concluded in Security Shepherd but in this case, did OWASP ZAP AJAX BFS find a false
positive vulnerability that OWASP ZAP AJAX DFS did not find. One difference in perfor-
mance can be seen in Hackazon where OWASP ZAP AJAX BFS found an XSS vulnerability
that OWASP ZAP AJAX DFS could not find. This confirms the improvement of implement-
ing a BFS crawler to find missing vulnerabilities that a DFS crawler would have missed. A
graphical representation of the result of a scan on Hackazon using the two different crawl-
ing techniques can be seen in Figure 5.1 and Figure 5.2. Table 5.7 concludes the graphical
representation in numbers.

Table 5.6: The total number of found vulnerabilities for OWASP ZAP AJAX Spider with BFS
on all deliberately vulnerable web application.
WVS/ Broken DVWA Hackazon Juice Security WackoPicko Total
Application Crystals Shop Shepherd
3 1 3 2 8 23 9 4 7 10 5 2
XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi XSS SQLi
ZAP AJAX BFS 2 0 3 0 2 1 0 1 0 1 3 1 14
ZAP AJAX DFS 2 0 3 1 1 1 0 1 0 0 3 1 13

Table 5.7: The crawling results for DFS and BFS running for 30 minutes on Hackazon.
Result DFS BFS
Number of states 262 184
URLs visited 97 98
Number of edges 270 185
Number of crawl paths 210 205
Failed events 884 863

42
 5.4. Control test session

Figure 5.1: Crawljax with DFS

Figure 5.2: Crawljax with BFS

43
 6 Discussion

This section aims to discuss interesting discoveries from the result presented in Section 5.
The chosen method in this thesis will be reviewed. Lastly, the thesis work is put in a wider
context.

6.1 Result
Multiple conclusions can be made by analysing the results of the tests done in this thesis and
the code bases selected. These conclusions based on the result will be discussed in the coming
sections.

6.1.1 Initial test session

The result of the initial test session presented in Section 5.1 produced some differences that
required deeper analysis to understand. The first interesting result to analyse was Arachni,
which in Table 5.1 was the one WVS that found the most vulnerabilities, but from Table 5.2
was the one WVS that produced the highest number of false positives. Note that all the false
positive vulnerabilities found, were of type XSS. The explanation for the high number of
both found vulnerabilities and false positive XSS vulnerabilities was found during the code
analysis. In Section 2.5.8, it was explained that the fuzzer in Arachni injects an ID in order to
discover an XSS vulnerability. This technique is used not only by Arachni but also by Black
Widow. The issue with Arachni is that it is always using the same ID during a crawling
session when it is trying to attack and discover XSS vulnerabilities. By using the same ID
for every attack, the ID will exist in several places in the web application. According to the
creators of Arachni, this is done for faster lookups. The reason for this is that a list would
need to be searched each time a new state is visited if multiple IDs are used.

To illustrate the problem, consider a web application which consists of page A, page B,
page C and page D shown in Figure 6.1. On page A and page B, there are two input
elements, A1 and B1, that are vulnerable to XSS attacks. Arachni is injecting the script
<xss_22f9795f4bff4665794f5b59/> in both input element A1 and B1, where the ID is the se-
quence after the dash. The script injected in input element A1 is reflected on page C and
input element B1 is reflected on page D. Then when Arachni is visiting pages C and D, it can

44
 6.1. Result

connect the injection point with the storing point wrong. Arachni can not determine if the
reflected script on page C was injected on page A or page B since the ID is the same. Black
Widow instead does have a unique ID for every script it successfully injects and saves the ID
in an array to later verify if the script was injected at the current visiting page.

Figure 6.1: Example of a web application to illustrate Arachni’s issue for identifying XSS
vulnerabilities.

Black Widow was the one WVS that had the fewest false positives presented in Table 5.2,
when looking at the most interesting WVSs. This performance is explained by Black Widow’s
solution to identify the ID of the injected script, which is described above. Eriksson et al.
[18] describe that for their solution to generate a false positive, the web application needs to
actively listen for the injected script, and extract the ID from the script. Then run the injected
script with the extracted ID in another place within the web application. The result confirms
the claim in the paper by Eriksson et al., that the technique used for stored XSS by Black
Widow makes it difficult to get a false positive.

The performance between the WVSs with support for crawling AJAX web applications
and the WVSs without this support was similar in some cases. OWASP ZAP’s traditional
crawler performed similarly to OWASP ZAP AJAX crawler in 50 % of the tested web applica-
tions. Neither of the two WVSs could find any vulnerabilities in Security Shepherd and both
of them found the SQLi in the login form in Juice Shop. In WackoPicko, both OWASP ZAP’s
crawlers found the SQLi vulnerability in the login form and the stored XSS vulnerability in
the guestbook comment section. The reason for finding both the SQLi vulnerabilities in the
login form for WackoPicko and Juice Shop is that each test is performed with an authenticated
scan. This implicates that the page where the login form exists is provided from the start and
is therefore manually included in every test if the authentication succeeds. This result does
not mean that the traditional crawler in OWASP ZAP performs well, instead it is the AJAX
crawler for OWASP ZAP that has poor performance. OWASP ZAP is a well maintained and
continuously improved open-source WVS but to crawl single-page web applications which

45
 6.1. Result

use the AJAX technology, there are still plenty of improvements to be made.

The three traditional WVSs Skipfish, Vega and Wapiti were the WVSs that performed the
worst and are therefore recommended by the authors to not be used. These WVSs are old and
have not been maintained or updated in a while. However, one interesting aspect of these
three WVSs is that Vega found several vulnerabilities in the AJAX application Hackazon.
Four of the six found SQLi vulnerabilities were false positives. Vega’s analyser determines
a successful SQLi by finding the injected payload somewhere in the DOM. In Hackazon the
payload was visible in the URL field and was not generating any SQL error message in the ap-
plication. As a result, four of the six SQLi vulnerabilities were concluded to be false positives.

In both of the AJAX web applications Juice Shop and Security Shepherd the performance
of finding vulnerabilities in all WVSs was low. Black Widow was the only WVS that found
one vulnerability of the 17 predetermined vulnerabilities that exist in Security Shepherd. In
Juice Shop did OWASP ZAP’s two different crawlers find the SQLi vulnerability in the login
form and Arachni found the XSS vulnerability in the product search field. These three WVSs
found two of the thirteen predetermined vulnerabilities that exist in Juice Shop. The reason
for the low number of found vulnerabilities in Juice Shop and Security Shepherd is not only
because they are AJAX web applications. Both of them have a hamburger menu icon, placed
in the left corner of the application, which the WVSs had a problem handling. The content
that this menu icon consists of is hidden until it has been clicked. As an event is fired for
displaying the hidden content, most of the WVSs (except Arachni, OWASP ZAP and Black
Widow) could not identify this event and therefore never try to interact with the content in
the menu.

Arachni and OWASP ZAP had a difficult time identifying the change from hidden to
displayed for the menu in the DOM when the menu icon was clicked in Security Shepherd.
They both could not identify the change as a new state. Black Widow succeeded in doing this
task in Security Shepherd but did not in Juice Shop. The reason for Black Widow performing
worse in Juice Shop is when it comes to the landing page, a pop-up window is displayed,
which can be seen in Figure 6.2, that makes the crawler get stuck. Then, Black Widow does
not visit any pages within Juice Shop. OWASP ZAP could not interact with the menu some-
times in Juice Shop if the pop-up window in Figure 6.2 was shown. This pop-up window
could be removed manually by the test leader but in the initial test session, this was a task for
the WVSs to bypass. OWASP ZAP could sometimes not identify the removal as a new state
in the DOM and could not continue to crawl Juice Shop to find the menu icon. This result
was however inconsistent because, during a new test run, OWASP ZAP succeeded in the
removal of the pop-up window. When this removal was successful, a new problem occurred.
The identification of the menu to go from hidden to display was not identified, and OWASP
ZAP could therefore not create a new state for the menu icon. Arachni had no issue with this
pop-up window but Arachni could, just as in Security Shepherd, not identify the change of
the menu from hidden to display and therefore not create a new state in its graph.

This is an important aspect when considering that Juice Shop is built using Angular and
Express. These are modern frameworks which are heavily used by developers, and it is
crucial that WVSs can handle these types of applications to be considered useful. From the
initial test session, it is clear that these types of frameworks are difficult for most WVSs to
handle. There are two possible research topics to investigate when trying to improve the
performance of WVSs when crawling web applications built with these types of frameworks.
One topic to investigate is which HTML-elements the crawler interacts with and the other
topic is a method for analysing a change in the dynamic DOM.

46
 6.1. Result

Figure 6.2: Juice Shop’s pop-up window seen at first visit.

Changes of settings
Based on the result from Section 5.1.1, two parameters that were altered are the strength and
threshold parameters. These parameters were from the beginning set as default. OWASP
ZAP is attacking every request it collects during the crawling session and by changing the
strength parameter to high, the number of payloads is increased, leading to the time of the
attack session increasing. One reason why the threshold parameter is set to default when
starting OWASP ZAP is because of the false positive rate. By having it set at low, there can be
a possibility that the number of vulnerabilities that are false positive increases. If it is set to
high instead, there is a possibility that OWASP ZAP misses detecting vulnerabilities. Default
is therefore a good starting point but in some cases, OWASP ZAP still misses detecting
vulnerabilities by having the setting set as default.

There was also another setting that was changed and that was what type of input ele-
ments OWASP ZAP interacts with by default. This setting as default only interacts with the
input element of type "button" and "submit". By enabling all types of input elements OWASP
ZAP is trying to interact with, the running time increases. This is the reason for OWASP ZAP
just to have a few of the most common interactable input elements that could be vulnerable
to attacks. The changes that were made can be seen in Table 5.5 to avoid missing the prede-
termined vulnerabilities that existed in the applications.

The result from the initial test session confirms what earlier work [37] has concluded.
That the difference in payload and what error messages are analysed for has a great impact
on the result of the scan. Another conclusion based on the need for setting changes is that the
lack of documentation for OWASP ZAP makes it difficult for a new user to easily get started.
This is something that the authors suggest needs improvement.

6.1.2 Implementation and control test session

In the discussion of the implementation, is the implemented BFS crawler discussed based on
different decisions and some limitations. It continues with a discussion of how a new tech-
nique for finding stored XSS in OWASP ZAP is possible to implement. The implementation
phase is, as Section 4 describes, based on the result from the code analysis and test session.
The validity of the result would be more reliable and valid if more extensive testing was done
on the result from the implementation.

47
 6.1. Result

BFS crawler implementation

As can be concluded by the result from Table 5.6, the proof-of-concept BFS crawler imple-
mented for this thesis performed equal to the DFS crawler when analysing the number of
found vulnerabilities, except for Hackazon. On Hackazon, one more XSS vulnerability was
found on the landing page. The result is further visualised by Figure 5.2. From the figure, it
is easily seen that the result from the implemented BFS has the behaviour of an expected BFS
crawler. Each layer from the landing page is explored one at a time in a way which makes the
result bear resemblance to the ripple effect on water. As discussed in Section 2.5.1, problems
occur when scanning a web application with a lot of possible paths from the landing page,
using a DFS crawler. This thesis, therefore, concludes that a BFS crawler is better suited for
testing web applications similar to Hackazon. These web applications include online retailers
such as Amazon, where there are a lot of possible state changes on the landing page.

A decision was made during the implementation phase to implement the BFS crawler
by changing the behaviour of the already used AJAX crawler (Crawljax) in OWASP ZAP.
This decision was done due to an API already existing for communication between OWASP
ZAP and Crawljax. It was therefore deemed more feasible when considering the time con-
straint put on this thesis, to use this API and crawler. The time constraint also put focus on
making this crawler a proof-of-concept. There are mainly two things which would need to be
updated and optimised to make this a final product. First, is the lack of opportunity to run
more than one browser per crawling session. Crawljax has the option to run multiple threads
(one browser per thread), which makes the crawling more effective. This was however not
something that was taken into consideration when implementing the BFS crawler. Based
on the code analysis, this should be able to be implemented using a shared resource with
locking. The second concern is optimising the gathering of possible actions in the current
state. Because Crawljax is implemented as a DFS crawler, there is no need to store the
possible actions extracted when changing states, since a state is explored when it is found.
When implementing a BFS crawler, all neighbouring states are found before starting the
exploration. The current implementation gathers the possible actions when visiting the state
for the second time since the first visit is only for deciding whether the newly found state is
a clone or has already been visited. To optimise the current implementation another field to
the array list of the ordered map described in Section 5.3 can be added, which contains the
actions. However, this solution can have a negative effect on the amount of memory used
during crawling, depending on the number of branches on each level.

The need for optimisation is confirmed when analysing Table 5.7. When running Hack-
azon for 30 minutes, the DFS crawler has visited almost 80 states more compared to the
BFS crawler and has 85 more edges, which represents the fired event. An important note to
consider is that even if there are more states visited in the DFS crawler, the number of URLs
visited is marginally higher when running the BFS crawler. This is concluded to be because
of the number of unique URLs being higher on the landing page. The implementation still
uses the crawl depth parameter to stop the crawling from going any further. This is done due
to the crawler still being able to get stuck in an infinite loop since the Levenshtein algorithm
is used by Crawljax on OWASP ZAP with the default threshold of 0. Means, any detected
changes in the DOM will be counted as a new state.

Stored XSS implementation

Another decision made during the implementation was to only implement the BFS crawler
and not the technique used by Black Widow to find stored XSS vulnerabilities. There are
multiple reasons for this choice. Firstly, it was deduced while doing the code analysis that
implementing this technique was not feasible during the time frame of this thesis. A lot of
changes to the OWASP ZAP code base would have been necessary to do since the sequence
of actions needed to be recorded and stored. There exists an add-on for OWASP ZAP called

48
 6.2. Method discussion

"Sequence scanner". In this add-on, the user needs to add a Zest script specifying the specific
sequence. This means the user needs to know the specific sequence for which the vulner-
ability might be executed. A possible solution for the implementation could be to try to
automatise this add-on and record the sequence during crawling. Another possibility is to
implement the attack phase during crawling. This would mean injecting scripts, searching
for them and if the earlier injected script is found, reporting it as a possible vulnerability.
There is however one problem with this solution and that is the modularity in OWASP ZAP.
As was earlier described in Section 2.5.8, the crawling phase is done before the attacking
phase in OWASP ZAP. Doing this type of implementation would mean for the user to decide
before the crawling if a scan for stored XSS vulnerabilities is going to be done. Code for doing
attacks would also be needed to be added to the crawler, such as payloads and randomised
IDs to identify the source and sink nodes. This would break the design choice made by
OWASP ZAP to keep everything as modular as possible.

6.2 Method discussion

This section aims to discuss the applied method used in this thesis to answer the research
questions. This will be done by attempting to acknowledge the alternative methods and
limitations. The section is partially divided in the same way as Section 4, which makes it
possible to critically view each part of the method separately.

6.2.1 Literature study

The literature study done in this thesis was the first step on which all the other steps in the
method relied on. The result from the study can be seen as successful, as enough information
and knowledge have been gathered to do the other steps. By analysing the selected papers,
a few things can be mentioned which could have been improved. First, the specification
which the selected papers were compared to. In the study, the specifications were focused
on the number of citations, whether the publication in which the papers were published was
credible or not and the reputation of the author. This could however not always be followed
during the study, since a few papers gathered still were deemed trustworthy and reliable even
if the number of citations was low. The reasons these papers were still deemed trustworthy
varied but can be summarised to be because of the way those studies were executed and the
sources the studies were based on. From this, it can be concluded that when the citations
are low, or some other aspect in the specification list is not achieved, the paper needs to be
analysed using other requirements.

6.2.2 Selection of WVSs and applications

The reasons for selecting which WVSs to test can easily be criticised. There is a multitude of
available WVSs and only a few were tested in this thesis. The main reason that limited the
choice of WVSs was the need for each tested scanner to be open-source. Another element
that limited the choice was the need for the majority of the scanners to have an AJAX crawler.
A few scanners still needed to be removed after the literature study, such as Enemy-of-the-
state. Including these scanners and more in future studies should increase the possibility
of finding interesting techniques used, and give more options on how to improve OWASP
ZAP. Another decision which affected the validity of the thesis is the choice of applications
tested. Just as with the chosen WVSs, there was a need for the majority of the application to
use the AJAX technology. This limited the possible applications available. One thing which
was not done in this thesis, but can be done to make the validity higher, is to customise
the vulnerabilities in the applications which support customisation. This would make it less
likely that the result is tampered with, due to WVSs being specifically trained on the known

49
 6.2. Method discussion

vulnerable web applications. The reason for this not being done in this study is simply the
time constraint.

6.2.3 Initial test session

The tests were done using the default settings for all scanners and an authenticated scan
where needed. A problem with doing tests on software which are currently being maintained
is that the software might get a vast update which changes the result or which features are
supported by the WVS. This makes replicability more difficult if no consideration is taken
to what versions are being used. In Table 4.2 the most recent version (when this thesis was
written) is stated and this is the version used for all tests. If another study were to be done
to confirm the reliability of the result, these are the versions which should be used. Another
issue is the fact that the difference in ease of use also affects what is considered to be default
settings. This could be seen when configuring for an authenticated scan, where OWASP
ZAP had multiple choices for what type of message would be sent during the authentication.
Whereas in Wapiti the user can only add the username and password as a parameter in the
program call. This affected the result, where OWASP ZAP was able to log in to most of the
applications tested and Wapiti could not. To make the study more replicable and reliable, a
further specifications list on what can be considered to be default settings is needed.

6.2.4 Code analysis

The method for the code analysis used in this study is based on the research presented in the
paper by Dunsmore et al. [16]. A possible change to make this analysis more reliable and
valid could be to use different methods for analysing the code. Comparing the result from
these methods and identifying if the result is independent of the analysis method used. The
reason for this not being done in this thesis is due to the result being deemed to be reliable
enough for making the necessary implementations.

Bug in OWASP ZAP

Another concern identified in OWASP ZAP was a bug, which is not a specific technique
but is still interesting to note. This bug occurred when scanning for XSS vulnerabilities in
WackoPicko. The issue identified was that OWASP ZAP did not inject data in all parameters
during a specific POST request. It missed the last parameter in the POST request. In this
specific case, the last parameter was the one vulnerable to a reflected XSS. This bug could
be traced to be a problem with adding the number of parameters in the POST request. The
element before the last element in this specific POST request was of type "userfile" and when
this occurred, the code never added to the number of parameters in the list. This led to the
last element not having the correct index, which meant the element before the last element
was attacked twice, and the last element was never attacked. By fixing this bug, the reflected
XSS vulnerability in WackoPicko was discovered.

Changes to Black Widow

During the initial test session, Black Widow gave two different types of error messages dur-
ing the scan. The first error message was: unhashable type: "dict". This caused the scanning
process to stop. The error message arose when Black Widow was trying to extract the ID of
an input element within a form but failed to find it. When Black Widow is trying to extract
the input element, a dictionary is created to put the result into. If this fails, as in this case, the
dictionary is empty which generates an error message when the dictionary is being hashed.
To avoid the error message, control of the dictionary was added as a solution. The control
checks if the dictionary is empty and if it is, the dictionary is converted to an empty string
since that is a hashable data type.

50
 6.3. The work in a wider context

The second error message that occurred during the initial test session was: type error: object
not iterable. This error message occurred when Black Widow was trying to loop a list with
successful XSS attacks to be written to a file, which contained a none iterable object. This
was an error message, which also caused the scanning process to stop. To handle this error
message, a try-except statement was added. Inside the try-block is the for-loop for checking
if an attack was successfully performed and if it is, the attack is written to a file. In the except
block, a print statement is written to the terminal telling the user why the error occurred.

Black Widow, as stated before in Section 2.5.7, is a proof-of-concept WVS, which can consist
of some small bugs or performance limitations. In this case, it was two bugs regarding the
error handling. These two errors do not affect the decisions the crawler takes but terminates
the crawling session and no result is produced. By adding the previously described solu-
tions for managing the two errors, the whole scanning process was able to run without any
interruptions of wrongly handled errors and produced a result.

6.3 The work in a wider context

By putting WVSs and this thesis work in a wider context, there is both societal and ethical
aspect to take into consideration when using this tool worldwide.

6.3.1 Societal aspects

One societal aspect regarding WVSs is if stakeholders are using WVSs on their web appli-
cations for maintaining them to be vulnerability free, the users can trustfully browse these
applications and not be targeted for attacks. By keeping the maintaining procedure of the
application, the trust of the user increases and the stakeholders can gain more revenue from
the web application. On the other hand, if a vulnerability is exploited on a web application it
can have economical effects. As was brought up in 1.1, a SQLi vulnerability can lead to data
being leaked and this can affect the number of users who are willing to trust the company
with their data. This would lead to a drop in users and revenue. This thesis aims to explore
the possibility to make WVSs more effective. Because of that, hopefully, this thesis makes
it less likely that stakeholders miss an important vulnerability and the risk of a drop in the
number of users because of data leakages decreases.

6.3.2 Ethical aspects

One ethical aspect regarding WVSs is if their intended use of them is violated. The intended
use of a WVS is for developers to use this tool to find vulnerabilities in web applications
they are creating, and to avoid attackers to exploit sensitive information. On the other hand,
attackers can violate the intention of use by instead using this tool to find vulnerabilities and
exploit the vulnerabilities by performing attacks on the web applications. In this thesis, open-
source WVSs are the main focus, and they are free to use for all. The thesis aims to research
possible aspects of an open-source WVS which can and should be improved. As always when
improving software like this it is important to remember the earlier mentioned misuse of a
WVS. An attacker could even use the result from this thesis and download the source code
of a WVS to make modifications that could improve the WVS for finding vulnerabilities.
For example, the BFS crawler implemented could be used for the wrong purposes. This is
something which has been reflected on during this thesis, but no solution that effectively
minimises the risk has been found since the code is open-source. There are many ways open-
source WVSs can be used to achieve a higher security standard for developers creating web
applications, but there is also no way to change the fact that WVSs can be used with evil
intentions.

51
 7 Conclusion

In the following section are the thesis’s research questions answered and interesting aspects
of how the result of this thesis could be used in future work.

7.1 Research questions

This thesis has aimed to evaluate the current status of WVSs ability to find vulnerabilities in
web applications using AJAX technologies. Then investigate the differences in techniques
to identify what possible changes can affect the result for OWASP ZAP. This section aims to
return and answer the formulated research questions in Section 1.3.

1. Which different techniques exist to detect SQL injections and XSS in web vulnerabilities scanners,
and how do they theoretically compare to the one existing in OWASP ZAP?

To answer the first research question has this thesis done a literature study and a code
analysis to identify the existing techniques and then compared OWASP ZAP’s techniques
with Aracnhi and Black Widow. The existing techniques for identifying SQLis are mainly
different payloads injected into an input field and then the response with data from the
database is sent back to the web application and analysed. Techniques for identifying XSS
vulnerabilities are by injecting a JavaScript code into an input field and searching for it inside
the web application. The analysis process is dependent on which type of XSS vulnerability
the WVS is searching for. In stored and reflected XSS the HTTP response is searched and in
DOM-based XSS is the DOM searched.

The differences in techniques used in OWASP ZAP compared to Arachni and Black Widow
for detecting XSS vulnerabilities are when and how. Arachni and Black Widow are injecting
the script during the crawling session and simultaneously searching for the injected script.
In OWASP ZAP the injecting of scripts happen after the crawling session, in a phase called
attack. During this phase, OWASP ZAP only searches for the injected script in the pages
already discovered, while Arachni and Black Widow can search for the script while dis-
covering new pages. Furthermore, Black Widow injects the script with an ID that is saved
to correctly report where the script was injected and where it was found. However, this
technique is not used by OWASP ZAP. For SQLi OWASP ZAP with default settings, does not

52
 7.1. Research questions

analyse the error messages based on what database is used for the web application, which
Arachni takes into consideration. Based on this result, it can be concluded that to make your
automated testing as exhaustive as possible, multiple WVSs should be used during the test
session of the web application under development.

2. How does the detection of vulnerabilities for these techniques differs between single-page ap-
plications and multi-page applications?

It was discussed in the motivation of this thesis, which challenges a WVS has to circum-
vent. These challenges include handling different network requests. Based on the outcome
of the literature study done in this thesis, the main key point is that the difference between
scanning single-page applications and multi-page applications is the way the crawling is
handled. This is due to the earlier described AJAX engine used in a single-page application.
The difference, therefore, does not lie in the type of input fields which are searched for by
the crawlers, but rather how a new state is defined. A new state, as was described in the
literature study, is defined by a traditional crawler in a multi-page application to be a new
URL. In an AJAX crawler, the definition of a new state differs between the WVSs, however, it
can be summarised to be a change in the DOM.

Based on the result of the initial test session, it can be observed that for some single-page
applications the result was a bit unexpected. The number of found vulnerabilities using a
WVS which supports an AJAX crawler did surprisingly not find more vulnerabilities than a
WVS which uses a traditional crawler. This implies that there is still more research needed to
vastly improve the result from crawling single-page applications.

3. How does the performance of some web vulnerability scanners implementing these techniques
compare to the performance of OWASP ZAP?

The earlier mentioned techniques used by other WVSs have been analysed to be more
effective in certain applications. Black Widow using the technique for stored XSS is currently
the most successful when comparing it to all other WVSs, based on the result in Section 5.
Another aspect to consider is if SQLis are analysed with database-specific messages, more
SQLi vulnerabilities are then detected compared to when using general database messages
as in the default settings for OWASP ZAP. Another technique Black Widow uses is a BFS
crawler while OWASP ZAP is using a DFS Crawler. The difference between these two
crawling algorithms is that neither of them has the ability to perform as intended for all
different types of web applications. A BFS crawler has difficulty crawling web applications
with a large depth, while a DFS crawler has difficulty crawling web applications with several
branches in a few levels of depth.

As stated in the motivation, there are different types of crawlers to use for single- and
multi-page web applications. However, the decision on which type of crawler to use can also
be based on the structure of the web application. A potential feature for WVSs is combining
different types of crawler algorithms. The users can by having more options themselves
decide which crawler algorithm they prefer or apply the best-suited crawler algorithm if
they know the structure of the web application.

4. Is it possible to implement these techniques in OWASP ZAP? If so, which modifications are needed?

This research question is divided into two parts. One conclusion for the first part, this
thesis has identified two techniques to add to OWASP ZAP which would affect the result of
the scan. The BFS crawler is one of these and has been implemented as a proof-of-concept
and tested. The tests have demonstrated for which type of applications this implementation

53
 7.2. Future work

would make a difference in the scanning result, mainly online retailers such as Amazon.
What these applications have in common is the massive amounts of possible actions on the
landing page. The thesis has also identified a second change which would possibly break the
design pattern used in OWASP ZAP, the stored XSS technique used by Black Widow. The
second part of this question has been thoroughly analysed in the discussion. The conclusion
about the stored XSS technique and how this would be implemented, reveal possible short-
comings in a modular design such as OWASP ZAP. This is something that each developer
needs to take into consideration when designing programs, where the advantages and dis-
advantages should be listed.

In the introduction of the thesis, it was stated that manual testing is a time-consuming
task and that automated tests are desirable when possible. This is done to in the end save
money for the company doing the testing. By looking at the numbers of found vulnerabili-
ties in OWASP ZAP and comparing it to the numbers of actual vulnerabilities in each web
application a conclusion can be made. That manual testing is still required as a complement
to automated testing to fully test a web application. From this, it can also be concluded that
automated testing is a field in which further research is necessary.

7.2 Future work

There is a multitude of possible future works based on this thesis. Firstly, it would be in-
teresting to finalise the code for the BFS crawling and make it production-ready. This thesis
already discusses potential actions required for doing this, by optimising the implementa-
tion. Secondly, investigating the opportunity of changing the crawler to another available
open-source project in OWASP ZAP, could be a possible way to more effectively crawl AJAX
applications. Thirdly, making the necessary steps for implementing the technique used in
Black Widow to find stored XSS vulnerabilities is another possible way for continuing work-
ing in this area. This is however discussed in this thesis and might if implemented, make
OWASP ZAP less modular. A final objective when researching a possible implementation is
to investigate the payload list and error messages for SQLi used in OWASP ZAP, and which
changes to these lists would make the scan more effective.

Further research about crawlers could be interesting since this was shown to be a weak
part of the current WVSs for AJAX applications. This can be done using other metrics, such
as the number of code lines executed in the tested web applications and crawling time. If
the source code were accessible, it would be intriguing to compare techniques and results of
commercial and open-source WVSs and to do a similar study like this one.

54
 Bibliography

[1] Adobe Flash Player EOL General Information Page. URL: https://www.adobe.com/
products/flashplayer/end-of-life.html (visited on 03/31/2022).
[2] Mini Singh Ahuja, Dr Jatinder Singh Bal, and Varnica. “Web crawler: Extracting the web
data”. In: International Journal of Computer Trends and Technology 13.3 (2014), pp. 132–137.
[3] Richard Amankwah, Jinfu Chen, Patrick Kwaku Kudjo, and Dave Towey. “An empir-
ical comparison of commercial and open-source web vulnerability scanners”. In: Soft-
ware: Practice and Experience 50.9 (2020), pp. 1842–1857. DOI: 10.1002/spe.2870.
[4] Arachni. URL: https://downloads.arachni- scanner.com/nightlies/ (vis-
ited on 03/03/2022).
[5] Jeffrey Beall. Potential predatory scholarly open-access publishers. 2021. URL: https : / /
beallslist.net/ (visited on 02/04/2022).
[6] Tim Berners-Lee, Roy Fielding, and Henrik Frystyk. Hypertext transfer protocol–
HTTP/1.0. 1996.
[7] BlackWidow. URL: https://github.com/SecuringWeb/BlackWidow (visited on
02/04/2022).
[8] Broken Crystals. URL: https : / / github . com / NeuraLegion / brokencrystals
(visited on 03/01/2022).
[9] Junghoo Cho and Hector Garcia-Molina. The Evolution of the Web and Implications for
an Incremental Crawler. Technical Report 1999-22. Stanford InfoLab, 1999. URL: http:
//ilpubs.stanford.edu:8090/376/.
[10] James Clark, Steve DeRose, et al. XML path language (XPath). 1999.
[11] Steven Cook. A Web Developer’s Guide to Cross-Site Scripting. Jan. 2003.
[12] Damn Vulnerable Web Application. URL: https://github.com/digininja/DVWA
(visited on 02/04/2022).
[13] Chiragh Dawson. IronWASP: An introduction. 2017. URL: https : / / resources .
infosecinstitute.com/topic/ironwasp-part-1-2/ (visited on 03/02/2022).

55
 Bibliography

[14] Adam Doupé, Ludovico Cavedon, Christopher Kruegel, and Giovanni Vigna. “En-
emy of the State: A State-Aware Black-Box Web Vulnerability Scanner”. In: 21st
USENIX Security Symposium (USENIX Security 12). Bellevue, WA: USENIX Associa-
tion, Aug. 2012, pp. 523–538. URL: https : / / www . usenix . org / conference /
usenixsecurity12/technical-sessions/presentation/doupe.
[15] Adam Doupé, Marco Cova, and Giovanni Vigna. “Why Johnny can’t pentest: An anal-
ysis of black-box web vulnerability scanners”. In: International Conference on Detection of
Intrusions and Malware, and Vulnerability Assessment. Springer. 2010, pp. 111–131.
[16] Alastair Dunsmore, Marc Roper, and Murray Wood. “Further Investigations into the
Development and Evaluation of Reading Techniques for Object-Oriented Code Inspec-
tion”. In: Proceedings of the 24th International Conference on Software Engineering. ICSE
’02. Orlando, Florida: Association for Computing Machinery, 2002, pp. 47–57. ISBN:
158113472X. DOI: 10.1145/581339.581349.
[17] Enemy-of-the-state. URL: https : / / github . com / adamdoupe / enemy - of - the -
state (visited on 03/01/2022).
[18] Benjamin Eriksson, Giancarlo Pellegrino, and Andrei Sabelfeld. “Black Widow: Black-
box Data-driven Web Scanning”. In: 2021 IEEE Symposium on Security and Privacy (SP).
2021, pp. 1125–1142. DOI: 10.1109/SP40001.2021.00022.
[19] Edmund Foster. Testing Web Application Security Scanners against a Web 2.0 Vulnerable
Web Application. 2018.
[20] Jesse James Garrett. Ajax: A new approach to web applications. 2005.
[21] Hackazon. URL: https://github.com/rapid7/hackazon (visited on 02/04/2022).
[22] William G Halfond, Jeremy Viegas, and Alessandro Orso. “A classification of SQL-
injection attacks and countermeasures”. In: Proceedings of the IEEE international sym-
posium on secure software engineering. Vol. 1. IEEE. 2006, pp. 13–15.
[23] Zar Chi Su Su Hlaing and Myo Khaing. “A Detection and Prevention Technique on
SQL Injection Attacks”. In: 2020 IEEE Conference on Computer Applications(ICCA). 2020,
pp. 1–6. DOI: 10.1109/ICCA49400.2020.9022833.
[24] Hsiu-Chuan Huang, Zhi-Kai Zhang, Hao-Wen Cheng, and Shiuhpyng Winston Shieh.
“Web Application Security: Threats, Countermeasures, and Pitfalls”. In: Computer 50.6
(2017), pp. 81–85. DOI: 10.1109/MC.2017.183.
[25] Siham el Idrissi, Naoual Berbiche, Fatima Guerouate, and Sbihi Mohamed. “Perfor-
mance evaluation of web application security scanners for prevention and protection
against vulnerabilities”. In: International Journal of Applied Engineering Research 12 (Jan.
2017), pp. 11068–11076.
[26] IronWASP. URL: https : / / github . com / Lavakumar / IronWASP (visited on
03/02/2022).
[27] Nishtha Jatana, Sahil Puri, Mehak Ahuja, Ishita Kathuria, and Dishant Gosain. “A sur-
vey and comparison of relational and non-relational database”. In: International Journal
of Engineering Research & Technology 1.6 (2012), pp. 1–5.
[28] Kali Linux. URL: https://www.kali.org/ (visited on 02/04/2022).
[29] Stefan Kals, Engin Kirda, Christopher Kruegel, and Nenad Jovanovic. “SecuBat: A Web
Vulnerability Scanner”. In: Proceedings of the 15th International Conference on World Wide
Web. WWW ’06. Edinburgh, Scotland: Association for Computing Machinery, 2006,
pp. 247–256. ISBN: 1595933239. DOI: 10.1145/1135777.1135817.
[30] Mohd Ehmer Khan and Farmeena Khan. “A Comparative Study of White Box, Black
Box and Grey Box Testing Techniques”. In: International Journal of Advanced Computer
Science and Applications 3.6 (2012). DOI: 10.14569/IJACSA.2012.030603.

56
 Bibliography

[31] Swati Khandelwal. Fortnite Flaws Allowed Hackers to Takeover Gamers’ Accounts. 2019.
URL : https://thehackernews.com/2019/01/fortnite- account- hacked.
html (visited on 03/03/2022).
[32] Engin Kirda, Christopher Kruegel, Giovanni Vigna, and Nenad Jovanovic. “Noxes: A
Client-Side Solution for Mitigating Cross-Site Scripting Attacks”. In: Proceedings of the
2006 ACM Symposium on Applied Computing. SAC ’06. Dijon, France: Association for
Computing Machinery, 2006, pp. 330–337. ISBN: 1595931082. DOI: 10.1145/1141277.
1141357.
[33] Amit Klein. “DOM based cross site scripting or XSS of the third kind”. In: Web Applica-
tion Security Consortium, Articles 4 (2005), pp. 365–372.
[34] Kevin Jonathan Koswara and Yudistira Dwi Wardhana Asnar. “Improving Vulnera-
bility Scanner Performance in Detecting AJAX Application Vulnerabilities”. In: 2019
International Conference on Data and Software Engineering (ICoDSE). 2019, pp. 1–5. DOI:
10.1109/ICoDSE48700.2019.9092613.
[35] Sebastian Lekies, Ben Stock, and Martin Johns. “25 Million Flows Later: Large-Scale
Detection of DOM-Based XSS”. In: Proceedings of the 2013 ACM SIGSAC Conference on
Computer &amp; Communications Security. CCS ’13. Berlin, Germany: Association for
Computing Machinery, 2013, pp. 1193–1204. ISBN: 9781450324779. DOI: 10 . 1145 /
2508859.2516703.
[36] Vladimir I Levenshtein et al. “Binary codes capable of correcting deletions, insertions,
and reversals”. In: Soviet physics doklady. Vol. 10. 8. Soviet Union. 1966, pp. 707–710.
[37] Erik Matti. “Evaluation of open source web vulnerability scanners and their tech-
niques used to find SQL injection and cross-site scripting vulnerabilities”. MA thesis.
Linköping University, Department of Computer and Information Science, 2021, p. 50.
[38] Barton P. Miller, Louis Fredriksen, and Bryan So. “An Empirical Study of the Reliability
of UNIX Utilities”. In: Commun. ACM 33.12 (Dec. 1990), pp. 32–44. ISSN: 0001-0782. DOI:
10.1145/96267.96279.
[39] Charlie Miller, Zachary N.J. Peterson, et al. “Analysis of mutation and generation-based
fuzzing”. In: Independent Security Evaluators, Tech. Rep 4 (2007).
[40] The Hacker News. Penetration Testing Your AWS Environment - A CTO’s Guide. 2021.
URL : https://thehackernews.com/2021/10/penetration-testing-your-
aws.html (visited on 12/16/2021).
[41] Peter Oehlert. “Violating assumptions with fuzzing”. In: IEEE Security Privacy 3.2
(2005), pp. 58–62. DOI: 10.1109/MSP.2005.55.
[42] Oracle VM VirtualBox. URL: https : / / www . virtualbox . org/ (visited on
02/02/2022).
[43] OWASP Juice Shop. URL: https://github.com/juice- shop/juice- shop (vis-
ited on 02/04/2022).
[44] OWASP Security Shepard. URL: https://github.com/OWASP/SecurityShepherd
(visited on 02/04/2022).
[45] OWASP Vulnerable Web Applications Directory. URL: https : / / owasp . org /
www - project - vulnerable - web - applications - directory/ (visited on
03/30/2022).
[46] OWASP ZAP. URL: https : / / github . com / zaproxy / zaproxy (visited on
02/04/2022).
[47] Van-Thuan Pham, Marcel Böhme, Andrew E. Santosa, Alexandru Răzvan Căciulescu,
and Abhik Roychoudhury. “Smart Greybox Fuzzing”. In: IEEE Transactions on Software
Engineering 47.9 (2021), pp. 1980–1997. DOI: 10.1109/TSE.2019.2941681.

57
 Bibliography

[48] Marc Rennhard, Damiano Esposito, Lukas Ruf, and Arno Wagner. “Improving the ef-
fectiveness of web application vulnerability scanning”. In: International Journal on Ad-
vances in Internet Technology 12.1/2 (2019), pp. 12–27.
[49] Vladislav Shkapenyuk and Torsten Suel. “Design and implementation of a high-
performance distributed Web crawler”. In: Proceedings 18th International Conference on
Data Engineering. 2002, pp. 357–368. DOI: 10.1109/ICDE.2002.994750.
[50] Vandana Shrivastava and Prashant Sahai. “Comparative Analysis of Web Crawling Al-
gorithms for Improvement in Web Crawler”. In: Proceedings of International Conference
on Sustainable Computing in Science, Technology and Management (SUSCOM), Amity Uni-
versity Rajasthan, Jaipur-India. 2019.
[51] Skipfish. URL: https://gitlab.com/kalilinux/packages/skipfish (visited
on 02/04/2022).
[52] Andrew van der Stock, Brian Glas, Neil Smithline, and Torsten Gigler. OWASP Top
Ten. 2021. URL: https : / / owasp . org / www - project - top - ten/ (visited on
11/15/2021).
[53] International Telecommunication Union. Measuring digital development. Facts and figures
2020. 2020. URL: https://www.itu.int/en/ITU-D/Statistics/Documents/
facts/FactsFigures2020.pdf.
[54] Vega. URL: https://github.com/subgraph/Vega (visited on 02/07/2022).
[55] Akanksha Verma, Amita Khatana, and Sarika Chaudhary. “A Comparative Study of
Black Box Testing and White Box Testing”. In: International Journal of Computer Sciences
and Engineering 5 (Dec. 2017), pp. 301–304. DOI: 10.26438/ijcse/v5i12.301304.
[56] Akanksha Verma, Amita Khatana, and Sarika Chaudhary. “A comparative study of
black box testing and white box testing”. In: Int. J. Comput. Sci. Eng 5.12 (2017), pp. 301–
304.
[57] W3AF. URL: https : / / github . com / andresriancho / w3af (visited on
02/04/2022).
[58] WackoPicko. URL: https : / / github . com / adamdoupe / WackoPicko (visited on
03/01/2022).
[59] Wapiti. URL: https : / / github . com / wapiti - scanner / wapiti (visited on
02/04/2022).
[60] WebGoat. URL: https://github.com/WebGoat/WebGoat (visited on 03/03/2022).
[61] Lauren Wood, Arnaud Le Hors, Vidur Apparao, Steve Byrne, Mike Champion, Scott
Isaacs, Ian Jacobs, Gavin Nicol, Jonathan Robie, Robert Sutor, et al. “Document object
model (dom) level 1 specification”. In: W3C recommendation 1 (1998).
[62] Linxuan Yu, Yeli Li, Qingtao Zeng, Yanxiong Sun, Yuning Bian, and Wei He. “Summary
of web crawler technology research”. In: Journal of Physics: Conference Series 1449.1 (Jan.
2020), p. 012036. DOI: 10.1088/1742-6596/1449/1/012036.
[63] Hubert Zimmermann. “OSI Reference Model - The ISO Model of Architecture for Open
Systems Interconnection”. In: IEEE Transactions on Communications 28.4 (1980), pp. 425–
432. DOI: 10.1109/TCOM.1980.1094702.

58
 A Appendix A

A.1 Configurations for scanners during testing

Arachni
./arachni [URL]
–check=xss,sql_injection
–plugin=autologin:url=[ loginUrl],parameters="[userField]=[username ]&[passField]=[password]",check="[logout
string]”
–scope-exclude-pattern=[URL]
–timeout HH:MM:SS

Black Widow
python3 crawl.py –url URL

To configure credentials when accessing restricted parts of the web applications, the in-
put to textarea/text for username and password was changed on row 776, 778 and 784 in
Functions.py. To blacklist certain URLs, remove the comment on row 294 in Functions.py and
add the desired string, which is a signature of the blacklisted URL.

OWASP ZAP
Open up OWASP ZAP.

1. Manual Explore on [URL] and sign in with username: [username] and password: [pass-
word], then sign out.

2. Create a context from include in context.

3. Change URL regex to [URL].*

4. Change Authentication to Form-based or JSON-based authentication.

5. Pick POST:login()(password, username) as login form target URL and URL to GET Lo-
gin Page.

6. Make sure username/password parameters are correct.

59
 A.1. Configurations for scanners during testing

7. Regex pattern used to identify logged out messages is an HTML element found in the
web application.

8. Create a user by adding [username] and [password] to users.

9. Right click on Post:login()(password, username) and choose Spider/AJAX spider to

crawl web app. Select the created context and user.

10. Right-click on the [URL] directory and run an active scan. Select the created context
and user. Then run.

11. Generate vulnerability report and URL report.

Skipfish
skipfish -uv -o [output] –auth-form [loginUrl] –auth-user-field [userField] –auth-pass-field [passField]
–auth-user [username] –auth-pass [password] –auth-verify-url [verifyUrl] [url]

Vega
Open up Vega.

1. Create a user by clicking on the person with green plus icon

2. Name the user as [name] and choose macros as authentication type.

3. Create a new macro.

4. Name the macro to [name].

5. Press add item and select the post request to the login page for the desired scanning
URL.

6. Mark the item in the item list and change the user credential to [username] and [pass-
word] if it is not filled in already. Then next and finish.

7. Under scan, got to edit target scope.

8. Add a new scope called [name].

9. Add [URL] in the base path.

10. Add [URL or regex pattern] to the exclude field.

11. Press OK and then run a scan with the edited scope [name] and with the user [name].

Wapiti
wapiti -f html -o [output] -u URL -m sql,blindsql,xss -a [username]%[password]

W3AF
First, log in to web application in a browser and monitored the network in the dev tool for
the used browser. Save the cookie to a .txt file called headers_file. Then start W3AF by running
the command: python2 ./w3af_console

1. plugins

2. audit sqli

3. audit xss

4. crawl web_spider

5. crawl config web_spider

60
 A.1. Configurations for scanners during testing

6. set only_forward true

7. set follow_regex .*

8. set ignore_regex [regex pattern for desired blacklisted URL]

9. back

10. auth detailed

11. auth config detailed

12. set username [username]

13. set password [password]

14. set username_field [field_name]

15. set password_field [field_name]

16. set auth_url [log in URL]

17. set check_url [URL that only exist for authenticated user]

18. set check_string "string only visible for authenticated user"

19. set data_format [format of sent credentials]

20. back

21. output html_file

22. back

23. profiles

24. save_as [profile name]

25. back

26. target

27. set target [URL]

28. back

29. http-settings

30. set headers_file [.txt retrieved in the beginning]

31. set timeout 15

32. back

33. start

61