CHAPTER ONE

Security Testing: A Survey

Michael Felderer*, Matthias Büchler†, Martin Johns‡,
Achim D. Brucker‡, Ruth Breu*, Alexander Pretschner†
*University of Innsbruck, Innsbruck, Austria
†
Technische Universität München, Munich, Germany
‡
SAP, Karlsruhe, Germany

Contents
1. Introduction 2
2. Software Testing 3
3. Security Engineering 6
3.1 Basic Concepts 6
3.2 Secure Software Development Life Cycle 9
4. Security Testing 10
4.1 Basic Concepts 10
4.2 Security Testing in the Secure Software Development Life cycle 14
5. Security Testing Techniques 16
5.1 Model-Based Security Testing 16
5.2 Code-Based Testing and Static Analysis 21
5.3 Penetration Testing and Dynamic Analysis 24
5.4 Security Regression Testing 27
6. Application of Security Testing Techniques 31
6.1 Selection Criteria for Security Testing Approaches 31
6.2 A Three-Tiered Business Application 33
7. Summary 40
Acknowledgments 41
References 41
About the Authors 49

Abstract
Identifying vulnerabilities and ensuring security functionality by security testing is a
widely applied measure to evaluate and improve the security of software. Due to
the openness of modern software-based systems, applying appropriate security testing
techniques is of growing importance and essential to perform effective and efficient
security testing. Therefore, an overview of actual security testing techniques is of high
value both for researchers to evaluate and refine the techniques and for practitioners to
apply and disseminate them. This chapter fulfills this need and provides an overview of
recent security testing techniques. For this purpose, it first summarize the required

Advances in Computers, Volume 101 # 2016 Elsevier Inc. 1

ISSN 0065-2458 All rights reserved.
http://dx.doi.org/10.1016/bs.adcom.2015.11.003
2 Michael Felderer et al.

background of testing and security engineering. Then, basics and recent developments
of security testing techniques applied during the secure software development life
cycle, ie, model-based security testing, code-based testing and static analysis, penetra-
tion testing and dynamic analysis, as well as security regression testing are discussed.
Finally, the security testing techniques are illustrated by adopting them for an example
three-tiered web-based business application.

1. INTRODUCTION
Modern IT systems based on concepts like cloud computing,
location-based services, or social networking are permanently connected
to other systems and handle sensitive data. These interconnected systems
are subject to security attacks that may result in security incidents with high
severity affecting the technical infrastructure or its environment. Exploited
security vulnerabilities can cause drastic costs, eg, due to downtimes or the
modification of data. A high proportion of all software security incidents is
caused by attackers who exploit known vulnerabilities [1]. An important,
effective, and widely applied measure to improve the security of software
are security testing techniques which identify vulnerabilities and ensure
security functionality.
Software testing is concerned with evaluation of software products and
related artifacts to determine that they satisfy specified requirements, to
demonstrate that they are fit for purpose and to detect defects. Security test-
ing verifies and validates software system requirements related to security
properties like confidentiality, integrity, availability, authentication, autho-
rization, and nonrepudiation. Sometimes security properties come as classi-
cal functional requirements, eg, “user accounts are disabled after three
unsuccessful login attempts” which approximates one part of an authoriza-
tion property and is aligned with the software quality standard ISO/IEC
9126 [2] defining security as functional quality characteristic. However, it
seems desirable that security testing directly targets the above security prop-
erties, as opposed to taking the detour of functional tests of security mech-
anisms. This view is supported by the ISO/IEC 25010 [3] standard that
revises ISO/IEC 9126 and introduces Security as a new quality characteristic
which is not included in the characteristic functionality any more.
Web application security vulnerabilities such as Cross-Site Scripting or
SQL Injection, which can adequately be addressed by security testing tech-
niques, are acknowledged problems [4] with thousands of vulnerabilities
reported each year [5]. Furthermore, surveys as published by the National
Institute of Standards and Technology [6] show high cost of insecure
Security Testing: A Survey 3

software due to inadequate testing even on an economic level. Therefore,

support for security testing, which is still often considered as a “black
art,” is essential to increase its effectiveness and efficiency in practice. This
chapter intends to contribute to the growing need for information on secu-
rity testing techniques by providing an overview of actual security testing
techniques. This is of high value both for researchers to evaluate and refine
existing techniques and practitioners to apply and disseminate them. In this
chapter, security testing techniques are classified (and also the discussion
thereof ) according to their test basis within the secure software development
life cycle into four different types: (1) model-based security testing is grounded
on requirements and design models created during the analysis and design
phase, (2) code-based testing and static analysis on source and byte code created
during development, (3) penetration testing and dynamic analysis on running
systems, either in a test or production environment, as well as (4) security
regression testing performed during maintenance.
This chapter provides a comprehensive survey on security testing and is
structured as follows. Section 2 provides an overview of the underlying con-
cepts on software testing. Section 3 discusses the basic concepts of security
engineering and the secure software development life cycle. Section 4 pro-
vides an overview of security testing and its integration in the secure soft-
ware development life cycle. Section 5 discusses the security testing
techniques model-based security testing, code-based testing and static anal-
ysis, penetration testing, and dynamic analysis as well as security regression
testing in detail. Section 6 discusses the application of security testing tech-
niques to three tiered business applications. Finally, Section 7 summarizes
this chapter.

2. SOFTWARE TESTING
According to the classic definition in software engineering [7], software
testing consists of the dynamic verification that a program provides expected
behaviors on a finite set of test cases, a so called test suite, suitably selected from
the usually infinite execution domain. This dynamic notion of testing, so
called dynamic testing, evaluates software by observing its execution [8].
The executed system is called system under test (SUT). More general notions
of testing [9] consist of all life cycle activities, both static and dynamic,
concerned with evaluation of software products and related artifacts to
determine that they satisfy specified requirements, to demonstrate that
they are fit for purpose and to detect defects. This definition also takes
static testing into account, which checks software development artifact
4 Michael Felderer et al.

(eg, requirements, design, or code) without execution of these artifacts.

The most prominent static testing approaches are (manual) reviews and
(automated) static analysis, which are often combined with dynamic testing,
especially in the context of security. For security testing, the general notion
of testing comprising static and dynamic testing is therefore frequently
applied [10–12], and thus also in this chapter testing comprises static and
dynamic testing.
After running a test case, the observed and intended behaviors of a SUT
are compared with each other, which then results in a verdict. Verdicts can be
either of pass (behaviors conform), fail (behaviors do not conform), and
inconclusive (not known whether behaviors conform) [13]. A test oracle is a
mechanism for determining the verdict. The observed behavior may be
checked against user or customer needs (commonly referred to as testing
for validation), against a specification (testing for verification), A failure is an
undesired behavior. Failures are typically observed (by resulting in verdict
fail) during the execution of the system being tested. A fault is the cause
of the failure. It is a static defect in the software, usually caused by human
error in the specification, design, or coding process. During testing, it is
the execution of faults in the software that causes failures. Differing from
active execution of test cases, passive testing only monitors running systems
without interaction.
Testing can be classified utilizing the three dimensions objective, scope,
and accessibility [14, 15] shown in Fig. 1.
Test objectives are reason or purpose for designing and executing a test.
The reason is either to check the functional behavior of the system or its
nonfunctional properties. Functional testing is concerned with assessing the
functional behavior of an SUT, whereas nonfunctional testing aims at assessing
nonfunctional requirements with regard to quality characteristics like secu-
rity, safety, reliability or performance.
The test scope describes the granularity of the SUT and can be classified
into component, integration, and system testing. It also determines the test
basis, ie, the artifacts to derive test cases. Component testing (also referred to as
unit testing) checks the smallest testable component (eg, a class in an object-
oriented implementation or a single electronic control unit) in isolation.
Integration testing combines components with each other and tests those as
a subsystem, that is, not yet a complete system. System testing checks the
complete system, including all subsystems. A specific type of system testing
is acceptance testing where it is checked whether a solution works for the
user of a system. Regression testing is a selective retesting to verify that
Security Testing: A Survey 5

Objective
Nonfunctional Scope

System

Integration
Functional

Component

Accessibility
White-box Black-box
Figure 1 Testing dimensions objective, scope, and accessibility.

modifications have not caused side effects and that the SUT still complies
with the specified requirements [16].
In terms of accessibility of test design artifacts, we can classifying testing
methods into white- and black-box testing. In white-box testing, test cases
are derived based on information about how the software has been designed
or coded [7]. In black-box testing, test cases rely only on the input/output
behavior of the software. This classification is especially relevant for security
testing, as black-box testing, where no or only basic information about the
system under test is provided, enables to mimic external attacks from
hackers. In classical software testing, a related classification of test design
techniques [17] distinguishes between structure-based testing techniques (ie,
deriving test cases from internal descriptions like implementation code),
specification-based testing techniques (ie, deriving test cases from external
descriptions of software like specifications), and experience-based testing tech-
niques (ie, deriving test cases based on knowledge, skills, and background
of testers).
The process of testing comprises the core activities test planning, design,
implementation, execution, and evaluation [9]. According to Refs. [18] and
[9], test planning is the activity of establishing or updating a test plan. A test
plan includes the test objectives, test scope, and test methods as well as the
resources, and schedule of intended test activities. It identifies, amongst
6 Michael Felderer et al.

others, features to be tested and exit criteria defining conditions for when to
stop testing. Coverage criteria aligned with the tested feature types and the
applied test design techniques are typical exit criteria. Once the test plan has
been established, test control begins. It is an ongoing activity in which the
actual progress is compared against the plan which often results in concrete
measures. During the test design phase the general testing objectives defined
in the test plan are transformed into tangible test conditions and abstract test
cases. For test derivation, specific test design techniques can be applied,
which can according to ISO/IEC/IEEE 29119 [17] be classified into
specification-based, structure-based, and experience-based techniques. Test
implementation comprises tasks to make the abstract test cases executable. This
includes tasks like preparing test harnesses and test data, providing logging
support or writing test scripts which are necessary to enable the automated
execution of test cases. In the test execution phase, the test cases are then exe-
cuted and all relevant details of the execution are logged and monitored.
In manual test execution, testing is guided by a human, and in automated
testing by a specialized application. Finally, in the test evaluation phase
the exit criteria are evaluated and the logged test results are summarized
in a test report.
In model-based testing (MBT), manually selected algorithms automatically
and systematically generate test cases from a set of models of the system under
test or its environment [19]. Whereas test automation replaces manual test
execution with automated test scripts, MBT replaces manual test designs
with automated test designs and test generation.

3. SECURITY ENGINEERING
In this section, we cover basic concepts of security engineering as well
as an overview of the secure software development life cycle.

3.1 Basic Concepts

Security testing validates software system requirements related to security
properties of assets that include confidentiality, integrity, availability,
authentication, authorization, and nonrepudiation. These security proper-
ties can be defined as follows [20]:
• Confidentiality is the assurance that information is not disclosed to
unauthorized individuals, processes, or devices.
• Integrity is provided when data is unchanged from its source and has not
been accidentally or maliciously modified, altered, or destroyed.
Security Testing: A Survey 7

• Availability guarantees timely, reliable access to data and information ser-

vices for authorized users.
• Authentication is a security measure designed to establish the validity of
a transmission, message, or originator, or a means of verifying an indi-
vidual’s authorization to receive specific categories of information.
• Authorization provides access privileges granted to a user, program, or
process.
• Nonrepudiation is the assurance that none of the partners taking part in a
transaction can later deny of having participated.
Security requirements can be formulated as positive requirements, explicitly
defining the expected security functionality of a security mechanism, or as
negative requirements, specifying what the application should not do [10].
For instance, for the security property authorization as positive require-
ments could be “User accounts are disabled after three unsuccessful login
attempts,” whereas a negative requirement could be formulated as “The
application should not be compromised or misused for unauthorized
financial transactions by a malicious user.” The positive, functional view
on security requirements is aligned with the software quality standard
ISO/IEC 9126 [2] defining security as functional quality characteristic.
The negative, nonfunctional view is supported by the ISO/IEC 25010
[3] standard that revises ISO/IEC 9126 and introduces Security as a
new quality characteristic which is not included in the characteristic Func-
tionality any more.
An asset is a data item, or a system component that has to be protected. In
the security context, such an asset has one or multiple security properties
assigned that have to hold for that asset.
A fault is a textual representation of what goes wrong in a behavioral
description. It is the incorrect part of a behavioral description that needs
to be replaced to get a correct description. Since faults can occur in dead
code—code that is never executed—and because faults can be masked by
further faults, a fault does not necessarily lead to an error. At the other side,
an error is always produced by a fault. A fault is not necessarily related to
security properties but is the cause of errors and failures in general.
A vulnerability is a special type of fault. If the fault is related to security
properties, it is called a vulnerability. A vulnerability is always related to
one or more assets and their corresponding security properties. An exploi-
tation of a vulnerability attacks an asset by violating the associated security
property. Since vulnerabilities are always associated with the protection of an
asset, the security relevant fault is usually correlated with a mechanism that
8 Michael Felderer et al.

protects the asset. A vulnerability either means that (1) the responsible secu-
rity mechanism is completely missing, or (2) the security mechanism is in
place but is implemented in a faulty way.
An exploit is a concrete malicious input that makes use of the vulnerability
in the system under test (SUT) and violates the property of an asset. Vulner-
abilities can often be exploited in different ways. One concrete exploit
selects a specific asset and a specific property, and makes use of the vulner-
ability to violate the property for the selected asset.
A threat is the potential cause of an unwanted incident that harms or
reduces the value of an asset. For instance, a threat may be a hacker, power
outages, or malicious insiders. An attack is defined by the steps a malicious or
inadvertently incorrectly behaving entity performs to the end of turning a
threat into an actual corruption of an asset’s properties. This is usually done
by exploiting a vulnerability.
Security aspects can be considered on the network, operating system, and
application level. Each level has its own security threats and corresponding
security requirements to deal with them. Typical threats on the network
level are distributed denial-of-service or network intrusion. On the operat-
ing system level, all types of malware cause threats. Finally, on the applica-
tion level threats typical threats are related to access control or are application
type specific like Cross-Site Scripting in case of web applications. All levels
of security can be subject to tests.
Security testing simulates attacks and employs other kinds of penetration
testing attempting to compromise the security of a system by playing the role
of a hacker trying to attack the system and exploit its vulnerabilities [21].
Security testing requires specific expertise which makes it difficult and hard
to automate [22]. By identifying risks in the system and creating tests driven
by those risks, security vulnerability testing can focus on parts of a system
implementation in which an attack is likely to succeed.
Risks are often used as a guiding factor to define security test processes.
For instance, Potter and McGraw [22] consider the process steps creating
security misuse cases, listing normative security requirements, performing
architectural risk analysis, building risk-based security test plans, wielding
static analysis tools, performing security tests, performing penetration testing
in the final environment, and cleaning up after security breaches. Also the
Open Source Security Testing Methodology Manual (OSSTMM) [23]
and the OWASP Testing Guide [10] take risks into account for their pro-
posed security testing activities.
Security Testing: A Survey 9

3.2 Secure Software Development Life Cycle

Testing is often started very late in the software development life cycle
shortly before it is deployed. It has turned out that this is a very ineffective
and inefficient practice. One of the best methods to prevent security bugs
from appearing in production applications is to improve the software devel-
opment life cycle by including security in each of its phases, thereby exten-
ding it to a secure software development life cycle. A software development life
cycle is a series of steps, or phases, that provide a model for the development
and life cycle management of an application or piece of software. It is a struc-
ture imposed on the development of software artifacts. A generic software
development life cycle model considering testing as an orthogonal dimen-
sion comprises the phases analysis, design, development, deployment, and
maintenance [10]. Each phase delivers specific artifacts, ie, the analysis phase
results in requirements, design provides design models, development
delivers code, deployment results in a running system, and finally all artifacts
are maintained.
A secure software development life cycle takes security aspects into
account in each phase of software development. A crucial concept within
the secure software development life cycle is risk. A risk is the likelihood
of an unwanted incident and its consequence for a specific asset [24]. Taking
into account the negative nature of many security requirements, the concept
of risk can be employed to direct the selection or application of security
counter-measures like testing [22, 24]. In all phases of the secure software
development process, but especially at the design level [25], risk analyses
provide effective means to guide security testing and thus detect faults
and vulnerabilities.
Major security development processes are the Security Development
Life cycle (SDL) [26] from Microsoft and the Open Software Assurance
Maturity Model (OpenSAMM) [27] from OWASP.
Microsofts SDL is an established security life cycle for software develop-
ment projects pursuing the following major principles [26]:
• Secure by Design: Security is a built-in quality attribute affecting the
whole software life cycle.
• Security by Default: Software systems are constructed in a way that
potential harm caused by attackers is minimized, eg, software is deployed
with least necessary privilege.
• Secure in Deployment: software deployment is accompanied by tools
and guidance supporting users and/or administrators.
10 Michael Felderer et al.

• Communications: software developers are prepared for occurring threats

communicating openly and timely with users and/or administrators.
The SDL is composed of security practices attached with the major activities
of a software life cycle, ie, requirements, design, implementation, veri-
fication, and deployment in case of SDL, which are extended by the two
activities training and response. For instance, the security practice
“establish security requirements” is attached to requirements analysis,
“use threat modeling” to design, “perform static analysis” to implementa-
tion, “perform fuzz testing” to verification, and “certify release and archive”
to release.
Similar to the SDL, OpenSAMM attaches security practices to core
activities, ie, governance, construction, verification, and deployment in case
of OpenSAMM, within the software development life cycle. For instance,
verification includes the security practices design review, code review, as
well as (dynamic) security testing.
In particular, OpenSAMM attaches each security practice with three
maturity levels and a starting point of zero:
• Level 0: Implicit starting point representing the activities in the practice
being unfulfilled
• Level 1: Initial understanding and ad-hoc provision of security practice
• Level 2: Increase efficiency and/or effectiveness of the security practice
• Level 3: Comprehensive mastery of the security practice at scale
For each security practice and maturity level, OpenSAMM does not only
define the objectives and activities, but also gives support to achieve this par-
ticular level. This comprises assessment questions, success metrics, costs, and
personnel needed to achieve the targeted maturity level.

4. SECURITY TESTING
In this section, we cover basic concepts of security testing and the inte-
gration of security testing in the secure software development life cycle.

4.1 Basic Concepts

Security testing is testing of security requirements related to security prop-
erties like confidentiality, integrity, availability, authentication, authoriza-
tion, and nonrepudiation.
Security testing identifies whether the specified or intended security
properties are, for a given set of assets of interests, correctly implemented.
This can be done by trying to show conformance with the security
Security Testing: A Survey 11

properties, similar to requirements-based testing; or by trying to address

known vulnerabilities, which is similar to traditional fault-based, or destruc-
tive, testing. Intuitively, conformance testing considers well defined,
expected inputs. It tests if the system satisfies the security properties with
respect to these well-defined expected inputs. In contrast, addressing known
vulnerabilities means using malicious, nonexpected input data that is likely
to exploit the considered vulnerabilities.
As mentioned in the previous section, security requirements can be pos-
itive and functional, explicitly defining the expected security functionality of
a security mechanism, or negative and nonfunctional, specifying what the
application should not do. This classification of security requirements also
impacts security testing. For positive security requirements, classical testing tech-
niques can be applied, whereas for negative security requirements (a combination
of ) additional measures like risk analyses, penetration testing, or vulnerability
knowledge bases are essential. This classification is also reflected in by clas-
sification in the literature as provided by Tian-yang et al. [11] as well as by
Potter and McGraw [28].
According to Tian-yang et al. [11] two principal approaches can be dis-
tinguished, ie, security functional testing and security vulnerability testing. Security
functional testing validates whether the specified security requirements are
implemented correctly, both in terms of security properties and security
mechanisms. Security vulnerability testing addresses the identification of
unintended system vulnerabilities. Security vulnerability testing uses the
simulation of attacks and other kinds of penetration testing attempting to com-
promise the security of a system by playing the role of a hacker trying to
attack the system and exploit its vulnerabilities [21]. Security vulnerability
testing requires specific expertise which makes it difficult and hard to auto-
mate [28]. By identifying risks in the system and creating tests driven by
those risks, security vulnerability testing can focus on parts of a system imple-
mentation in which an attack is likely to succeed.
Potter and McGraw [28] distinguish between testing security mechanisms to
ensure that their functionality is properly implemented, and performing risk-
based security testing motivated by understanding and simulating the attacker’s
approach. Testing security mechanisms can be performed by standard test
organizations with classical functional test techniques, whereas risk-based
security testing requires specific expertise and sophisticated analysis [28].
For security vulnerability testing approaches, Shahriar and Zulkernine
[29] propose seven comparison criteria, ie, vulnerability coverage, source of test
cases, test case generation method, testing level, test case granularity, tool automation
12 Michael Felderer et al.

as well as target applications. Tool automation is further refined into the

criteria test case generation, oracle generation, and test case execution. The authors
classify 20 informally collected approaches according to these criteria. The
main aim of the criteria is support for security practitioners to select an
appropriate approach for their needs. Therefore Shahriar and Zulkernine
blend abstract criteria like source of test cases or test case generation method
with technological criteria like tool automation or target applications.
Differing from classical software testing, where black- and white-box test
design are nowadays considered very similar, ie, in both cases testing pro-
ceeds from abstract models [8], the distinction is essential for security testing.
White-box testing performs testing based on information about how the
software has been designed or coded, and thus enables testing from an inter-
nal software producer point of view [7]. Black-box testing relies only on the
input/output behavior of the software, and thus enables to mimic external
attacks from hackers. The classification into white- and black-box testing is
also pointed out by Bachmann and Brucker [12], who additionally classify
security testing techniques due to execution (dynamic vs static testing) and
automation (manual vs automated testing).
In addition, due to the critical role of negative security requirements,
classical testing which focuses on testing functional requirements and secu-
rity testing differ. It seems desirable that security testing directly targets secu-
rity properties, as opposed to taking the detour of functional tests of security
mechanisms. As the former kind of (nonfunctional) security properties
describe all executions of a system, testing them is intrinsically hard. Because
testing cannot show the absence of faults, an immediately useful perspective
directly considers the violation of these properties. This has resulted in the
development of specific testing techniques like penetration testing that sim-
ulates attacks to exploit vulnerabilities. Penetration tests are difficult to craft
because tests often do not directly cause observable security exploits, and
because the testers must think like an attacker [28], which requires specific
expertise. During penetration testing, testers build a mental model of secu-
rity properties, security mechanisms, and possible attacks against the system
and its environment. Specifying security test models in an explicit and pro-
cessable way, results in a model-based security testing approach. In such an
approach, security test models provide guidance for the systematic specifi-
cation and documentation of security test objectives and security test cases,
as well as for their automated generation and evaluation.
(Functional) testing normally focuses on the presence of some correct behavior
but not the absence of additional behavior, which is implicitly specified by
Security Testing: A Survey 13

System
Intended-specified as implemented
system functionality

Missing or incorrect Unintended side-effect

functionality: most behvior: most
security mechanisms are here vulnerabilities are here
Figure 2 Most faults in security mechanisms are related to missing or incorrect func-
tionality, most vulnerabilities are related to unintended side-effect behavior (adapted
from Thompson [30]).

negative requirements. Testing routinely misses hidden action and the result
is dangerous side effect behaviors that ship with a software. Fig. 2 illustrates
this side effect nature of most software vulnerabilities that security testing has
to cope with [30].
The circle represents an application’s intended functionality including
security mechanisms, which is usually defined by the requirements specifi-
cation. The amorphous shape superimposed on the circle represents the
application’s actual, implemented functionality. In an ideal system, the
coded application would completely overlap with its specification, but in
practice, this is hardly ever the case. The areas of the circle that the coded
application does not cover represents typical functional faults (ie, behavior
that was implemented incorrectly and does not conform to the specifica-
tion), especially also in security mechanisms. Areas that fall outside of the
circular region represent unindented and potentially dangerous functional-
ity, where most security vulnerabilities lay. The mismatch between specifi-
cation and implementation shown in Fig. 2 leading to faults in security
mechanisms and vulnerabilities can be reduced by taking security and espe-
cially security testing aspects into account early and in all phases of the soft-
ware development life cycle as discussed in Section 4.2.
14 Michael Felderer et al.

4.2 Security Testing in the Secure Software Development

Life cycle
As mentioned before, testing within the security life cycle plays the role to
validate and verify security requirements. Due to the negative nature of
many security requirements and the resulting broad range of subordinate
requirements, also testing activities cover a broad range of scopes and
employed methods. In keeping with research and experience, it is essential
to take testing into account in all phases of the secure software development
life cycle, ie, analysis, design, development, deployment, as well as mainte-
nance. Thus, security testing must be holistic covering the whole secure
software development life cycle [12]. In concrete terms, Fig. 3 shows a rec-
ommended distribution of static and dynamic testing efforts among the
phases of the secure software development life cycle according to Ref.
[10]. It shows that security testing should be balanced over all phases, with
a focus on the early phases, ie, analysis, design, and implementation.
To provide support for the integration of security testing into all phases
of the secure software development process, major security development
processes (see Section 3.2), consider the integration of testing. In the Secu-
rity Development Life cycle (SDL) [26] from Microsoft practices with strong
interference with testing efforts are the following:
• SDL Practice #2 (Requirements): Establish Security and Privacy
Requirements

10–15%
10–35%

10–15% Analysis
Design
Development
Deployment
Maintenance
15–35%
15–35%

Figure 3 Proportion of test effort in secure software development life cycle according
to Ref. [10].
Security Testing: A Survey 15

• SDL Practice #4 (Requirements): Perform Security and Privacy Risk

Assessments
• SDL Practice #5 (Design): Establish Design Requirements
• SDL Practice #7 (Design): Use Threat Modeling
• SDL Practice #10 (Implementation): Perform Static Analysis
• SDL Practice #11 (Verification): Perform Dynamic Analysis
• SDL Practice #12 (Verification): Perform Fuzz Testing
• SDL Practice #13: Conduct Attack Surface Review
• SDL Practice #15: Conduct Final Security Review
In OpenSAMM [27] from OWASP, the verification activity includes the
security practices design review, code review, as well as (dynamic) security
testing.
The OWASP Testing Guide [10] and the OWASP Code Review Guide
[31] provide a detailed overview of the variety of testing activities of web
application security. While the Testing Guide has a focus on black-box test-
ing, the Code Review Guide is a white-box approach focusing on manual
code review. Overall, the Testing Guide distinguishes 91 different testing
activities split into 11 subcategories (ie, information gathering, configuration
and deployment management testing, identity management testing, authen-
tication testing, authorization testing, session management testing, data val-
idation testing, error handling, cryptography, business logic testing, as well as
client side testing). Applying security testing techniques to web applications
is covered in Section 6.
The OWASP testing framework workflow, which is also contained in
the OWASP Testing Guide, contains checks and reviews of respective arti-
facts in all secure software development phases, creation of UML and threat
models in the analysis and design phases, unit and system testing during
development and deployment, penetration testing during deployment, as
well as regression testing during maintenance. Proper security testing
requires a mix of techniques as there is no single testing technique that
can be performed to effectively cover all security testing and their application
within testing activities at unit, integration, and system level. Nevertheless,
many companies adopt only one security testing approach, for instance pen-
etration testing [10].
Fig. 4 abstracts from concrete security testing techniques mentioned
before, and classifies them according to their test basis within the secure soft-
ware development life cycle.
Model-based security testing is grounded on requirements and design
models created during the analysis and design phase. Code-based testing and
16 Michael Felderer et al.

Model-based Code-based testing Penetration testing Security

security testing and static analysis and dynamic analysis regression testing

Analysis Design Development Deployment Maintenance

Requirements Design models Code Running system
Figure 4 Security testing techniques in the secure software development life cycle.

static analysis is based on source and byte code created during development.
Penetration testing and dynamic analysis is based on running systems, either in a
test or production environment. Finally, security regression testing is performed
during maintenance. We also apply this classification to structure the discus-
sion of security testing techniques in the following section.

5. SECURITY TESTING TECHNIQUES

This section discusses the security testing techniques model-based
testing, code-based testing and static analysis, penetration testing and
dynamic analysis as well as regression testing in detail. For each testing tech-
nique, basic concepts as well as current approaches are covered.

5.1 Model-Based Security Testing

In model-based testing (MBT) manually selected algorithms automatically
and systematically generate test cases from a set of models of the system under
test or its environment [19]. MBT is an active area of research [32, 33] and
offers big potential to improve test processes in industry [14, 19, 34]. Its pro-
spective benefits include early and explicit specification and review of system
behavior, better test case documentation, the ability to automatically gener-
ate useful (regression) tests and control test coverage, improved maintenance
of test cases as well as shorter schedules and lower costs [19].
Process. The process of MBT consists of three main steps integrated into
the overall process of test design, execution, and evaluation. (1) A model of
the SUT and/or its environment is built from informal requirements, exis-
ting specification documents, or a SUT. The resulting model of the SUT
dedicated to test generation is often called test model. (2) If they are execut-
able, one execution trace of such a model acts as a test case: input and
expected output for an SUT. Because there are usually infinitely many
and infinitely long execution traces, models can therefore be used to gener-
ate an infinite number of tests. To cut down their number and length, test
selection criteria are applied. These guide the generation of tests. (3) Once the
Security Testing: A Survey 17

test model and the test selection criteria are defined, a set of test cases is gen-
erated from the test model as determined by the chosen test selection criteria.
Test generation is typically performed automatically. The generated test
cases are traces of the model and thus in general at a higher level than the
events or actions of an SUT. Therefore, the generated test cases are further
refined to a more concrete level or adapted to the SUT to support their auto-
mated execution.
Model-based security testing. Model-based security testing (MBST) is an
MBT approach that validates software system requirements related to secu-
rity properties. It combines security properties like confidentiality, integrity,
availability, authentication, authorization, and nonrepudiation with a model
of the SUT and identifies whether the specified or intended security features
hold in the model.
Both MBT and MBST have in common, that the input artifact is a model
and not the SUT. Therefore the abstraction gap between the model and the
SUT has to be addressed. In particular, an identified (security) issue at the
model level does not automatically confirm an (security) issue at the
SUT. Therefore an additional step is needed to map an abstract test case
to an executable test case that can be executed on the SUT.
Selection criteria: “Good” test cases. Arguably, “good” test cases detect
potential, rather than actual, defects with good cost effectiveness [35].
Potential defects need to be described by defect hypotheses. In order to turn
these hypotheses into operational adequacy criteria [36], they need to be
captured by some form of explicit defect model [35, 37, 38]. One form
of defect is a fault, understood as the root cause of an incorrect system state
(error) or incorrect system output (failure). As we show below, vulnerabil-
ities can be understood as faults.
In addition to explicit models of (the functionality of ) the system under
test, model-based security testing usually makes use of one or more of the
three following models for test selection: properties, vulnerabilities, and
attackers. Models of an attacker encode an attacker’s behavior: the data they
need, the different steps they take, the way they craft exploits. Attacker
models can be seen as models of the environment of a system under test,
and knowledge about a targeted vulnerability usually is left implicit. In con-
trast, models of vulnerabilities explicitly encode weaknesses in a system or a
model of the system. In this sense, they can be seen as faults that are used
for the generation of “good” test case generation (see above). Finally, prop-
erties describe desired characteristics of a model, or an implementation, and
they include confidentiality, availability, integrity, and so on. Models of
18 Michael Felderer et al.

properties are needed to describe the properties of an asset that are supposed
not to be violated: they describe what exactly the security tester targets, and
what an exploit is supposed to exploit.
It is noteworthy that all forms of security testing, model-based or not,
always work with an implicit or explicit hypothesis about a potential
vulnerability.
Vulnerabilities as Faults. Frequently, as a reaction to known relevant
threats, assets are protected by explicit security mechanisms. Mechanisms
include input sanitization, Address Space Layout Randomization (ASLR),
encryption of password files, but also intrusion detection systems and access
control components. Mechanisms are components of a system and can
always be syntactically recognized: there is a piece of code (the mechanism)
that is supposed to protect the asset; or there is no such piece of code.
A vulnerability is a special kind of fault with security implications. It is
defined as the absence of a correctly functioning mechanism. This can mean
both (1) that there is no mechanism at all (eg, no input sanitization takes
place which can lead to buffer overflows or SQL Injections) and (2) that
the mechanism does not work correctly, ie, is partially or incorrectly
implemented, for instance, if an access control policy is faulty.
Security testing can then be understood in three seemingly different
ways: (1) to test if specific security properties of an asset can be violated
(properties and property models); (2) to test the functionality of a mechanism
(attacker models); and (3) to directly try to exploit a vulnerability (vulnerabil-
ity models). The boundaries are blurred, however: With the above definition
of vulnerabilities as the absence of effectively working defense mechanisms,
and the observation that attacker models always involve implicit or explicit
hypotheses on vulnerabilities, activities (2) and (3) are close to identical. In
practice, they only differ in terms of the perspective that the tester takes:
the mechanism or the vulnerability. Because the above definition also binds
vulnerabilities to—possibly unspecified—assets, the goal of activities (2) and
(3) always is activity (1). It hence seems hard to provide a crisp conceptual
distinction between the three activities of (1) testing security properties, (2)
testing security mechanisms, and (3) testing for vulnerabilities.
Classification of model-based (security) testing. Several publications have been
published that propose taxonomies and classifications of existing MBT [32,
33] and MBST approaches [39, 40]. We will focus on the classification pro-
posed by Schieferdecker et al. [40] considering different perspectives used in
securing a system. The authors claim that MBST needs to be based on dif-
ferent types of models and distinguish three types of input models for security
Security Testing: A Survey 19

test generation, ie, architectural and functional models, threat, fault and risk models,
as well as weakness and vulnerability models. Architectural and functional models
of the SUT are concerned with security requirements and their implementa-
tion. They focus on the expected system behavior. Threat, fault and
risk models focus on what can go wrong, and concentrate on causes and
consequences of system failures, weaknesses or vulnerabilities. Weakness
and vulnerability models describe weaknesses or vulnerabilities by themselves.
In the following, we exemplary describe selected approaches, that make
use of different models according to the classification of Schieferdecker et al.

5.1.1 A Model-Based Security Testing Approach for Web Applications

An approach that makes use of functional, fault, and vulnerability models
according to Schieferdecker et al. is presented by Büchler et al. [41]. They
published a semiautomatic security testing approach for web applications
from a secure model. The authors assume there is a formal model M for
the specification of the System under Test (SUT). This model is secure as
it does not violate any of the specified security goals (eg, confidentiality
and authenticity). Thus, a model-checker will report M⊨φ for all security
properties φ defining the security goals of the model. The model is built
using abstract messages that are defined by the modeler. These messages rep-
resent common actions a user of the web application can perform. The idea
is that these abstract messages are sent to the server to tell it which actions the
client wants to perform, eg, log in to the web application, view profiles of
different users, delete profiles, update profiles, and so on. Thus, the modeler
does not care about details at the browser/protocol level but only about
abstract messages that represent web application actions.
To make use of such a secure model, Büchler et al. [41] define semantic
mutation operators that represent common, well-known vulnerabilities at
source code level. Semantic mutation operators are an abstraction that these
vulnerabilities so that they can be injected into the model. After having
applied a mutation operator to an original model, the model checker may
provide a trace from this mutated model that violates a security property.
This trace is called an attack trace because it shows which sequence of abstract
messages have to be exchanged in order to lead the system to a state where
the security property is violated. Since abstract attack traces are at the same
level of abstraction as the input model, they need to be instantiated to turn
them operational. The approach proposes a multistep instantiation since web
applications are usually accessed via a browser. In the first step, abstract mes-
sages are translated into abstract browser actions. The second step is a
20 Michael Felderer et al.

mapping from these browser actions to executable API calls to make them
operational in a browser. Finally, a test execution engine executes the
operationalized test cases on the SUT to verify, if the implementation of
the model suffers from the same vulnerability as reported by the model
checker at the abstract level.

5.1.2 A Model-Based Framework for Security Policy Specification,

Deployment, and Testing
Mouelhi et al. [42] propose an approach based on architectural, functional,
and fault models and focus on security policies. They propose a model-based
approach for the specification, deployment, and testing of security policies in
Java applications. The approach starts with a generic security meta-model of
the application. It captures the high level access control policy implemented
by the application and is expressed in a dedicated domain-specific language.
Before such a model is further used, the model is verified to check the
soundness and adequacy of the model with respect to the requirements.
Afterwards the model is automatically transformed to policy decision points
(PDP). Since such PDPs are usually not generated from scratch but are based
on existing frameworks, the output of the transformation is, for instance, an
XACML (Extended Access Control Markup Language) file that captures the
security policy. This transformation step is essential in MBT since an iden-
tified security issue at model level does not automatically imply the same
issue at implementation level, nor does a model without security issues
automatically imply the same on the implementation. Mouelhi et al. make
use of mutations at the model level to ensure that the implementation con-
forms to the initial security model. An existing test suite is executed on an
implementation generated from a mutated security model. If such mutants
are not detected by the existing test suite, it will be adapted to cover the
mutated part of the security model as well. Finally the test objective is to
check that the implementation (security policy) is synchronized with the
security model.

5.1.3 Risk-Based Security Testing

In the following, we consider approaches that are based on risk models.
Risk-based testing in general is a type of software testing that explicitly con-
siders risks of the software system as the guiding factor to solve decision
problems in all phases of the test process, ie, test planning, design, imple-
mentation, execution, and evaluation [1, 43, 44]. It is based on the intuitive
idea to focus testing activities on those areas that trigger the most critical
Security Testing: A Survey 21

situations for a software system [45]. The precise understanding of risks as

well as their focused treatment by risk-based testing has become one of
the cornerstones for critical decisions within complex software development
projects and recently gained much attention [44]. Lately, the international
standard ISO/IEC/IEEE 29119 Software Testing [17] on testing tech-
niques, processes, and documentation even explicitly considers risks as an
integral part of the test planning process. In the following, we describe three
risk-based approaches to security testing in more detail.
Grossmann et al. [46] present an approach called Risk-Based Security
Testing that combines risk analysis and risk-based test design activities based
on formalized security test patterns. The involved security test patterns are
formalized by using a minimal test design strategies language framework
which is represented as a UML profile. Such a (semi-)formal security test
pattern is then used as the input for a test generator accompanied by the test
design model out of which the test cases are generated. The approach is
based on the CORAS method [24] for risk analysis activities. Finally, a tool
prototype is presented which shows how to combine the CORAS-based
risk analysis with pattern-based test generation.
Botella et al. [47] describe an approach to security testing called Risk-
Based Vulnerability Testing, which is guided by risk assessment and cover-
age to perform and automate vulnerability testing for web applications.
Risk-Based Vulnerability testing adapts model-based testing techniques
using a pattern-based approach for the generation of test cases according
to previously identified risks. For risk identification and analysis, the
CORAS method [24] is utilized. The integration of information from risk
analysis activities with the model-based test generation approach is realized
by a test purpose language. It is used to formalize security test patterns in
order to make them usable for test generators. Risk-Based Vulnerability
Testing is applied to security testing of a web application.
Zech et al. [48, 49] propose a new method for generating negative secu-
rity tests for nonfunctional security testing of web applications by logic pro-
gramming and knowledge engineering. Based on a declarative model of the
system under test, a risk analysis is performed and used for derivation of
test cases.

5.2 Code-Based Testing and Static Analysis

Many vulnerabilities can only be detected by looking at the code. While tra-
ditionally not understood as a testing technique, static analysis of the program
code is an important part of any security development process, as it allows to
22 Michael Felderer et al.

detect vulnerabilities at early stages of the development life cycle where fix-
ing of vulnerabilities is comparatively cheap [50]. In Microsoft’s SDL [26],
SAST is part of the implementation phase to highlight that this technique
should be applied as soon as the first line of code is written. Note that in this
section, we only discuss purely static approaches, ie, approaches that do not
require an executable test system. Thus, we discuss hybrid approaches, ie,
approaches that combine static analysis with dynamic testing (such as con-
colic testing) in Section 5.3.
Code reviews can either be done manually or automated. The latter is
often called static code analysis (SCA) or Static Application Security Testing
(SAST). Moreover, we can either analyze the source code (ie, the code that
was written by a developer) of the program or the compiled source code (ie,
binaries or byte-code). As they are closely related, we discuss them not sep-
arately. From a software vendor’s perspective who is aiming at building
secure software, the analysis on the source code is preferred over a binary
analysis, as the source code analysis is more precise and can provide detailed
recommendations to developers on how to fix a vulnerability on the source
code level.

5.2.1 Manual Code Review

Manual code review is the process by which an expert is reading program
code “line-by-line” to identify vulnerabilities. This requires expertise in
three areas: the application architecture, the implementation techniques
(programming languages, frameworks used to build the application), as well
as security. Thus, a good manual code review should start with a threat
model or at least an interview with the developers to get a good under-
standing of the application architecture, its attack surface, as well as the
implementation techniques. After this, the actual code review can start in
which code is, guided by the identified attack surface, manually analyzed
for security vulnerabilities. Finally, the results of the analysis are reported
back to development to fix the identified vulnerabilities as well as to educate
architects and developers to prevent similar issues in the future. Overall,
manual code reviews are a tedious process that requires skill, experience,
persistence, and patience.

5.2.2 Static Application Security Testing

Automated static program analysis for finding security vulnerabilities, also
called Static Application Security Testing (SAST) [51], is an attempt to auto-
mated code reviews: in principle, a SAST tool analyses the program code
Security Testing: A Survey 23

of a software component (eg, an application or a library) automatically and

reports potential security problems (potential vulnerabilities). This limits the
manual effort to reviewing the reported problems and, thus, increases the
scalability (ie, the amount of program code that can be analyzed in a certain
amount of time) significantly. Moreover, on the one hand, SAST tools
“encapsulate” most of the required security expertise and, thus, they can
(and should) be used by developers that are not necessarily security experts.
On the other hand, SAST tools only report vulnerabilities they are looking
for and, thus, there is still a need for a small team of experts that configures
the SAST tools correctly [52, 53].
For computing the set of potential security problems in a program, a
SAST tool mainly employs two different types of analysis:
1. Syntactic checks such as calling insecure API functions or using insecure
configuration options. An example of this class would be an analysis
of Java programs for calls to java.util.random (which does not provide
a cryptographically secure random generator).
2. Semantic checks that require an understanding of the program semantics
such as the data flow or control flow of a program. An example of this
class would be an analysis checking for direct (not sanitized) data-flows
from an program input to a SQL statement (indicating a potential SQL
Injection vulnerability).
As SAST tools work on overapproximations of the actual program code as
well as apply heuristics checks, the output of a SAST tool is a list of potential
security vulnerabilities. Thus, or each finding, an human expert is necessary
to decide:
• If the finding represents a vulnerability, ie, a weakness that can be
exploited by an attacker (true positive), and, thus, needs to be fixed.
• If the finding cannot be exploited by an attacker ( false positive) and, thus,
does not need to be fixed.
Similarly, if an SAST tool does not report security issues, this can have two
reasons:
• The source code is secure (true negative)
• The source code has security vulnerability but due to limitations of the
tool, the tool does not report a problem ( false negative).
There are SAST tools available for most of the widely used programming
language, eg, FindBugs [54] that is able to analyzes Java byte code and, thus,
can analyze various languages running on the Java Virtual Machine. There
are also specializes techniques for Java programs (eg, [55]), or C/C++ (eg,
[56]) as well as approaches that work on multiple languages (eg, [57]). For a
24 Michael Felderer et al.

survey on static analysis methods, we refer the reader elsewhere [51, 58].
Moreover, we discuss further static analysis techniques in the context of a
small case study in Section 6.
Besides the fact that SAST tools can be applied very early in the software
development life cycle as well as the fact that source code analysis can pro-
vide detailed fix recommendations, SAST has one additional advantages
over most dynamic security testing techniques: SAST tools can analyze all
control flows of a program. Therefore, SAST tools achieve, compared to
dynamic test approaches, a significant higher coverage of the program under
test and, thus, produce a significant lower false negative rate. Thus, SAST is a
very effective method [59] for detecting programming related vulnerabilities
early in the software development life cycle.

5.3 Penetration Testing and Dynamic Analysis

In contrast to white-box security testing techniques (see Section 5.2), black-
box security testing does not require access to the source code or other
development artifacts of the application under test. Instead, the security test
is conducted via interaction with the running software.

5.3.1 Penetration Testing

A well-known form of black-box security testing is Penetration Testing. In a
penetration test, an application or system is tested from the outside in a setup
that is comparable to an actual attack from a malicious third party. This
means, in most settings the entity that is conducting the test has potentially
only limited information about the system under test and is only able to
interact with the system’s public interfaces. Hence, a mandatory prerequisite
for this approach is a (near) productive application, that is feature complete
and sufficiently filled with data, so that all implemented workflows can be
executed during the test. Penetration tests are commonly done for applica-
tions that are open for networked communication.
The NIST Technical Guide to Information Security Testing and Assess-
ment [60] partitions the penetration testing process in four distinct phases
(see Fig. 5):
1. Planning: No actual testing occurs in this phase. Instead, important side
conditions and boundaries for the test are defined and documented. For
instance, the relevant components of the applications that are subject of
the test are determined and the nature/scope of the to be conducted tests
and their level of invasiveness.
Security Testing: A Survey 25

Additional discovery

Planning Discovery Attack

Reporting

Figure 5 Phases of a penetration test [60].

2. Discovery: This phase consists of a steps. First, all accessible external inter-
faces of the system under test are systematically discovered and enumer-
ated. This set of interfaces constitutes the system’s initial attack surface.
The second part of the discovery phase is vulnerability analysis, in which
the applicable vulnerability classes that match the interface are identified
(eg, Cross-Site Scripting for HTTP services or SQL Injection for appli-
cations with database back end). In a commercial penetration test, this
phase also includes the check if any of the found components is suscep-
tible to publicly documented vulnerabilities which is contained in pre-
compiled vulnerability databases.
3. Attack: Finally, the identified interfaces are tested through a series of
attack attempts. In these attacks, the testers actively attempts to compro-
mise the system via sending attack payloads. In case of success, the found
security vulnerabilities are exploited in order to gain further information
about the system, widen the access privileges of the tester and find further
system components, which might expose additional interfaces. This
expanded attack surface is fed back into the discovery phase, for further
processing.
4. Reporting: The reporting phase occurs simultaneously with the other
three phases of the penetration test and documents all findings along with
their estimated severeness.

5.3.2 Vulnerability Scanning

In general penetration tests are a combination of manual testing through
security experts and the usage of black-box vulnerability scanners. Black-box
web vulnerability scanners are a class of tools that can be used to identify
security issues in applications through various techniques. The scanner
queries the application’s interfaces with a set of predefined attack payloads
26 Michael Felderer et al.

and analyses the application’s responses for indicators if the attack was suc-
cessful and, if this is not the case, hints how to alter the attack in the subse-
quent tries. Bau et al. [4] as well as Adam et al. [61] provide overviews of
recent commercial and academic black-box vulnerability scanners.

5.3.3 Dynamic Taint Analysis

An important variant of black-box testing is an analysis technique called
taint analysis. A significant portion of today’s security vulnerabilities are
string-based code injection vulnerabilities [62], which enable the attacker to inject
syntactic content into dynamically executed programming statements,
which—in the majority of all cases—leads to full compromise of the vulner-
able execution context. Examples for such vulnerabilities include SQL
Injection [63] and Cross-Site Scripting [64]. Such injection vulnerabilities
can be regarded as information flow problems, in which unsanitized data
paths from untrusted sources to security sensitive sinks have to be found.
To achieve this, a well established approach is (dynamic) data tainting.
Untrusted data is outfitted with taint information on runtime, which is
only cleared, if the data passes a dedicated sanitization function. If data
which still carries taint information reaches a security sensitive sink (eg,
an API that converts string data into executable code), the application can
react appropriately, for instance through altering, autosanitization the data
or completely stopping the corresponding process. If taint tracking is uti-
lized in security testing, the main purpose is to notify the tester that insecure
data flows, that likely lead to code injection, exist. Unlike static analysis,
that also targets the identification of problematic data flows, dynamic taint
analysis is conducted transparently while the application under test is exe-
cuted. For this, the execution environment, eg, the language runtime,
has to be made taint aware, so that the attached taint information of the
untrusted data is maintained through the course of program execution, so
that it can reliably be detected when tainted data ends up in security
sensitive sinks.

5.3.4 Fuzzing
Fuzzing or fuzz testing is a dynamic testing technique that is based on the idea
of feeding random data to a program “until it crashes.” It was pioneered in
the late 1980s by Barton Miller at the University of Wisconsin [65]. Since
then, fuzz testing has been proven to be an effective technique for finding
vulnerabilities in software. While the first fuzz testing approaches where
purely based on randomly generated test data (random fuzzing), advances
Security Testing: A Survey 27

in symbolic computation, model-based testing, as well as dynamic test case

generation have lead to more advanced fuzzing techniques such as mutation-
based fuzzing, generation-based fuzzing, or gray-box fuzzing.
Random fuzzing is the simplest and oldest fuzz testing technique: a stream
of random input data is, in a black-box scenario, send to the program under
test. The input data can, eg, be send as command line options, events, or
protocol packets. This type of fuzzing in, in particular, useful for test how
a program reacts on large or invalid input data. While random fuzzing
can find already severe vulnerabilities, modern fuzzers do have a detailed
understanding of the input format that is expected by the program
under test.
Mutation-based fuzzing is one type of fuzzing in which the fuzzer has some
knowledge about the input format of the program under test: based on exis-
ting data samples, a mutation-based fuzzing tools generated new variants
(mutants), based on a heuristics, that it uses for fuzzing. there are a wide
range of mutation-based fuzzing approaches available for different domains.
We refer the interested reader elsewhere for details [66, 67].
Generation-based fuzzing uses a model (of the input data or the vulnera-
bilities) for generating test data from this model or specification. Compared
to pure random-based fuzzing, generation-based fuzzing achieves usually a
higher coverage of the program under test, in particular if the expected input
format is rather complex. Again, for details we refer the interested reader
elsewhere [68, 69].
Advanced fuzzing techniques combine several of the previously mentioned
approaches, eg, use a combination of mutation-based and generation-based
techniques as well as observe the program under test and use these observa-
tions for constructing new test data. This turns fuzzing into a gray-box test-
ing technique that also utilizes symbolic computation that is usually
understood as a technique used for static program analysis. Probably the first
and also most successful application of gray-box fuzzing is SAGE from
Microsoft [70, 71], which combines symbolic execution (a static source code
analysis technique) and dynamic testing. This combination is today known
as concolic testing and inspired several advanced security testing eg, [72, 73],
as well as functional test approaches.

5.4 Security Regression Testing

Due to ever changing surroundings, new business needs, new regulations,
and new technologies, a software system must evolve and be maintained,
28 Michael Felderer et al.

or it becomes progressively less satisfactory [74]. This makes it especially

challenging to keep software systems permanently secure as changes
either in the system itself or in its environment may cause new threats
and vulnerabilities [75]. A combination of regression and security testing
called security regression testing, which ensures that changes made to a
system do not harm its security, are therefore of high significance and
the interest in such approaches has steadily increased [76]. Regression test-
ing techniques ensure that changes made to existing software do not
cause unintended effects on unchanged parts and changed parts of the
software behave as intended [77]. As requirements, design models, code
or even the running system can be changed, regression testing techniques
are orthogonal to the security testing techniques discussed in the previous
sections.
Yoo and Harman [78] classify regression testing techniques into three
categories: test suite minimization, test case prioritization and test case
selection.
Test suite minimization seeks to reduce the size of a test suite by eliminat-
ing redundant test cases from the test suite.
Test case prioritization aims at ordering the execution of test cases in the
regression test suite based on a criterion, for instance, on the basis of history,
coverage, or requirements, which is expected to lead to the early detection
of faults or the maximization of some other desirable properties.
Test case selection deals with the problem of selecting a subset of test
cases that will be used to test changed parts of software. It requires to
select a subset of the tests from the previous version, which are likely to
detect faults, based on different strategies. Most reported regression testing
techniques focus on this regression testing technique [78]. The usual strat-
egy is to focus on the identification of modified parts of the SUT and to
select test cases relevant to them. For instance, the retest-all technique is
one naive type of regression test selection by reexecuting all tests from
the previous version on the new version of the system. It is often used
in industry due to its simple and quick implementation. However, its
capacity in terms of fault detection is limited [79]. Therefore, considerable
amount of work is related to the development of effective and scalable
selective techniques.
In the following, we discuss available security testing approaches
according to the categories minimization, prioritization, and selection.
The selected approaches are based on a systematic classification of security
regression testing approaches by Felderer and Fourneret [76].
Security Testing: A Survey 29

5.4.1 Test Suite Minimization

Test suite minimization seeks to reduce the size of a test suite by eliminating
test cases from the test suite based on a given criterion. Current approaches
on minimization [80–82] address vulnerabilities.
Toth et al. [80] propose an approach that applies automated security test-
ing for detection of vulnerabilities by exploring application faults that may
lead to known malware, such as viruses or worms. The approach considers
only failed tests from the previous version revealing faults for rerun in a new
system version after fault fixing.
He et al. [81] propose an approach for detecting and removing vulner-
abilities for minor releases of web applications. In their approach, only
strong-association links between pages from a previous version, optimized
through iterations, are selected for exploration of web pages that contain
vulnerabilities.
Finally, Garvin et al. [82] propose testing of self-adaptive system for
already known failures. The authors avoid reproducing already known fail-
ures by considering only those tests for execution that exercise the known
failures in previous versions.

5.4.2 Test Case Prioritization

Test case prioritization is concerned with right ordering of test cases that
maximizes desirable properties, such as early fault detection. Also, current
approaches to prioritization [83–85] address only vulnerabilities.
Huang et al. [83] propose a prioritization technique for security regres-
sion testing. Their technique gathers historical records, whose most effective
execution order is determined by a genetic algorithm.
Yu et al. [84] propose fault-based prioritization of test cases, which
directly utilizes the knowledge of their capability to detect faults.
Finally, Viennot et al. [85] propose a mutable record-replay system,
which allows a recorded execution of an application to be replayed with
a modified version of the application. Their execution is prioritized
by defining a so called d-optimal mutable replay based on a cost function
measuring the difference between the original execution and the mutable
replay.

5.4.3 Test Case Selection

Test case selection approaches choose a subset or all test cases to test
changed parts of software. As for classical regression testing [78], also for
security regression testing most approaches fall into this category [76].
30 Michael Felderer et al.

These approaches test both, security mechanisms and vulnerabilities.

Several subset-based approaches [86–90] and retest-all approaches [91–94]
have been proposed.
Felderer et al. [86] provide a UML-based approach for regression testing
of security requirements of type authentication, confidentiality, availability,
authorization, and integrity. Tests, represented as sequence diagrams, are
selected based on test requirements changes. Kassab et al. [87] propose an
approach to improve regression testing based on nonfunctional require-
ments ontologies. Tests are selected based on change and impact analysis
of nonfunctional requirements, such as security, safety, performance, or reli-
ability. Each test linked to a changed or modified requirement is selected for
regression testing. Anisetti et al. [88] propose an approach for providing test
evidence for incremental certification of security requirements of services.
This approach is based on change detection in the service test model, which
will determine if new test cases need to be generated, or existing ones to be
selected for reexecution on the evolved service. Huang et al. [89] address
security regression testing of access control policies. This approach selects
tests if they cover changed elements in the policy. Finally, Hwang et al.
[90] propose three safe coverage-based selection techniques for testing evo-
lution of security policies, each of which includes a sequence of rules to spec-
ify which subjects are permitted or denied to access which resources under
which conditions. The three techniques are based on two coverage criteria,
ie, (1) coverage of changed rules in the policy, and (2) coverage of different
program decisions for the evolved and the original policy.
Vetterling et al. [91] propose an approach for developing secure systems
evaluated under the Common Criteria [95]. In their approach, tests covering
security requirements are created manually and represented as sequence dia-
grams. In case of a change, new tests are written if necessary and then all tests
are executed on the new version. Bruno et al. [92] propose an approach for
testing security requirements of web service releases. The service user can
periodically reexecute the test suite against the service in order to verify
whether the service still exhibits the functional and nonfunctional require-
ments. Kongsli [93] proposes the use of so called misuse stories in agile meth-
odologies (for instance, extreme programming) to express vulnerabilities
related to features and/or functionality of the system in order to be able
to perform security regression testing. The author suggests that test execu-
tion is ideal for agile methodologies and continuous integration. Finally, Qi
et al. [94] propose an approach for regression fault-localization in web appli-
cations. Based on two programs, a reference program and a modified
Security Testing: A Survey 31

program, as well as input failing on the modified program, their approach

uses symbolic execution to automatically synthesize a new input that is very
similar to the failing input and does not fail. On this basis, the potential
cause(s) of failure are found by comparing control flow behavior of the pass-
ing and failing inputs and identifying code fragments where the control flows
diverge.

6. APPLICATION OF SECURITY TESTING TECHNIQUES

In this section, we make a concrete proposal on how to apply the secu-
rity test techniques (and the tools implementing them) to a small case study: a
business application using a three tiered architecture. We focus on security
testing techniques that detect the most common vulnerability types that
were disclosed in the Common Vulnerabilities and Exposures (CVE) index
[5] over the period of the last 15 years (see Fig. 6). This clearly shows that the
vast majority of vulnerabilities, such as XSS, buffer overflows, are still caused
by programming errors. Thus, we focus in this section on security testing
techniques that allow to find these kind of vulnerabilities.
For instance, we exclude techniques for ensuring the security of the
underlying infrastructure such as the network configuration, as, eg, discussed
in Refs. [96, 97] as well as model-based testing techniques (as discussed in
Section 5.1) that are in particular useful for finding logical security flaws.
While a holistic security testing strategy makes use of all available security
testing strategies, we recommend to concentrate efforts first on techniques
for the most common vulnerabilities. Furthermore, we also do not explicitly
discuss retesting after changes of the system under test, which is addressed by
suitable (security) regression testing approaches (as discussed in Section 5.4).

6.1 Selection Criteria for Security Testing Approaches

When selecting a specific security testing method and tool, many aspects
need to be taken into account, eg:
• Attack surface: different security testing methods find different attack and
vulnerability types. Many techniques are complementary so it is advis-
able to use multiple security testing methods together to efficiently
detect a range of vulnerabilities as wide as possible.
• Application type: different security testing methods perform differently
when applied to different application types. For example, a method that
performs well against a mobile application may not be able to perform as
well against three-tier client-server applications.
1200

1000

800

#Cross-site scripting
#Buffer errors
600 #Cross-site request forgery
#SQL injection
#Information leak / disclosure
400

200

0
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Figure 6 Number of entires in the common vulnerabilities and exposures (CVE) index by category.
Security Testing: A Survey 33

• Performance and resource utilization: different tools and methods require dif-
ferent computing power and different manual efforts.
• Costs for licenses, maintenance, and support: to use security testing tools effi-
ciently in a large enterprise, they need to be integrated into, eg, bug
tracking or reporting solutions—often they provide their own server
applications for this. Thus, buying a security testing tool is usually not
a one-time effort—it requires regular maintenance and support.
• Quality of results: different tools that implement the same security testing
technique provide a different quality level (eg, in terms of fix recommen-
dations or false positives rates).
• Supported technologies: security testing tools usually only support a limited
number of technologies (eg, programming languages, interfaces, or build
systems). If these tools support multiple technologies, they do not nec-
essary support all of them with the same quality. For example, a source
analysis tool that supports Java and C might work well for Java but not as
well for C.
In the following, we focus on the first two aspects: the attack surface and the
application type. These two aspects are, from a security perspective the first
ones to consider for selecting the best combinations of security testing
approaches for a specific application type (product). In a subsequent step,
the other factors need to be considered for selecting a specific tool that fits
the needs of the actual development as well as the resource and time
constraints.

6.2 A Three-Tiered Business Application

In this chapter, we use a simple multitiered business application, eg, for
booking business travels, as a running example. This architecture is, on
the first hand, very common for larger applications and, on the other hand,
covers a wide variety of security testing challenges.
Let us assume that we want to plan the security testing activities for busi-
ness application that is separated into three tiers:
• First tier: A front-end that is implemented as rich-client using modern
web development techniques, ie, HTML5 and JavaScript, eg, using
frameworks such as AngularJS or JQuery.
• Second tier: A typical middle-tier implemented in Java (eg, using Java
Servlets hosted in an application server such as Apache Tomcat).
• Third tier: A third-party database that provides persistency for the busi-
ness data that is processed in the second tier.
34 Michael Felderer et al.

w
a

Figure 7 Architecture of example application.

Fig. 7 illustrates this example architecture where the dotted vertical lines
mark the trust boundaries of the application.

6.2.1 The First Tier: Web Applications

Web applications are predominantly affected by code injection vulnerabil-
ities, such as SQL Injection [63] or Cross-Site Scripting (XSS) [64]. This is
shown in Fig. 6: as it can be seen, besides buffer overflows (and similar mem-
ory corruption vulnerabilities), which in general do not occur in Web appli-
cations, injection vulnerabilities represent the vast majority of reported
issues.
In consequence, security testing approaches for Web applications are
primarily concerned with detection of such code injection vulnerabilities.
In the remainder of this section, we utilize the field of Web applications
as a case study in security testing and comprehensively list academic
approaches in this area.

6.2.1.1 Code-Based Testing and Static Analysis

As introduced in Section 5.2, static analysis of source code is a powerful tool
to detect vulnerabilities in source code. Static analysis allows to analyze the
code of a given application without actually executing it. To detect injection
vulnerabilities with such an approach, in most cases the static analysis
attempts to approximate the data flows within the examined source code.
This way, data flows from sources that can be controlled by the attacker
(eg, the incoming HTTP request) into security sensitive sinks (eg, APIs that
create the HTTP response of send SQL to the database).
Different mechanisms and techniques proposed in the program analysis
literature can be applied to Web applications. The actual implementation of
such an analysis can utilize different techniques such as model checking,
Security Testing: A Survey 35

data-flow analysis, or symbolic execution, typically depending on the

desired precision and completeness. Three different properties are relevant
for the analysis phase. First, the static analysis of the source code itself can be
performed on different levels, either only within a given function
(intraprocedural analysis) or the interaction of functions can be analyzed as well
(interprocedural analysis). Second, the execution context can be examined in
detail or neglected, which then typically reduces the precision of the analysis.
A third aspect of static analysis deals with the way how the flow of data or
code is analyzed.
One of the first tools in this area was WebSSARI [98], a code analysis tool
for PHP applications developed by Huang et al. based on a CQual-like type
system [99, 100].
Pixy is a static taint analysis tool presented by Jovanovic et al. for auto-
mated identification of XSS vulnerabilities in PHP web applications
[101, 102]. The tool performs an interprocedural, context-sensitive, and
flow-sensitive data flow analysis to detect vulnerabilities in a given applica-
tion. Combined with a precise alias analysis, the tool is capable of detecting a
variety of vulnerabilities. In a similar paper, Livshits and Lam demonstrated
how a context-sensitive pointer alias analysis together with string analysis
can be realized for Java-based web applications [55].
Xie and Aiken introduced a static analysis algorithm based on so-called
block and function summaries to detect vulnerabilities in PHP applications
[103]. The approach performs the analysis in both an intraprocedural and an
interprocedural way. Dahse and Holz refined this approach and demon-
strated that a fine-grained analysis of built-in PHP features and their inter-
action significantly improves detection accuracy [104, 105].
Wasserman and Su implemented a static code analysis approach to detect
XSS vulnerabilities caused by weak or absent input validation [106]. The
authors combined work on taint-based information flow analysis with string
analysis previously introduced by Minamide [107].
A tool for static analysis of JavaScript code was developed by Saxena
et al. [108]. The tool named Kudzu employs symbolic execution to find
client-side injection flaws in Web applications. Jin et al. employ static
analysis[109] to find XSS vulnerabilities within HTML5-based mobile
applications. Interestingly, they found new ways to conduct XSS attacks
within such apps by abusing the special capabilities of mobile phones.
Further static analysis approaches to detect injection vulnerabilities have
been proposed by Fu et al.[110], Wassermann and Su [111], and Halfond
et al. [112]
36 Michael Felderer et al.

6.2.1.2 Dynamic Analysis and Black-Box Testing

A complementary approach is dynamic analysis, in which a given application
is executed with a particular set of inputs and the runtime behavior of the
application is observed. Many of these approaches employ dynamic taint
tracking and consist of two different components. A detection component
to identify potentially vulnerable data flows and a validation component to
eliminate false positives.
SECUBAT [113] is a general purpose Web vulnerability scanner that also
has XSS detection capabilities. To achieve its goals, SECUBAT employs three
different components: a crawling component, an attack component, and an
analysis component. Whenever the crawling component discovers a suspi-
cious feature, it passes the page under investigation to the attack component,
which then scans this page for web forms. If a form is found, appropriate
payloads are inserted into the fields of the form and submitted to the server.
The response of the server is then interpreted by the analysis component.
SNUCK [114] is another dynamic testing tool. In a first step, SNUCK uses a
legitimate test input to dynamically determine a possible injection and the
corresponding context via XPATH expressions. Based on the determined
context, the tool chooses a set of predefined attack payloads and injects them
into the application.
While SECUBAT and SNUCK focus on server-side security problems,
FLASHOVER [115] focuses on a client-side problem. More specifically, FLASH-
OVER detects client-side reflected XSS in Adobe Flash applets. It does so by
decompiling the source code and statically detecting suspicious situations.
Then it constructs an attack payload and executes the exploit via dynamic
testing.
Recently, Bau et al. [4] and Doupe et al. [61] prepared comprehensive
overviews on commercial and academic black-box vulnerability scanners
and their underlying approaches.

6.2.1.3 Taint Tracking

As discussed previously, most Web-specific security vulnerabilities can pre-
dominantly be regarded as information flow problems, caused by unsanitized
data paths from untrusted sources to security sensitive sinks. Hence, taint
tracking (see Section 5.3.3) is a well-established method to detect injection
vulnerabilities.
Dynamic taint tracking was initially introduced by Perl in 1989 [116] and
since then has been adopted for numerous programming languages and
frameworks [117]. Subsequent works describe finer grained approaches
Security Testing: A Survey 37

towards dynamic taint propagation. These techniques allow the tracking of

untrusted input on the basis of single characters. For instance, Nguyen-
Tuong et al. [118] and Pietraszek and Vanden Berghe [119] proposed fine
grained taint propagation for the PHP runtime to detect various classes of
injection attacks, such as SQL Injection or XSS. Based on dynamic taint
propagation, Su and Wassermann [120] describe an approach that utilizes
specifically crafted grammars to deterministically identify SQL injection
attempts.
The first taint tracking paper that aimed at automatically generating
Cross-Site Scripting attacks was authored by Martin et al. [121]. The pres-
ented mechanism expects a program adhering to the Java Servlet specifica-
tion and a taint-based vulnerability specification as an input, and generates a
valid Cross-Site Scripting or SQL Injection attack payload with the help of
dynamic taint tracking and model checking. While this approach requires a
security analyst to manually write a vulnerability specification, Kieyzun et al.
focus on the fully automatic generation of taint-based vulnerability payloads
[122]. Furthermore, they extend the dynamic tainting to the database.
Hence, as opposed to the first approach, this approach is able to also detect
server-side persistent XSS vulnerabilities.
Besides server-side injection vulnerabilities, Web applications are also
susceptible to injection problems on the client-side, ie, Cross-Site Scripting
problems caused by insecure JavaScript executed in the browser [123].
Through making the browser’s JavaScript engine taint-aware, this vul-
nerability class can be detected during testing: Lekies et al. implemented a
browser-based byte-level taint-tracking engine to detect reflected and
persistent client-side XSS vulnerabilities [124]. By leveraging the taint infor-
mation, their approach is capable of generating a precise XSS payload
matching the injection context. A similar approach was taken by FLAX
[125], which also employs taint tracking in the browser. Instead of using
an exploit generation technique, FLAX utilizes a sink-aware fuzzing tech-
nique, which executes variations of a predefined list of context-specific
attack vectors.

6.2.2 The Second Tier: Java-Based Server Applications

Also the second tier, ie, the Java-based business application, is predominantly
affected by code injection vulnerabilities, such as SQL Injection [63] or
Cross-Site Scripting [64]. Even if we secure the front-end against these
attacks, we cannot rely on these protection as an attacker might circumvent
the front-end by attacking the second tier directly (eg, using WSDL-based,
38 Michael Felderer et al.

RESTful, or OData-based interfaced). Thus, we need to apply the same

standards for secure development also to the second tier.

6.2.2.1 Code-Based Testing and Static Analysis

There are many static analysis approaches available for languages tradition-
ally used for server systems, eg, C/C++ or Java; both in the academic world
as well as commercial offerings. Commercial tools are at least available since
10 years [57, 126] and used widely since at least 5 years [52, 53]. Due to space
reason, we only discuss a few selected works for Java: most notability,
FindBugs [54] is a “ready-to-run” tool that finds a large number of potential
security vulnerabilities in Java programs and, moreover, can be considered
the first widely used static analysis tool for Java that included security checks.
Already in 2005, Livshits presented a method based on taint propagation for
finding data-flow related vulnerabilities in Java programs [55]. Tripp et al.
[127] improved this idea to make it applicable to large scale programs by
using a optimized pointer analysis as prerequisite for building the call graph.
An alternative to using a call-graph-based tainting analysis, there are also
approaches, such as [128] that are based on slicing. Moreover, there are
two different Open Source frameworks available that for building own static
analysis tools for Java: Wala [129], which itself is written in Java, and Swaja
[130], which is written in OCaml.

6.2.2.2 Penetration Testing and Dynamic Analysis

From an implementation perspective, the most important dynamic testing
approach is—besides manual penetration testing—fuzzing. As we imple-
ment the second tier, we focus on gray-box testing approaches that combine
white- and black-box testing. The most well known and industrially proven
approach is SAGE [71], which is used by Microsoft. Besides SAGE, there are
also other approaches that share the same basic concept: using instrumenta-
tion or static analyses of the implementation to improve the quality, effi-
ciency, or effectiveness of dynamic security testing. For example, Bekrar
et al. [72] uses a binary taint analysis to increase the fuzz testing coverage
while [131] uses symbolic execution to achieve the same goal. Gray-box
fuzzing is also successfully applied to commercial operating system [132].
When using a gray-box testing technique on a multitier architecture, one
can decide to only test the second tier in isolation or to test all tiers “at once.”
In the latter case, only selected tiers might be subject to taint tracking or sym-
bolic execution, as already such a partial covering of the implementation
with white-box techniques allows to improve the testing results. Moreover,
Security Testing: A Survey 39

even partial application of white-box techniques helps to diagnose the root

cause of an vulnerability and, thus, usually helps to minimize the time
required for fixing a vulnerability.
Until recently, thus gray-box testing approaches where not generally
available in commercial security testing solutions. This has changed: first
vendors are staring to integrate similar techniques in their tools, using the
Interactive Security Application Testing (ISAT).

6.2.3 The Third Tier: Back-End Systems

In our example, we assume that the back-end system (ie, the database) is sup-
plied by a third-party vendor. Therefore, we only have access to the systems
as black-box. Still, we need to security test it as we are implementing stored
procedures (SQL) on top of it as well as need to configure its security (eg, to
ensure proper access control and a secure and trustworthy communication to
the second tier). Moreover, we might want to assess the overall
implementation-level security of the externally developed product.

6.2.3.1 Security Testing the Implementation

For increasing our confidence in the security of the database, a third-party
component, itself, we can apply manual penetration testing as well as fuzzing
for checking for buffer overflows in the interfaces that are exposed via the
network. As we assume that we only have access to the binary, we can only
use black-box fuzzers such as presented by Woo et al. [133] or gray-box
fuzzers that are based on binary analysis, eg, [134].
Moreover, all code that we develop on top of the core database, ie, stored
procedures written in an SQL-dialect such as PL/SQL should be analyzed
for injection attacks using dedicated static analysis approaches such as Ref.
[135] (also many commercially available static analysis tools support the anal-
ysis of SQL dialects). Moreover, we can apply several dynamic security test-
ing approaches that specialize on testing SQL Injections: Appelt et al. [136]
present a mutation-based testing approach for finding SQL Injections while
Wang et al. [137] use a model-based fuzzing approach. For an experience
report on using fuzzers for finding SQL Injections see Ref. [138].

6.2.3.2 Security Testing the Configuration

Finally, even if all software components are implemented securely, we still
need to ensure that the actual configuration used during operations is also
secure. While we touch this topic only very briefly, we want to emphasize
that it is important to test the secure configuration of the communication
40 Michael Felderer et al.

channels, the access control of all tiers, as well as to keep the systems up to
date by patching known vulnerabilities. Testing the security of the commu-
nication channels includes approaches that check the correct validation of
SSL/TLS certificates (eg, Frankencerts [139]) as well as protocol fuzzers such
as SNOOZE [140] or SECFUZZ [141]. For testing the correct access con-
trol, various model-based approaches (eg, [142–144]) haven been applied to
case studies of different size. Finally, tools like Nessus [145] that rather easily
allow to scan networks for applications with known vulnerabilities and, thus,
applications that need to be updated or patched.

7. SUMMARY
In this chapter, we provided an overview of recent security testing
techniques and their practical application in context of a three-tiered busi-
ness application. For this purpose, we first summarized the required back-
ground on software testing and security engineering. Testing consists of
static and dynamic life cycle activities concerned with evaluation of software
products and related artifacts. It can be performed on the component, inte-
gration, and system level. With regard to accessibility of test design artifacts
white-box testing (ie, deriving test cases based on design and code informa-
tion) as well as black-box testing (ie, relying only on input/output behavior
of software) can be distinguished. Security testing validates software system
requirements related to security properties of assets that include confidenti-
ality, integrity, availability, authentication, authorization, and nonre-
pudiation. Security requirements can be positive and functional, explicitly
defining the expected security functionality of a security mechanism, or
negative and nonfunctional, specifying what the application should not
do. Due to the negative nature of many security requirements and the
resulting broad range of subordinate requirements, it is essential to take test-
ing into account in all phases of the secure software development life cycle
(ie, analysis, design, development, deployment as well as maintenance) and
to combine different security testing techniques.
For a detailed discussion of security testing techniques in this chapter, we
therefore classified them according to their test basis within the secure soft-
ware development life cycle into four different types: (1) model-based security
testing is grounded on requirements and design models created during the
analysis and design phase, (2) code-based testing and static analysis on source
and byte code created during development, (3) penetration testing and dynamic
analysis on running systems, either in a test or production environment, as
Security Testing: A Survey 41

well as (4) security regression testing performed during maintenance. With

regard to model-based security testing, we considered testing based on archi-
tectural and functional models, threat, fault and risk models, as well as weak-
ness and vulnerability models. Concerning, code-based testing and static
analysis we took manual code reviews as well as static application security
testing into account. With regard to penetration testing and dynamic anal-
ysis, we considered penetration testing itself, vulnerability scanning,
dynamic taint analysis, as well as fuzzing. Concerning security regression
testing, we discussed approaches to test suite minimization, test case prior-
itization, and test case selection. To show how the discussed security testing
techniques could be practically applied, we discuss their usage for a three-
tiered business application based on a web client, an application server, as
well as a database back end.
Overall, this chapter provided a broad overview of recent security testing
techniques. It fulfills the growing need for information on security testing
techniques to enable their effective and efficient application. Along these
lines, this chapter is of value both for researchers to evaluate and refine exis-
ting security testing techniques as well as for practitioners to apply and
disseminate them.

ACKNOWLEDGMENTS
The work was supported in part by the research projects QE LaB—Living Models for Open
Systems (FFG 822740) and MOBSTECO (FWF P 26194-N15).

REFERENCES
[1] I. Schieferdecker, J. Grossmann, M. Schneider, Model-based security testing,
in: Proceedings 7th Workshop on Model-Based Testing, 2012.
[2] ISO/IEC, ISO/IEC 9126-1:2001 software engineering—product quality—Part 1:
quality model, 2001.
[3] ISO/IEC, ISO/IEC 25010:2011 systems and software engineering—systems and
software quality requirements and evaluation (SQuaRE)—system and software quality
models 2011.
[4] J. Bau, E. Bursztein, D. Gupta, J. Mitchell, State of the art: automated black-box web
application vulnerability testing, in: 2010 IEEE Symposium on Security and Privacy
(SP), IEEE, 2010, pp. 332–345.
[5] MITRE, Common vulnerabilities and exposures, http://cve.mitre.org.
[6] NIST, The economic impacts of inadequate infrastructure for software testing, 2002
(available at www.nist.gov/director/planning/upload/report02-3.pdf [accessed April
7, 2015]).
[7] P. Bourque, R. Dupuis (Eds.), Guide to the Software Engineering Body of Knowledge
Version 3.0 SWEBOK, IEEE, 2014, http://www.computer.org/web/swebok.
[8] P. Ammann, J. Offutt, Introduction to Software Testing, Cambridge University Press,
Cambridge, UK, 2008.
42 Michael Felderer et al.

[9] ISTQB, Standard glossary of terms used in software testing. Version 2.2, Tech. Rep.,
ISTQB 2012.
[10] OWASP Foundation, OWASP Testing Guide v4, https://www.owasp.org/index.
php/OWASP_Testing_Project (accessed March 11, 2015).
[11] G. Tian-yang, S. Yin-sheng, F. You-yuan, Research on software security testing,
World Acad. Sci. Eng. Technol. 69 (2010) 647–651.
[12] R. Bachmann, A.D. Brucker, Developing secure software: a holistic approach to
security testing, Datenschutz und Datensicherheit (DuD) 38 (4) (2014) 257–261.
[13] ISO/IEC, Information technology—open systems interconnection—conformance
testing methodology and framework, 1994 (international ISO/IEC multi-part
standard No. 9646).
[14] M. Utting, B. Legeard, Practical Model-Based Testing: A Tools Approach, Morgan
Kaufmann Publishers Inc., San Francisco, CA, 2007, ISBN 0123725011.
[15] J. Zander, I. Schieferdecker, P.J. Mosterman, Model-Based Testing for Embedded
Systems, vol. 13, CRC Press, 2012.
[16] IEEE, IEEE standard glossary of software engineering terminology, Institute of
Electronical and Electronics Engineers, Washington, DC, 1990 (IEEE Std 610121990).
[17] ISO, ISO/IEC/IEEE 29119 software testing, 2013 (available at http://www.
softwaretestingstandard.org/ [accessed April 7, 2015]).
[18] IEEE, IEEE standard for software and system test documentation, 2008 (IEEE Std 829-
2008).
[19] I. Schieferdecker, Model-based testing, IEEE Softw. 29 (1) (2012) 14–18.
[20] Committee on National Security Systems, National Information Assurance Glossary,
Tech. Rep. 4009, Committee on National Security Systems, 2010.
[21] B. Arkin, S. Stender, G. McGraw, Software penetration testing, IEEE Secur. Priv. 3 (1)
(2005) 84–87.
[22] B. Potter, G. McGraw, Software security testing, IEEE Secur. Priv. 2 (5) (2004)
81–85.
[23] P. Herzog, The open source security testing methodology manual 3, 2010, http://
www.isecom.org/research/osstmm.html (accessed April 11, 2015).
[24] M.S. Lund, B. Solhaug, K. Stolen, Model-Driven Risk Analysis, Springer, 2011.
[25] D. Verdon, G. McGraw, Risk analysis in software design, IEEE Secur. Priv. 2 (4)
(2004) 79–84.
[26] M. Howard, S. Lipner, The Security Development Lifecycle: SDL: A Process for
Developing Demonstrably More Secure Software, Microsoft Press, 2006, ISBN
0735622140.
[27] OWASP, OpepSAMM, http://www.opensamm.org/ (accessed March 30, 2015).
[28] B. Potter, G. McGraw, Software security testing, IEEE Secur. Priv. 2 (5) (2004) 81–85.
[29] H. Shahriar, M. Zulkernine, Automatic testing of program security vulnerabilities,
in: 33rd Annual IEEE International Computer Software and Applications Conference,
2009, COMPSAC’09, vol. 2, IEEE, 2009, pp. 550–555.
[30] H.H. Thompson, Why security testing is hard, IEEE Secur. Priv. 1 (4) (2003) 83–86.
[31] OWASP Foundation, OWASP Code Review Guide v1.1, https://www.owasp.org/
index.php/Category:OWASP_Code_Review_Project (accessed March 11, 2015).
[32] A.C. Dias-Neto, G.H. Travassos, A picture from the model-based testing area: con-
cepts, techniques, and challenges, Adv. Comput. 80 (2010) 45–120.
[33] M. Utting, A. Pretschner, B. Legeard, A taxonomy of model-based testing approaches,
Softw. Test. Verif. Reliab. 22 (2) (2012) 297312.
[34] W. Grieskamp, N. Kicillof, K. Stobie, V. Braberman, Model-based quality assurance
of protocol documentation: tools and methodology, Softw. Test. Verif. Reliab. 21 (1)
(2011) 55–71.
[35] A. Pretschner, Defect-based testing, in: Dependable Software Systems Engineering, IOS
Press, 2015, http://www.iospress.nl/book/dependable-software-systems-engineering/.
Security Testing: A Survey 43

[36] H. Zhu, P.A.V. Hall, J.H.R. May, Software unit test coverage and adequacy, ACM
Comput. Surv. 29 (4) (1997) 366–427, ISSN 0360-0300.
[37] L.J. Morell, A theory of fault-based testing, IEEE Trans. Softw. Eng. 16 (8) (1990)
844–857, ISSN 0098-5589.
[38] A. Pretschner, D. Holling, R. Eschbach, M. Gemmar, A generic fault model for
quality assurance, in: A. Moreira, B. Schätz, J. Gray, A. Vallecillo, P. Clarke (Eds.),
Model-Driven Engineering Languages and Systems, Lecture Notes in Computer
Science, vol. 8107, Springer, Berlin, ISBN 978-3-642-41532-6, 2013, pp. 87–103.
[39] M. Felderer, B. Agreiter, P. Zech, R. Breu, A classification for model-based security
testing, in: The Third International Conference on Advances in System Testing and
Validation Lifecycle (VALID 2011), 2011, pp. 109–114.
[40] I. Schieferdecker, J. Grossmann, M. Schneider, Model-based security testing,
in: Proceedings 7th Workshop on Model-Based Testing, 2012.
[41] M. Büchler, J. Oudinet, A. Pretschner, Semi-automatic security testing of web
applications from a secure model, in: 2012 IEEE Sixth International Conference on
Software Security and Reliability (SERE), IEEE, 2012, pp. 253–262.
[42] T. Mouelhi, F. Fleurey, B. Baudry, Y. Traon, A model-based framework for security
policy specification, deployment and testing, in: Proceedings of the 11th International
Conference on Model Driven Engineering Languages and Systems, MoDELS ’08,
Toulouse, France, Springer, Berlin, ISBN 978-3-540-87874-2, 2008, pp. 537–552.
[43] P. Gerrard, N. Thompson, Risk-Based e-Business Testing, Artech House Publishers,
2002.
[44] M. Felderer, I. Schieferdecker, A taxonomy of risk-based testing, Int. J. Softw. Tools
Technol. Transf. 16 (5) (2014) 559–568.
[45] M.-F. Wendland, M. Kranz, I. Schieferdecker, A systematic approach to risk-based
testing using risk-annotated requirements models, in: The Seventh International
Conference on Software Engineering Advances, ICSEA 2012, 2012, pp. 636–642.
[46] J. Grossmann, M. Schneider, J. Viehmann, M.-F. Wendland, Combining risk analysis
and security testing, in: Leveraging Applications of Formal Methods, Verification and
Validation. Specialized Techniques and Applications, Springer, 2014, pp. 322–336.
[47] J. Botella, B. Legeard, F. Peureux, A. Vernotte, Risk-based vulnerability testing
using security test patterns, in: Leveraging Applications of Formal Methods, Veri-
fication and Validation. Specialized Techniques and Applications, Springer, 2014,
pp. 337–352.
[48] P. Zech, M. Felderer, B. Katt, R. Breu, Security test generation by answer set
programming, in: 2014 Eighth International Conference on Software Security and
Reliability, IEEE, 2014, pp. 88–97.
[49] P. Zech, M. Felderer, R. Breu, Security risk analysis by logic programming, in: Risk
Assessment and Risk-Driven Testing, Springer, 2014, pp. 38–48.
[50] M.P. Gallaher, B.M. Kropp, The economic impacts of inadequate infrastructure for
software testing, Tech. Rep. Planning Report 02-03, National Institute of
Standards & Technology 2002.
[51] B. Chess, J. West, Secure Programming with Static Analysis, first ed., Addison-Wesley
Professional, Boston, MA, 2007, ISBN 9780321424778.
[52] A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri-Gros,
A. Kamsky, S. McPeak, D. Engler, A few billion lines of code later: using static analysis
to find bugs in the real world, Commun. ACM 53 (2010) 66–75, ISSN 0001-0782.
[53] A.D. Brucker, U. Sodan, Deploying static application security testing on a large scale,
in: S. Katzenbeisser, V. Lotz, E. Weippl (Eds.), GI Sicherheit 2014, Lecture Notes in
Informatics, vol. 228, GI, ISBN 978-3-88579-622-0, 2014, pp. 91–101.
[54] N. Ayewah, D. Hovemeyer, J.D. Morgenthaler, J. Penix, W. Pugh, Experiences using
static analysis to find bugs, IEEE Softw. 25 (2008) 22–29 (special issue on software
development tools, September/October (25:5)).
44 Michael Felderer et al.

[55] V.B. Livshits, M.S. Lam, Finding security errors in java programs with static analysis,
in: Proceedings of the 14th Usenix Security Symposium, 2005, pp. 271–286.
[56] D. Evans, Static detection of dynamic memory errors, SIGPLAN Not. 31 (1996)
44–53, ISSN 0362-1340.
[57] B. Chess, G. McGraw, Static analysis for security, IEEE Secur. Priv. 2 (6) (2004)
76–79, ISSN 1540-7993.
[58] M. Pistoia, S. Chandra, S.J. Fink, E. Yahav, A survey of static analysis methods
for identifying security vulnerabilities in software systems, IBM Syst. J. 46 (2)
(2007) 265–288, ISSN 0018-8670.
[59] R. Scandariato, J. Walden, W. Joosen, Static analysis versus penetration testing: a
controlled experiment, in: Proceedings of the 24th IEEE International Symposium
on Software Reliability Engineering, IEEE, 2013, pp. 1–10.
[60] K. Scarfone, M. Souppaya, A. Cody, A. Orebaugh, Technical guide to information
security testing and assessment, Special Publication 800-115, National Institute of
Standards and Technology (NIST) 2008.
[61] A. Doupe, M. Cova, G. Vigna, Why Johnny can’t pentest: an analysis of black-box
Web vulnerability scanners, in: DIMVA 2010, 2010.
[62] M. Johns, Code injection vulnerabilities in web applications—Exemplified at cross-site
scripting, Ph.D. thesis, University of Passau, 2009.
[63] W.G. Halfond, J. Viegas, A. Orso, A classification of SQL-injection attacks and coun-
termeasures, in: Proceedings of the IEEE International Symposium on Secure Software
Engineering, IEEE, 2006, pp. 65–81.
[64] J. Grossman, R. Hansen, P. Petkov, A. Rager, Cross Site Scripting Attacks: XSS
Exploits and Defense, Syngress, 2007 (Seth Fogie).
[65] B.P. Miller, L. Fredriksen, B. So, An empirical study of the reliability of UNIX
utilities, Commun. ACM 33 (12) (1990) 32–44, ISSN 0001-0782.
[66] S. Rawat, L. Mounier, Offset-aware mutation based fuzzing for buffer overflow
vulnerabilities: few preliminary results, in: Fourth International IEEE Conference
on Software Testing, Verification and Validation, ICST 2012, Berlin, Germany,
21-25 March, 2011 Workshop Proceedings, 2011, pp. 531–533.
[67] F. Duchene, S. Rawat, J. Richier, R. Groz, KameleonFuzz: evolutionary fuzzing for
black-box XSS detection, in: Fourth ACM Conference on Data and Application
Security and Privacy, CODASPY’14, San Antonio, TX, USA, March 03-05, 2014,
2014, pp. 37–48.
[68] D. Yang, Y. Zhang, Q. Liu, BlendFuzz: a model-based framework for fuzz testing
programs with grammatical inputs, in: 11th IEEE International Conference on Trust,
Security and Privacy in Computing and Communications, TrustCom 2012, Liver-
pool, United Kingdom, June 25-27, 2012, 2012, pp. 1070–1076.
[69] J. Zhao, Y. Wen, G. Zhao, H-Fuzzing: a new heuristic method for fuzzing data
generation, in: Proceedings of the 8th IFIP International Conference Network and
Parallel Computing NPC 2011, Changsha, China, October 21-23, 2011, 2011,
pp. 32–43.
[70] P. Godefroid, M.Y. Levin, D.A. Molnar, Automated whitebox fuzz testing,
in: Proceedings of the Network and Distributed System Security Symposium, NDSS
2008, San Diego, California, USA, 10th February-13th February 2008, The Internet
Society, 2008.
[71] P. Godefroid, M.Y. Levin, D.A. Molnar, SAGE: whitebox fuzzing for security testing,
Commun. ACM 55 (3) (2012) 40–44.
[72] S. Bekrar, C. Bekrar, R. Groz, L. Mounier, A taint based approach for smart fuzzing,
in: 2012 IEEE Fifth International Conference on Software Testing, Verification and
Validation, Montreal, QC, Canada, April 17-21, 2012, 2012, pp. 818–825.
Security Testing: A Survey 45

[73] H.J. Abdelnur, R. State, O. Festor, Advanced fuzzing in the VoIP space, J. Comput.
Virol. 6 (1) (2010) 57–64.
[74] M.M. Lehman, Software’s future: managing evolution, IEEE Softw. 15 (1) (1998)
40–44.
[75] M. Felderer, B. Katt, P. Kalb, J. Jürjens, M. Ochoa, F. Paci, L.M.S. Tran, T.T. Tun,
K. Yskout, R. Scandariato, F. Piessens, D. Vanoverberghe, E. Fourneret, M. Gander,
B. Solhaug, R. Breu, Evolution of security engineering artifacts: a state of the art
survey, Int. J. Secur. Softw. Eng. 5 (4) (2014) 48–97.
[76] M. Felderer, E. Fourneret, A systematic classification of security regression testing
approaches, Int. J. Softw. Tools Technol. Transf. (2015) 1–15.
[77] H.K.N. Leung, L. White, Insights into regression testing (software testing),
in: Proceedings Conference on Software Maintenance 1989, IEEE, 1989, pp. 60–69.
[78] S. Yoo, M. Harman, Regression testing minimisation, selection and prioritisation: a
survey, Softw. Test. Verif. Reliab. 1 (1) (2010) 121–141.
[79] E. Fourneret, J. Cantenot, F. Bouquet, B. Legeard, J. Botella, SeTGaM: generalized
technique for regression testing based on UML/OCL models, in: 2014 Eighth
International Conference on Software Security and Reliability (SERE), 2014,
pp. 147–156.
[80] G. Tóth, G. Kőszegi, Z. Hornák, Case study: automated security testing on the
trusted computing platform, in: Proceedings of the 1st European Workshop on System
Security, EUROSEC ’08, Glasgow, Scotland, ACM, ISBN 978-1-60558-119-4,
2008, pp. 35–39.
[81] T. He, X. Jing, L. Kunmei, Z. Ying, Research on strong-association rule based
web application vulnerability detection, in: 2nd IEEE International Conference on
Computer Science and Information Technology, 2009, ICCSIT 2009, 2009,
pp. 237–241.
[82] B.J. Garvin, M.B. Cohen, M.B. Dwyer, Using feature locality: can we leverage history
to avoid failures during reconfiguration? in: Proceedings of the 8th Workshop on
Assurances for Self-adaptive Systems, ASAS ’11, Szeged, Hungary, ACM, ISBN
978-1-4503-0853-3, 2011, pp. 24–33.
[83] Y.-C. Huang, K.-L. Peng, C.-Y. Huang, A history-based cost-cognizant test case
prioritization technique in regression testing, J. Syst. Softw. 85 (3) (2012) 626–637, ISSN
0164-1212, http://www.sciencedirect.com/science/article/pii/S0164121211002780
(novel approaches in the design and implementation of systems/software architecture).
[84] Y.T. Yu, M.F. Lau, Fault-based test suite prioritization for specification-based testing,
Inf. Softw. Technol. 54 (2) (2012) 179–202, http://www.sciencedirect.com/science/
article/pii/S0950584911001947.
[85] N. Viennot, S. Nair, J. Nieh, Transparent mutable replay for multicore debugging
and patch validation, in: Proceedings of the Eighteenth International Conference
on Architectural Support for Programming Languages and Operating Systems,
ASPLOS ’13, Houston, Texas, USA, ACM, 2013, pp. 127–138.
[86] M. Felderer, B. Agreiter, R. Breu, Evolution of security requirements tests for service-
centric systems, in: Engineering Secure Software and Systems: Third International
Symposium, ESSoS 2011, Springer, 2011, pp. 181–194.
[87] M. Kassab, O. Ormandjieva, M. Daneva, Relational-model based change manage-
ment for non-functional requirements: approach and experiment, in: 2011 Fifth
International Conference on Research Challenges in Information Science (RCIS),
2011, pp. 1–9.
[88] M. Anisetti, C.A. Ardagna, E. Damiani, A low-cost security certification scheme for
evolving services, in: 2012 IEEE 19th International Conference on Web Services
(ICWS), 2012, pp. 122–129.
46 Michael Felderer et al.

[89] C. Huang, J. Sun, X. Wang, Y. Si, Selective regression test for access control system
employing RBAC, in: J.H. Park, H.-H. Chen, M. Atiquzzaman, C. Lee, T.-h. Kim,
S.-S. Yeo (Eds.), Advances in Information Security and Assurance, Lecture Notes in
Computer Science, vol. 5576, Springer, Berlin, ISBN 978-3-642-02616-4, 2009,
pp. 70–79.
[90] J. Hwang, T. Xie, D. El Kateb, T. Mouelhi, Y. Le Traon, Selection of regression system
tests for security policy evolution, in: Proceedings of the 27th IEEE/ACM International
Conference on Automated Software Engineering, ACM, 2012, pp. 266–269.
[91] M. Vetterling, G. Wimmel, A. Wisspeintner, Secure systems development based on
the common criteria: the PalME project, in: Proceedings of the 10th ACM SIGSOFT
Symposium on Foundations of Software Engineering, SIGSOFT ’02/FSE-10,
Charleston, South Carolina, USA, ACM, 2002, pp. 129–138.
[92] M. Bruno, G. Canfora, M. Penta, G. Esposito, V. Mazza, Using test cases as contract
to ensure service compliance across releases, in: B. Benatallah, F. Casati, P. Traverso
(Eds.), Service-Oriented Computing—ICSOC 2005, Lecture Notes in Computer
Science, vol. 3826, Springer, Berlin, ISBN 978-3-540-30817-1, 2005, pp. 87–100.
[93] V. Kongsli, Towards agile security in web applications, in: Companion to the 21st ACM
SIGPLAN Symposium on Object-Oriented Programming Systems, Languages, and
Applications, OOPSLA ’06, Portland, Oregon, USA, ACM, New York, NY, ISBN
1-59593-491-X, 2006, pp. 805–808.
[94] D. Qi, A. Roychoudhury, Z. Liang, K. Vaswani, DARWIN: an approach to
debugging evolving programs, ACM Trans. Softw. Eng. Methodol. 21 (3) (2012)
19:1–19:29.
[95] ISO/IEC, ISO/IEC 15408-1:2009 information technology—security techniques—
evaluation criteria for IT security—part 1: introduction and general model, 2009.
[96] N. Bjørner, K. Jayaraman, Checking cloud contracts in Microsoft Azure,
in: Proceedings of the 11th International Conference Distributed Computing and
Internet Technology ICDCIT 2015, Bhubaneswar, India, February 5-8, 2015,
2015, pp. 21–32.
[97] A.D. Brucker, L. Brügger, B. Wolff, Formal Firewall Conformance Testing: An
Application of Test and Proof Techniques, Softw. Test. Verif. Reliab. 25 (1) (2015)
34–71.
[98] Y.-W. Huang, F. Yu, C. Hang, C.-H. Tsai, D.-T. Lee, S.-Y. Kuo, Securing
web application code by static analysis and runtime protection, in: International
Conference on the World Wide Web (WWW), WWW ’04, New York, NY,
USA, ACM, New York, NY, ISBN 1-58113-844-X, 2004, pp. 40–52.
[99] J.S. Foster, M. Fähndrich, A. Aiken, A theory of type qualifiers, SIGPLAN Not. 34 (5)
(1999).
[100] J.S. Foster, T. Terauchi, A. Aiken, Flow-sensitive type qualifiers, SIGPLAN Not.
37 (5) (2002).
[101] N. Jovanovic, C. Kruegel, E. Kirda, Pixy: a static analysis tool for detecting web
application vulnerabilities (short paper), in: IEEE Symposium on Security and Privacy,
SP ’06, IEEE Computer Society, Washington, DC, ISBN 0-7695-2574-1, 2006,
pp. 258–263.
[102] N. Jovanovic, C. Kruegel, E. Kirda, Precise alias analysis for static detection of web
application vulnerabilities, in: Workshop on Programming languages and analysis
for security, PLAS ’06, Ottawa, Ontario, Canada, ACM, New York, NY, ISBN
1-59593-374-3, 2006, pp. 27–36.
[103] Y. Xie, A. Aiken, Static detection of security vulnerabilities in scripting languages,
in: USENIX Security Symposium, vol. 15, 2006, pp. 179–192.
Security Testing: A Survey 47

[104] J. Dahse, T. Holz, Static detection of second-order vulnerabilities in web applications,

in: Proceedings of the 23rd USENIX Security Symposium, 2014.
[105] J. Dahse, T. Holz, Simulation of built-in PHP features for precise static code analysis,
in: ISOC-NDSS, 2014.
[106] G. Wassermann, Z. Su, Static detection of cross-site scripting vulnerabilities, in: ICSE
’08, Leipzig, Germany, ACM, New York, NY, ISBN 978-1-60558-079-1, 2008,
pp. 171–180.
[107] Y. Minamide, Static approximation of dynamically generated web pages,
in: International Conference on the World Wide Web (WWW), 2005.
[108] P. Saxena, D. Akhawe, S. Hanna, F. Mao, S. McCamant, D. Song, A symbolic
execution framework for javaScript, in: IEEE Symposium on Security and Privacy,
SP ’10, IEEE Computer Society, Washington, DC, ISBN 978-0-7695-4035-1,
2010, pp. 513–528.
[109] X. Jin, X. Hu, K. Ying, W. Du, H. Yin, G.N. Peri, Code injection attacks on
HTML5-based mobile apps: characterization, detection and mitigation, in: 21st
ACM Conference on Computer and Communications Security (CCS), 2014.
[110] X. Fu, X. Lu, B. Peltsverger, S. Chen, K. Qian, L. Tao, A static analysis framework
for detecting SQL injection vulnerabilities, in: 31st Annual International Computer
Software and Applications Conference, 2007, COMPSAC 2007, vol. 1, IEEE,
2007, pp. 87–96.
[111] G. Wassermann, Z. Su, Sound and precise analysis of web applications for injection vul-
nerabilities, in: Proceedings of Programming Language Design and Implementation
(PLDI’07), San Diego, CA, 2007.
[112] W.G.J. Halfond, A. Orso, P. Manolios, Using positive tainting and syntax-aware
evaluation to counter SQL injection attacks, in: Proceedings of the 14th ACM
SIGSOFT International Symposium on Foundations of Software Engineering,
ACM, 2006, pp. 175–185.
[113] S. Kals, E. Kirda, C. Kruegel, N. Jovanovic, SecuBat: a web vulnerability scanner,
in: International Conference on the World Wide Web (WWW), WWW ’06,
Edinburgh, Scotland, ACM, New York, NY, ISBN 1-59593-323-9, 2006,
pp. 247–256.
[114] F. d’Amore, M. Gentile, Automatic and context-aware cross-site scripting filter
evasion, Department of Computer, Control, and Management Engineering
Antonio Ruberti Technical Reports 2012.
[115] S.V. Acker, N. Nikiforakis, L. Desmet, W. Joosen, F. Piessens, FlashOver: auto-
mated discovery of cross-site scripting vulnerabilities in rich internet applications,
in: ASIACCS, 2012.
[116] L. Wall, T. Christiansen, J. Orwant, Programming Perl, third ed., O’Reilly,
Sebastopol, CA, 2000.
[117] E.J. Schwartz, T. Avgerinos, D. Brumley, All you ever wanted to know about dynamic
taint analysis and forward symbolic execution (but might have been afraid to ask),
in: IEEE Symposium on Security and Privacy, 2010.
[118] A. Nguyen-Tuong, S. Guarnieri, D. Greene, J. Shirley, D. Evans, Automatically hard-
ening web applications using precise tainting, in: 20th IFIP International Information
Security Conference, 2005.
[119] T. Pietraszek, C.V. Berghe, Defending against injection attacks through context-
sensitive string evaluation, in: Recent Advances in Intrusion Detection (RAID2005),
2005.
[120] Z. Su, G. Wassermann, The essence of command injection attacks in web applications,
in: Proceedings of POPL’06, 2006.
48 Michael Felderer et al.

[121] M. Martin, M.S. Lam, Automatic generation of XSS and SQL injection attacks with
goal-directed model checking, in: USENIX Security Symposium, SEC’08, San Jose,
CA, USENIX Association, Berkeley, CA, 2008, pp. 31–43.
[122] A. Kieyzun, P.J. Guo, K. Jayaraman, M.D. Ernst, Automatic creation of SQL
injection and cross-site scripting attacks, in: ICSE ’09, IEEE Computer Society,
Washington, DC, ISBN 978-1-4244-3453-4, 2009, pp. 199–209.
[123] A. Klein, DOM Based Cross Site Scripting or XSS of the Third Kind, 2005, http://
www.webappsec.org/projects/articles/071105.shtml.
[124] S. Lekies, B. Stock, M. Johns, 25 million flows later-large-scale detection of DOM-
based XSS, in: ACM Conference on Computer and Communications Security
(CCS), 2013.
[125] P. Saxena, S. Hanna, P. Poosankam, D. Song, FLAX: systematic discovery of client-
side validation vulnerabilities in rich web applications, in: ISOC-NDSS, The Internet
Society, 2010.
[126] G. McGraw, Software Security: Building Security In, Addison-Wesley Professional,
2006, ISBN 0321356705.
[127] O. Tripp, M. Pistoia, S.J. Fink, M. Sridharan, O. Weisman, TAJ: effective taint analysis
of web applications, SIGPLAN Not. 44 (2009) 87–97, ISSN 0362-1340.
[128] B. Monate, J. Signoles, Slicing for security of code, in: TRUST, 2008, pp. 133–142.
[129] WALA, T. J. Watson Libraries for Analysis, http://wala.sf.net.
[130] L. Hubert, N. Barre, F. Besson, D. Demange, T.P. Jensen, V. Monfort, D. Pichardie,
T. Turpin, Sawja: static analysis workshop for Java, in: FoVeOOS, 2010, pp. 92–106.
[131] I. Haller, A. Slowinska, M. Neugschwandtner, H. Bos, Dowsing for overflows: a
guided fuzzer to find buffer boundary violations, in: Proceedings of the 22nd USENIX
Conference on Security, SEC’13, Washington, DC, USENIX Association, Berkeley,
CA, ISBN 978-1-931971-03-4, 2013, pp. 49–64.
[132] S.B. Mazzone, M. Pagnozzi, A. Fattori, A. Reina, A. Lanzi, D. Bruschi, Improving
Mac OS X security through gray box fuzzing technique, in: Proceedings of the
Seventh European Workshop on System Security, EuroSec ’14, Amsterdam, The
Netherlands, ACM, New York, NY, ISBN 978-1-4503-2715-2, 2014, pp. 2:1–2:6.
[133] M. Woo, S.K. Cha, S. Gottlieb, D. Brumley, Scheduling black-box mutational fuzzing,
in: Proceedings of the 2013 ACM SIGSAC Conference on Computer & Communica-
tions Security, CCS ’13, Berlin, Germany, ACM, New York, NY, ISBN 978-1-4503-
2477-9, 2013, pp. 511–522.
[134] A. Lanzi, L. Martignoni, M. Monga, R. Paleari, A smart fuzzer for x86 executables,
in: Third International Workshop on Software Engineering for Secure Systems,
2007, SESS ’07: ICSE Workshops 2007, 2007, pp. 7–7.
[135] G. Buehrer, B.W. Weide, P.A.G. Sivilotti, Using parse tree validation to prevent SQL
injection attacks, in: Proceedings of the 5th International Workshop on Software Engi-
neering and Middleware, SEM ’05, Lisbon, Portugal, ACM, New York, NY, ISBN
1-59593-205-4, 2005, pp. 106–113.
[136] D. Appelt, C.D. Nguyen, L.C. Briand, N. Alshahwan, Automated testing for SQL
injection vulnerabilities: an input mutation approach, in: Proceedings of the 2014
International Symposium on Software Testing and Analysis, ISSTA 2014, San Jose,
CA, USA, ACM, New York, NY, ISBN 978-1-4503-2645-2, 2014, pp. 259–269.
[137] J. Wang, P. Zhang, L. Zhang, H. Zhu, X. Ye, A model-based fuzzing approach for DBMS,
in: 2013 8th International Conference on Communications and Networking in China
(CHINACOM), IEEE Computer Society, Los Alamitos, CA, 2013, pp. 426–431.
[138] R. Garcia, Case study: experiences on SQL language fuzz testing, in: Proceedings
of the Second International Workshop on Testing Database Systems, DBTest ’09,
Providence, Rhode Island, ACM, New York, NY, ISBN 978-1-60558-706-6,
2009, pp. 3:1–3:6.
Security Testing: A Survey 49

[139] C. Brubaker, S. Jana, B. Ray, S. Khurshid, V. Shmatikov, Using frankencerts for

automated adversarial testing of certificate validation in SSL/TLS implementations,
in: Proceedings of the 2014 IEEE Symposium on Security and Privacy, SP ’14,
IEEE Computer Society, Washington, DC, ISBN 978-1-4799-4686-0, 2014,
pp. 114–129.
[140] G.Banks, M. Cova, V. Felmetsger, K.C. Almeroth, R.A. Kemmerer, G. Vigna,
SNOOZE: toward a stateful netwOrk prOtocol fuzZEr, in: Proceedings of the 9th
International Conference Information Security ISC 2006, Samos Island, Greece, August
30-September 2, 2006, 2006, pp. 343–358.
[141] P. Tsankov, M.T. Dashti, D.A. Basin, SECFUZZ: fuzz-testing security protocols,
in: 7th International Workshop on Automation of Software Test, AST 2012, Zurich,
Switzerland, June 2-3, 2012, 2012, pp. 1–7.
[142] A. Bertolino, Y.L. Traon, F. Lonetti, E. Marchetti, T. Mouelhi, Coverage-based test
cases selection for XACML policies, in: 2014 IEEE Seventh International Conference
on Software Testing, Verification and Validation, Workshops Proceedings, March 31 -
April 4, 2014, Cleveland, Ohio, USA, 2014, pp. 12–21.
[143] A.D. Brucker, L. Brügger, P. Kearney, B. Wolff, An approach to modular and testable
security models of real-world health-care applications, in: ACM symposium on Access
Control Models and Technologies (SACMAT), Innsbruck, Austria, ACM Press, New
York, NY, ISBN 978-1-4503-0688-1, 2011, pp. 133–142.
[144] E. Martin, Testing and analysis of access control policies, in: 29th International
Conference on Software Engineering—Companion, 2007, ICSE 2007, 2007,
pp. 75–76.
[145] R. Rogers, R. Rogers, Nessus Network Auditing, second ed., Syngress Publishing,
Burlington, MA, 2008, ISBN 9780080558653, 9781597492089.

ABOUT THE AUTHORS

Michael Felderer is a senior researcher
and project manager within the Quality
Engineering research group at the Institute
of Computer Science at the University of
Innsbruck, Austria. He holds a Ph.D. and a
habilitation in computer science. His research
interests include software and security testing,
empirical software and security engineering,
model engineering, risk management, soft-
ware processes, and industry-academia col-
laboration. Michael Felderer has coauthored
more than 70 journal, conference, and work-
shop papers. He works in close cooperation with industry and also transfers
his research results into practice as a consultant and speaker on industrial
conferences.
50 Michael Felderer et al.

Matthias Büchler is a Ph.D. student at the

Technische Universität München. He holds
a master’s degree in computer science
(Information Security) from the Swiss Federal
Institute of Technology Zurich (ETHZ). His
research interests include information secu-
rity, security modeling, security engineering,
security testing, domain specific languages,
and usage control.

Martin Johns is a research expert in the

Product Security Research unit within SAP
SE, where he leads the Web application
security team. Furthermore, he serves on
the board of the German OWASP chapter.
Before joining SAP, Martin studied Mathe-
matics and Computer Science at the Univer-
sities of Hamburg, Santa Cruz (CA), and
Passau. During the 1990s and the early years
of the new millennium he earned his living
as a software engineer in German companies
(including Infoseek Germany, and TC
Trustcenter). He holds a Diploma in Computer Science from University
of Hamburg and a Doctorate from the University of Passau.

Achim D. Brucker is a research expert

(architect), security testing strategist, and pro-
ject lead in the Security Enablement Team of
SAP SE. He received his master’s degree in
computer science from University Freiburg,
Germany and his Ph.D. from ETH Zurich,
Switzerland. He is responsible for the Security
Testing Strategy at SAP. His research interests
include information security, software engi-
neering, security engineering, and formal
methods. In particular, he is interested in tools
and methods for modeling, building, and val-
idating secure and reliable systems. He also participates in the OCL standard-
ization process of the OMG.
Security Testing: A Survey 51

Ruth Breu is head of the Institute of

Computer Science at the University of
Innsbruck, leading the research group Quality
Engineering and the competence center QE
LaB. She has longstanding experience in the
areas of security engineering, requirements
engineering, enterprise architecture manage-
ment and model engineering, both with
academic and industrial background. Ruth
is coauthor of three monographs and more
than 150 scientific publications and serves
the scientific community in a variety of
functions (eg Board Member of FWF, the Austrian Science Fund, Member
of the NIS Platform of the European Commission).

Alexander Pretschner holds the chair of

Software Engineering at Technische Univer-
sität München. Research interests include soft-
ware quality, testing, and information security.
Master’s degrees in computer science from
RWTH Aachen and the University of Kansas
and Ph.D. degree from Technische Universität
München. Prior appointments include a full
professorship at Karlsruhe Institute of Tech-
nology, an adjunct associate professorship at
Kaiserslautern University of Technology, a
group manager’s position at the Fraunhofer
Institute of Experimental Software Engineering in Kaiserslautern, a senior sci-
entist’s position at ETH Zurich, and visiting professorships at the Universities
of Rennes, Trento, and Innsbruck.