SECURING MACHINE

LEARNING ALGORITHMS

DECEMBER 2021
0
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ABOUT ENISA

The European Union Agency for Cybersecurity, ENISA, is the Union’s agency dedicated to
achieving a high common level of cybersecurity across Europe. Established in 2004 and
strengthened by the EU Cybersecurity Act, the European Union Agency for Cybersecurity
contributes to EU cyber policy, enhances the trustworthiness of ICT products, services and
processes with cybersecurity certification schemes, cooperates with Member States and EU
bodies, and helps Europe prepare for the cyber challenges of tomorrow. Through knowledge
sharing, capacity building and awareness raising, the Agency works together with its key
stakeholders to strengthen trust in the connected economy, to boost resilience of the Union’s
infrastructure, and, ultimately, to keep Europe’s society and citizens digitally secure. More
information about ENISA and its work can be found here: www.enisa.europa.eu.

CONTACT
For contacting the authors please use [email protected]
For media enquiries about this paper, please use [email protected]

EDITORS
Apostolos Malatras, Ioannis Agrafiotis, Monika Adamczyk, ENISA

ACKNOWLEDGEMENTS
We would like to thank the Members and Observers of the ENISA ad hoc Working Group on
Artificial Intelligence for their valuable input and feedback.

LEGAL NOTICE
Notice must be taken that this publication represents the views and interpretations of ENISA,
unless stated otherwise. This publication should not be construed to be a legal action of ENISA
or the ENISA bodies unless adopted pursuant to the Regulation (EU) No 2019/881.
This publication does not necessarily represent state-of the-art and ENISA may update it from
time to time.

Third-party sources are quoted as appropriate. ENISA is not responsible for the content of the
external sources including external websites referenced in this publication.

This publication is intended for information purposes only. It must be accessible free of charge.
Neither ENISA nor any person acting on its behalf is responsible for the use that might be made
of the information contained in this publication.

COPYRIGHT NOTICE
© European Union Agency for Cybersecurity (ENISA), 2021
Reproduction is authorised provided the source is acknowledged.

Copyright for the image on the cover: © Shutterstock

For any use or reproduction of photos or other material that is not under the ENISA copyright,
permission must be sought directly from the copyright holders.

ISBN: 978-92-9204-543-2 – DOI: 10.2824/874249 - Catalogue Nr.: TP-06-21-153-EN-N

1
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

TABLE OF CONTENTS

EXECUTIVE SUMMARY 3

1. INTRODUCTION 4
1.1 OBJECTIVES 4

1.2 METHODOLOGY 4

1.3 TARGET AUDIENCE 5

1.4 STRUCTURE 6

2. MACHINE LEARNING ALGORITHMS TAXONOMY 7

2.1 MAIN DOMAIN AND DATA TYPES 8

2.2 LEARNING PARADIGMS 9

2.3 NAVIGATING THE TAXONOMY 10

2.4 EXPLAINABILITY AND ACCURACY 10

2.5 AN OVERVIEW OF AN END-TO-END MACHINE LEARNING LIFECYCLE 11

3. ML THREATS AND VULNERABILITIES 13

3.1 IDENTIFICATION OF THREATS 13

3.2 VULNERABILITIES MAPPED TO THREATS 16

4. SECURITY CONTROLS 18
4.1 SECURITY CONTROLS RESULTS 18

5. CONCLUSION 26

A ANNEX: TAXONOMY OF ALGORITHMS 28

B ANNEX: MAPPING SECURITY CONTROLS TO THREATS 34

C ANNEX: IMPLEMENTING SECURITY CONTROLS 38

D ANNEX: REFERENCES 43

2
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

EXECUTIVE SUMMARY

The vast developments in digital technology influence every aspect of our daily lives. Emerging
technologies, such as Artificial Intelligence (AI), which are in the epicentre of the digital
evolution, have accelerated the digital transformation contributing in social and economic
prosperity. However, the application of emerging technologies and AI in particular, entails perils
that need to be addressed if we are to ensure a secure and trustworthy environment. In this
report, we focus on the most essential element of an AI system, which are machine learning
algorithms. We review related technological developments and security practices to identify
emerging threats, highlight gaps in security controls and recommend pathways to enhance
cybersecurity posture in machine learning systems.

Based on a systematic review of relevant literature on machine learning, we provide a taxonomy This report
for machine learning algorithms, highlighting core functionalities and critical stages. The provides a
taxonomy sheds light on main data types used by algorithms, the type of training these
taxonomy for
algorithms entail (supervised, unsupervised) and how output is shared with users. Particular
emphasis is given to the explainability and accuracy of these algorithms. Next, the report
machine learning
presents a detailed analysis of threats targeting machine learning systems. Identified threats algorithms, a
include inter alia, data poisoning, adversarial attacks and data exfiltration. All threats are detailed analysis
associated to particular functionalities of the taxonomy that they exploit, through detailed tables. of threats and
Finally, we examine mainstream security controls described in widely adopted standards, such
security controls
as ISO 27001 and NIST Cybersecurity framework, to understand how these controls can
effectively detect, deter and mitigate harms from the identified threats. To perform our analysis,
in widely adopted
we map all the controls to the core functionalities of machine learning systems that they protect standards
and to the vulnerabilities that threats exploit in these systems.

Our analysis indicates that the conventional security controls, albeit very effective for
information systems, need to be complemented by security controls tailored to machine learning
functionalities. To identify these machine-learning controls, we conduct a systematic review of
relevant literature, where academia and research institutes propose ways to avoid and mitigate
threats targeting machine learning algorithms. Our report provides an extensive list of security
controls that are applicable only for machine learning systems, such as “include adversarial
examples to training datasets”. For all controls, we map the core functionality of machine
learning algorithms that they intend to protect to the vulnerabilities that threats exploit.

Our findings indicate that there is no unique strategy in applying a specific set of security
controls to protect machine learning algorithms. The overall cybersecurity posture of
organisations who use machine learning algorithms can be enhanced by carefully choosing
controls designed for these algorithms. As these controls are not validated in depth, nor
standardised in how they should be implemented, further research should focus on creating
benchmarks for their effectiveness. We further identified cases where the deployment of
security controls may lead to trade-offs between security and performance. Therefore, the
context in which controls are applied is crucial and next steps should focus on considering
specific use cases and conducting targeted risk assessments to better understand these trade-
offs. Finally, given the complexity of securing machine learning systems, governments and
related institutions have new responsibilities in raising awareness regarding the impact of
threats on machine learning. It is important to educate data scientists on the perils of threats
and on the design of security controls before machine learning algorithms are used in
organisations’ environments. By engaging experts in machine learning in cybersecurity issues,
we may create the opportunity to design innovative security solutions and mitigate the emerging
threats on machine learning systems.

3
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

1. INTRODUCTION

Artificial Intelligence (AI) has grown significantly in recent years and driven by computational
advancements has found wide applicability. By providing new opportunities to solve decision-
making problems intelligently and automatically, AI is being applied to more and more use
cases in a growing number of sectors. The benefits of AI are significant and undeniable.
However, the development of AI is also accompanied by new threats and challenges, which
relevant professionals will have to face.

In 2020, ENISA published a threat landscape report on AI1. This report, published with the
support of the Ad-Hoc Working Group on Artificial Intelligence Cybersecurity2, presents the
Agency’s active mapping of the AI cybersecurity ecosystem and its threat landscape. This threat
landscape not only lays the foundation for upcoming cybersecurity policy initiatives and
technical guidelines, but also stresses relevant challenges.

Machine learning (ML), which can be defined as the ability for machines to learn from data to
solve a task without being explicitly programmed to do so, is currently the most developed and
promising subfield of AI for industrial and government infrastructures. It is also the most
commonly used subfield of AI in our daily lives.

ML algorithms and their specificities, such as the fact that they need large amount of data to
learn, make them the subject of very specific cyber threats that project teams must consider.
The aim of this study is to help project teams identify the specific threats that can target ML
algorithms, associated vulnerabilities, and security controls for addressing these vulnerabilities.

Building on the ENISA AI threat landscape mapping, this study focuses on cybersecurity threats
specific to ML algorithms. Furthermore, vulnerabilities related to the aforementioned threats and
importantly security controls and mitigation measures are proposed.

The adopted description of AI is a deliberate simplification of the state of the art regarding that
vast and complex discipline with the intent of not precisely or comprehensively define it but
rather pragmatically contextualise the specific technique of machine learning.

1.1 OBJECTIVES
The objectives of this publication are:

• To produce a taxonomy of ML techniques and core functionalities to establish a logical

link between threats and security controls.
• To identify the threats targeting ML techniques and the vulnerabilities of ML algorithms,
as well as the relevant security controls and how these are currently being used in the
field to ensure minimisation of security risks.
• To propose recommendations on future steps to enhance cybersecurity in systems that
rely on ML techniques.

1.2 METHODOLOGY
To produce this report, the work was divided into three stages. At the core of the methodology
was an extensive literature review (full list of references may be found in Annex D). The aim

1
https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges
2
See https://www.enisa.europa.eu/topics/iot-and-smart-infrastructures/artificial_intelligence/ad-hoc-working-
group/adhoc_wg_calls

4
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

was to consult documents that are more specific to ML algorithms in general in order to build the
taxonomy, and to consult documents more specific to security to identify threats, vulnerabilities,
and security controls. At the end of the systematic review, more than 200 different documents
(of which a hundred are related to security) on various algorithms of ML had been collected and
analysed.

First, we introduced a high-level ML taxonomy. To understand the vulnerabilities of different ML

algorithms, how they can be threatened and protected, it is crucial to have an overview of their
core functionalities and lifecycle. To do so, a first version of the desk research on ML-focussed
sources was compiled and the ML lifecycle presented in ENISA's work on AI cybersecurity
challenges was consulted3. We then analysed and synthesised all references to produce a first
draft of the taxonomy. The draft was submitted and interviews were held with the ENISA Ad-Hoc
Working Group on Artificial Intelligence Cybersecurity. After considering their feedback, the ML
taxonomy and lifecycle were validated.

The second step was to identify the cybersecurity threats that could target ML algorithms
and potential vulnerabilities. For this task, the threat landscape from ENISA’s report on AI
cybersecurity challenges was the starting point, which was then enriched through desk research
with sources related to the security of ML algorithms. Additionally, the expertise of the ENISA
Ad-Hoc Working Group on Artificial Intelligence Cybersecurity was sought. This work allowed us
to select threats and identify associated vulnerabilities. Subsequently, they were linked to the
previously established ML taxonomy.

The last step of this work was the identification of the security controls addressing the
vulnerabilities. To do this, we utilised the desk research and enriched it with the most relevant
standard security controls from ISO 27001/2 and the NIST 800-53 framework. The output was
reviewed with the experts of the ENISA Ad-Hoc Working Group on Artificial Intelligence
Cybersecurity. This work allowed us to identify security controls that were then linked to the ML
taxonomy.

It is important to note that we opted to enrich the ML-targeted security controls with more
conventional ones to highlight that applications using ML must also comply with more classic
controls in order to be sufficiently protected. Considering measures that are specific to ML
would only give a partial picture of the security work needed on these applications.

1.3 TARGET AUDIENCE

The target audience of this report can be divided into the following categories:

• Public/governmental sector (EU institutions and agencies, Member States’

regulatory bodies, supervisory authorities in the field of data protection, military and
intelligence agencies, law enforcement community, international organisations, and
national cybersecurity authorities): to help them with their risk analysis, identify threats
and understand how to secure ML algorithms.
• Industry (including Small and Medium Enterprises (SMEs)) that makes use of AI
solutions and/or is engaged in cybersecurity, including operators of essential services:
to help them with their risk analysis, identify threats and understand how to secure ML
algorithms.
• AI technical community, AI cybersecurity experts and AI experts (designers,
developers, ML experts, data scientists, etc.) with an interest in developing secure
solutions and in integrating security and privacy by design in their solutions.
• Cybersecurity community: to identify threats and security controls that can apply to
ML algorithms.

3
https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges

5
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

• Academia and research community: to obtain knowledge on the topic of securing

ML algorithms and identify existing work in the field.
• Standardisation bodies: to help identify key aspects to consider regarding securing
ML algorithms.

1.4 STRUCTURE
The report aims to help the target audience to identify the cyber threats to consider and the
security controls to deploy in order to secure their ML applications. Accordingly, the report is
structure into three sections:

• ML algorithms taxonomy: first, a taxonomy to describe the main characteristics of the

algorithms is defined. The different ML algorithms are categorised based on their core
functionalities (e.g., the learning paradigm) and the lifecycle of a ML algorithm is
defined.
• Identification of relevant threats and vulnerabilities: secondly, a list of the
cybersecurity threats and associated vulnerabilities to consider for ML algorithms is
defined. Threats are mapped to the taxonomy to highlight the link between them, the
core functionalities, and the lifecycle of the ML algorithms.
• Security controls: thirdly, a list of security controls for addressing the previously
considered vulnerabilities is given. They are also mapped to the ML taxonomy.

This report focuses on threats that target ML algorithms and on the associated security controls.
It is important to note that this publication examines security controls that are specific to ML
algorithms as well as standard security controls that are also applicable to ML algorithms and
systems making use of them. To use this publication effectively, it is important to note that:

• As is the case for any application, when using ML, one must also consider traditional
security standards (e.g. ISO 27001/2, NIST 800-53), because ML applications are
subject not only to AI/ML specific threats but also to general nature cybersecurity
threats.
• The context of the application (e.g. manipulated data, business case, deployment)
must be considered to correctly assess the risks and prioritise deployment of the
security controls accordingly.

6
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

2. MACHINE LEARNING
ALGORITHMS TAXONOMY

One of the objectives of this work was to devise a (non-exhaustive) taxonomy, to support the
process of identifying which specific threats can target ML algorithms, their associated
vulnerabilities, and security controls for addressing these vulnerabilities. An important disclaimer
needs to be made concerning this taxonomy, namely that it is not meant to be complete or
exhaustive when it comes to ML, instead it aims to support the security analysis of ML
algorithms in this report.

Based on the desk research and interviews with experts of the ENISA AI Working group, we
identified 40 of the most commonly used ML algorithms. A taxonomy was built based on the
analysis of these algorithms. In particular, it was noted that ML algorithms were driven mainly by
the learning paradigms and the problem they address (main domain). These aspects were
therefore chosen to form the key taxonomy dimensions, as seen in Figure 1. It should be noted
that Annex A provides a complete listing of the 40 algorithms and their mapping to the features
of the taxonomy, whereas the Figure serves for illustration purposes.

Figure 1: Machine Learning Algorithm taxonomy

7
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

There is a strong correlation between the domain of application (the problem being addressed)
and the data type which is being worked on, as well as between data environments and learning
paradigm. Thus, further dimensions of the taxonomy were introduced accordingly.

2.1 MAIN DOMAIN AND DATA TYPES

Different algorithms are used in different domains of ML. Therefore, the algorithms have been
categorised according to the main domains represented. Three main domains were (non-
exhaustively) selected, namely Computer Vision, NLP (Natural Language Processing) & Speech
Processing (understanding and generating speech), and Classic Data Science.

The inputs that are given to a ML algorithm are data and therefore, the algorithms can be
categorised based on the types of data that is fed into them. In most cases, specific types of
data are used in certain domains of ML. Indeed, all the algorithms used in computer vision are
fed with images and videos, in the same way that all algorithms used in Natural Language
Processing are fed with text4. In Table 1, the main domains and the type of data used in each of
them are listed.

Table 1: Main domains and data types

Main
Data type Definition
domain

Visual representation of a matrix of pixels constituted of 1 channel for black and

Image white images, 3 elements (RGB) for coloured images or 4 elements (RGBA) for
Computer coloured images with opacity.
Vision
A succession of images (frames), sometimes grouped with a time series (a
Video
sound).

NLP & Text A succession of characters (e.g. a tweet, a text field).

Speech
processing Time series5 A series of data points (e.g. numerical) indexed in time order.

Data organised in a predefined model of array with one specific column for each
feature (e.g. textual, numerical data, date). To be more accurate, structured data
refer to organised data that can be found in a relational data base for example
Classic (that may contain textual columns as mentioned).
Data Structured Data
Science Quantitative data can be distinguished from qualitative data. Quantitative data
corresponds to the numerical data that can supports some arithmetic operations
whereas qualitative data is usually used as categorical data to classify data
according to their similarities.

Certain domains such as NLP and Computer Vision have been separated from Classic Data
Science. The purpose of this separation was to make a distinction between algorithms that may
be used specifically or predominantly for each domain.

4
Audio data are also used for speech recognition. For the purposes of this report, we consider only text for the NLP for the
taxonomy. considering that this will not create differences for the work on threats.
5
For the purposes of this report, time series belong to the two main domains: Classic Data Science and Speech
processing. By restraining Time series to Classic Data Science and Speech processing, we aspired to emphasise the
specific approaches that are used for this domain like ARIMA and Hidden Markov Model. Furthermore, we include audio
data under time series and made the choice to separate video from time series.

8
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

2.2 LEARNING PARADIGMS

Learning paradigm in ML relates to how a machine learns when data is fed to it. For example,
all the classification and regression algorithms use labelled data, meaning that they are doing
only supervised learning. Indeed, supervised learning, by definition, is the learning of labelled
data, which can be either numerical (in this case, the learning paradigm is regression), or
categorical (the learning paradigm is classification). An example of classification can be
differentiating a cat from a dog in a picture, and an example of regression can be predicting the
price of a house. On the other hand, a clustering algorithm uses unlabeled data, which is an
unsupervised type of learning. Therefore, one can conclude that each learning paradigm is a
specific case of one data environment.

In addition to the data types fed into the algorithms, we also focused on three learning
paradigms, namely supervised learning, unsupervised learning, and reinforcement learning:

• Supervised learning learns a function that maps an input to an output based on

example input-output pairs. It infers a function from labelled training data
consisting of a set of training examples.
• Unsupervised learning learns patterns from unlabelled data. It discovers hidden
patterns or data groupings without the need for human intervention.
• Reinforcement learning enables an agent to learn in an interactive environment by
trial and error using feedback from its own actions and experiences.

Table 2: Learning paradigms with typical subtypes.

Learning paradigm Subtypes Definition

Classification is the process of predicting the

Classification class of given data points. (Is the picture a cat
or a dog?)
Supervised learning
Regression models are used to predict a
Regression continuous value. (Predict the price of a house
based on its features).

Clustering is the task of dividing a set of data

points into several groups such that data points
Clustering
in the same groups are more similar each other
than from the data points of the other groups.
Unsupervised learning
Dimensionality reduction refers to techniques
Dimensionality reduction for reducing the number of input variables in
training data.

Rewarding is an area of ML concerned with how

intelligent agents ought to take actions in an
Reinforcement learning Rewarding environment to maximise the notion of
cumulative reward, learning by using feedback
from their experiences.

Each of these learning paradigms have different security-related properties which may lead to
attacks and therefore, it is relevant to represent this information in the taxonomy of ML algorithms,
from which security controls will be mapped. For instance, the most common learning paradigm
is classification and thus, it has many more examples of vulnerabilities due to its popularity.

9
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

2.3 NAVIGATING THE TAXONOMY

Each algorithm is placed in its corresponding cell of the taxonomy grid, according to its learning
paradigm, data type and main domain. For instance, Recurrent Neural Networks 6 (RNN), which
are a type of neural network helpful in modelling sequenced data, are used for regression in
supervised learning, so they must be mapped in the first column. Moreover, the data fed into
them can be text, time series, images, or videos so the RNN box covers all the corresponding
lines in the taxonomy.

However, some of the widely used and mentioned algorithms are based on common elementary
components, or are extensions of the same principle, and can therefore form families or clusters
of algorithms on this taxonomy grid. Hence, we map those specific algorithms in groups by
using nested boxes, as it allows for the representation of a wide variety of algorithms, while
showing that some have relationships with one another.

To continue with the previous example, a more recent version of RNN is LSTM 7 (Long-Short
Term Memory), which differs from RNN based on its optimisation techniques, making it faster to
learn and more precise. Since LSTM is a specific extension of RNN, the LSTM box was nested
in the RNN box in the taxonomy: this indicates that the two algorithms are part of the same
family.

2.4 EXPLAINABILITY AND ACCURACY

An important aspect of security of AI is that of explainability. Understanding the algorithms and
making them explainable makes them more accessible to as many people as possible. It also
helps to increase the trustworthiness of AI and support forensics and analysis of decisions.
Following inputs from the desk research exercise and from the research on attacks targeting ML
models, we additionally included two important parameters in the taxonomy:

• Explainability: For the purposes of this study, algorithms are deemed to be

"explainable" if the decision it makes can be understood by a human. That is to say,
decisions can be understood by a human such as a developer or an auditor and then
explained to an end-user, for example. To be fully explainable, an algorithm must be:
o Globally explainable: a user can identify the features’ importance for the
trained model.
o Locally explainable: a user can explain why the algorithm gives a specific
output (prediction) to a specific input data (features’ values).

• Accuracy (probability score): Some algorithms provide, in addition to a predictive

output, the probability of this prediction which can be interpreted as an “accuracy
level”. If an algorithm doing classification predicts that a picture of a cat is indeed a
picture of a cat at 95% accuracy, one can say that the algorithm has a “high accuracy
classification”. Otherwise, if the prediction was at 55% accuracy, one could say that the
algorithm has a “low accuracy classification”.

It is important to note that we focused on the algorithms’ explainability because this work is
important for other parts of the publication. For example, in one identified security control, it is
highlighted that it is necessary to ensure that ML projects comply with regulatory constraints
such as the GDPR, which describes some explainability requirements8.

6
https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf
7
https://www.bioinf.jku.at/publications/older/2604.pdf
8
GDPR Recital 71 “The data subject should have the right not to be subject to a decision, which may include a measure,
evaluating personal aspects relating to him or her which is based solely on automated processing and which produces legal
effects concerning him or her or similarly significantly affects him or her, such as automatic refusal of an online credit
application or e-recruiting practices without any human intervention. […] In any case, such processing should be subject to
1
Please use footnotes for providing additional or explanatory information and/or relevant links. References should be listed
in a dedicated section. Use only the function References/Insert Footnote

10
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

2.5 AN OVERVIEW OF END-TO-END MACHINE LEARNING LIFECYCLE

An ML system lifecycle includes several interdependent phases ranging from its design and
development (including sub-phases such as requirement analysis, data collection, training,
testing, integration), installation, deployment, operation, maintenance, and disposal. It defines
the phases that an organisation should follow to take advantage of AI and of ML models in
particular to derive practical business value. The latter can be represented as the architecture
illustrated in Figure 229:

Figure 2: Typical AI lifecycle (from the ENISA AI Threat Landscape)

Building on the AI lifecycle, we describe in Figure 3 an overview of a typical ML lifecycle with a

complete overview of the principal steps.

suitable safeguards, which should include specific information to the data subject and the right to obtain human
intervention, to express his or her point of view, to obtain an explanation of the decision reached after such assessment and
to challenge the decision.”
9
https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges
1
Please use footnotes for providing additional or explanatory information and/or relevant links. References should be listed
in a dedicated section. Use only the function References/Insert Footnote

11
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Figure 3: ML Algorithm lifecycle10,11

The aim of the ML algorithm taxonomy is to focus not only on the functionalities of the
algorithms but also on the ML models’ workflow represented by the lifecycle. This lifecycle
summarises the principle steps to produce an ML model. It is important to note that several
steps could have been added, such as data creation and data analysis (for instance, to analyse
if there are some personal data or biases). However, to simplify the lifecycle, some steps have
been condensed. Thus, for example, data cleaning has been included. Regarding data creation,
it was considered as being external to the ML lifecycle.

10
Optimisation is also known as model tuning.
11
Data cleaning and data processing have been separated to distinguish the cleaning phase from the adaptation phase of
the dataset for learning (dimension reduction, feature engineering, etc.).

12
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

3. ML THREATS AND
VULNERABILITIES

3.1 IDENTIFICATION OF THREATS

Based on the methodology described in the Introduction and using a combination of desktop
research and experts’ interviews, we identified a list of six high-level threats and seven sub-threats
that were then mapped to the taxonomy. It is important to note that:

• Threats against supporting infrastructures are not analysed in this publication.

• All threats relate to the previous ENISA publication on the AI Threat Landscape;
accordingly, they have been mapped to AI assets (environments, tools, data, etc.)

The table following summarises Machine Learning threats and includes:

• Threats and sub-threats definitions.

• Whether they are specific to ML algorithms or not.
• At which stage of the life cycle defined in the first section the threat is likely to occur.

13
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Table 3: Threats and sub-threats

Stage of the lifecycle

Model Training
Preprocessing

Model Testing
Data Cleaning
Threats | sub-

Model design

Optimisation
Definition

Deployment
threats

Monitoring
Evaluation
Collection

Model

Model
Data

Data
A type of attack in which the attacker works on
the ML algorithm's inputs to find small
perturbations leading to large modification of its
outputs (e.g. decision errors). It is as if the
attacker created an optical illusion for the
Evasion algorithm. Such modified inputs are often called
x
adversarial examples.
Example: the projection of images on a house
could lead the algorithm of an autonomous car to
take the decision to suddenly make it brake.

In some cases, the attacker has access to

Use of adversarial
information (model, model parameters, etc.) that
examples crafted
can allow him to directly build adversarial
in white or grey x
examples. One example is to directly use the
box conditions
model's gradient to find the best perturbation to
(e.g. FGSM…)
add to the input data to evade the model.

A type of attack in which the attacker explores a

model by providing a series of carefully crafted
inputs and observing outputs. These attacks can
be previous steps to more harmful types,
evasion or poisoning for example.
It is as if the attacker made the model talk to
Oracle then better compromise it or to obtain x
information204 about it (e.g. model extraction) or
its training data (e.g. membership inferences
attacks and Inversion attacks).
Example: an attacker studies the set of input-
output pairs and uses the results to retrieve
training data.

A type of attack in which the attacker altered

data or model to modify the ML algorithm's
behavior in a chosen direction (e.g. to sabotage
its results, to insert a backdoor). It is as if the
attacker conditioned the algorithm according to
Poisoning its motivations. x x x x x x x x
Such attacks are also called causative attacks.
Example: massively indicating to an image
recognition algorithm that images of dogs are
indeed cats to lead it to interpret it this way.

An attack in which the attacker corrupts the

labels of training data.
Label modification x x x x
This sub-threat is specific to Supervised
Learning.

This threat refers to the possibility of leakage of

all or partial information about the model.12
Model or data
Example: the outputs of a ML algorithm are so x x x x x x x x x x
disclosure
verbose that they give information about its
configuration (or leakage of sensitive data)

12
We have chosen to separate the oracle attacks from this threat to address the specifics of both threats and give them both a fair
representation. However, Oracle-type attacks may be considered as a ML specific sub-threat of model or data disclosure.

14
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Stage of the lifecycle

Model Training
Preprocessing

Model Testing
Data Cleaning
Threats | sub-

Model design

Optimisation
Definition

Deployment
threats

Monitoring
Evaluation
Collection

Model

Model
Data

Data
This threat refers to a leak of data manipulated
by ML algorithms. This data leakage can be
explained by an inadequate access control, a
handling error of the project team or simply
because sometimes the entity that owns the
Data disclosure x x x x x x x x x x
model and the entity that owns the data are
distinct. To train the model, it is often necessary
for the data to be accessed by the model
provider. This involve sharing the data and thus
share sensitive data with a third party.

This threat refers to a leak of the internals (i.e.

parameter values) of the ML model. This model
Model disclosure leakage could occur because of human error or x x x x x x x
contraction with a third party with a too low
security level.

This threat refers to the compromise of a

component or developing tool of the ML
Compromise of application.
ML application x x x x x x x x x x
Example: compromise of one of the open-source
components
libraries used by the developers to implement
the ML algorithm.

This threat refers to ML application failure (e.g.

denial of service due to bad input, unavailability
due to a handling error).
Example: the service level of the support
Failure or infrastructure of the ML application hosted by a
malfunction of ML third party is too low compared to the business x x
application needs, the application is regularly unavailable.
Note that this threat does not consider failure of
business use cases (for example, the algorithm
fails because it is not accurate enough to handle
all real-life situations it is exposed to).

The different stakeholders of the model can

make mistakes that result in a failure or
Human error malfunction of ML application. For example, due x x x x x x x x x x
to lack of documentation, they may use the
application in use-cases not initially foreseen.

ML algorithms usually consider input data in a

defined format to make their predictions. Thus, a
denial of service could be caused by input data
Denial of service
whose format is inappropriate. It may also
due to inconsistent
happen that a malicious user of the model x
data or a sponge
constructs an input data (a sponge example)
example
specifically designed to increase the
computation time of the model and thus
potentially cause a denial of service.

Cybersecurity
This threat refers to the possibility that a project
incident not
team may not report security incidents to
reported to x x x x x x x x x x
dedicated teams while a policy of mandatory
incident response
incident reporting has been defined.
teams

15
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

3.2 VULNERABILITIES MAPPED TO THREATS

To identify the security controls, we determined vulnerabilities associated with the threats
described in the previous section. It is important to note that the same vulnerabilities may be found
behind one or more threats (e.g. the “Poor access management” vulnerability). The table below
lists vulnerabilities of ML algorithms and maps them to the aforementioned threats.

Table 4: Threats and associated vulnerabilities

Threats | sub-threats Vulnerabilities

Lack of detection of abnormal inputs

Poor consideration of evasion attacks in the model design implementation

Poor consideration of evasion attacks in the model design implementation

Evasion
Lack of training based on adversarial attacks

Using a widely known model allowing the attacker to study it

Inputs totally controlled by the attacker which allows for input-output-pairs

Use of adversarial examples crafted in Too much information available on the model
white or grey box conditions (e.g.
FGSM…) Too much information about the model given in its outputs

Poor access rights management

The model allows private information to be retrieved

Too much information about the model given in its outputs

Oracle Too much information available on the model

Lack of consideration of attacks to which ML applications could be exposed to

Lack of security process to maintain a good security level of the components of the ML
application
Weak access protection mechanisms for ML model components

Model easy to poison

Lack of data for increasing robustness to poisoning

Poor access rights management

Poor data management

Undefined indicators of proper functioning, making complex compromise identification

Poisoning Lack of consideration of attacks to which ML applications could be exposed to

Use of uncontrolled data

Use of unsafe data or models (e.g. with transfer learning)

Lack of control for poisoning

No detection of poisoned samples in the training dataset

Weak access protection mechanisms for ML model components

Label modification Use of unreliable sources to label data

Poor access rights management

Model or data disclosure Existence of unidentified disclosure scenarios

Weak access protection mechanisms for ML model components

16
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Threats | sub-threats Vulnerabilities

Lack of security process to maintain a good security level of the components of the ML
application

Unprotected sensitive data on test environments

Too much information about the model given in its outputs

Data disclosure The model can allow private information to be retrieved

Disclosure of sensitive data for ML algorithm training

Too much information available on the model

Model disclosure
Too much information about the model given in its outputs

Poor access rights management

Too much information available on the model

Existence of several vulnerabilities because the ML application was not included into process
for integrating security into projects
Use of vulnerable components (among the whole supply chain)

Too much information about the model given in its outputs

Existence of unidentified compromise scenarios

Compromise of ML application Undefined indicators of proper functioning, making complex compromise identification
components
Bad practices due to a lack of cybersecurity awareness
Lack of security process to maintain a good security level of the components of the ML
application
Weak access protection mechanisms for ML model components
Existence of several vulnerabilities because ML specificities are not integrated to existing
policies
Existence of several vulnerabilities because ML application do not comply with security
policies
Contract with a low security third party

Existing biases in the ML model or in the data

ML application not integrated in the cyber-resilience strategy

Existence of unidentified failure scenarios

Undefined indicators of proper functioning, making complex malfunction identification

Failure or malfunction of ML Lack of explainability and traceability of decisions taken

application Lack of security process to maintain a good security level of the components of the ML
application
Existence of several vulnerabilities because ML specificities are not integrated in existing
policies

Contract with a low security third party

Application not compliant with applicable regulations

Poor access rights management

Human error
Lack of documentation on the ML application
Denial of service due to inconsistent
Use of uncontrolled data
data or a sponge example
Cybersecurity incident not reported to
Lack of cybersecurity awareness
incident response teams

17
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

4. SECURITY CONTROLS

4.1 SECURITY CONTROLS RESULTS

Having identified a set of threats that can target vulnerabilities in applications which use ML
algorithms, it is possible to identify which security controls can be put in place to mitigate them.
To do this, we commenced with the vulnerabilities identified in the previous Chapter and came up
with a list of 37 security controls that were then mapped to the taxonomy. Table 5 summarises
security controls for ML algorithms and lists:

• Security controls definitions.

• At which stage of the lifecycle the security controls can be applied.

For ease of reading, they were divided into three categories:

• “Organisational and Policy” are more traditional security controls, either organisational
or linked to security policies.
• “Technical” are more classic technical security controls.
• “Specific to ML” are security controls that are specific to applications using ML.

In Annex 5.C, a set of operational implementation examples are listed for each of the security
controls. This includes:

• For security controls not specific to ML algorithms: examples from the ISO 27001/2 13
family of standards or NIST 800-53 14 framework that should be considered when
implementing the security control.
• For security controls specific to ML: examples of techniques found in the current
literature. All sources are referenced and may be found in Annex 5.D.

The overall mapping of threats, vulnerabilities and security controls is available in Annex 5.B.

13
https://www.iso.org/isoiec-27001-information-security.html
14
https://csrc.nist.gov/publications/detail/sp/800-53/rev-5/final

18
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Table 5: Security controls

Stages of the lifecycle

Data Preprocessing

Model Deployment
Model design and

Model Evaluation
Implementation
Data Collection
Security controls Definition

Model Training

Model Testing
Data Cleaning

Optimisation

Monitoring
ORGANISATIONAL

Define access rights management using a

RBAC (Role Based Access Control) model
respecting the least privileged principle. This
should cover all components of the ML
model (e.g. host infrastructures) and allow
Apply a RBAC model, for the protection of resources such as the
respecting the least privileged model (e.g. its configuration, its code) and x x x x x x x x x x
principle the data it used (e.g. training data).
It is notable that the roles to be included also
concern the end user.
For example: the end user who can submit
inputs to the model should not be able to
have access to its configuration.
As for all projects, documentation must be
produced for AI to preserve knowledge on
the choices made during the project phase,
the application architecture, its configuration,
its maintenance, how to maintain its
Apply documentation
effectiveness over time and the assumptions x x x x x x x x x x
requirements to AI projects
made about the model use.
This documentation should also include the
changes that will be applied, including to the
documentation throughout the algorithm's
life cycle.
As all applications, those using ML can be
subject to regulations and laws (e.g.,
depending on collected data). Such
Assess the regulations and assessment must be done as soon as
laws the ML application must possible during the project phase, and x x x x x x x x x x
comply with should be regularly updated thereafter as
regulations are rapidly evolving (e.g., an AI
Act has been proposed at the European
level).
As all applications, those using ML must
comply with data security requirements to
ensure the overall lifecycle of the data they
Ensure ML applications
use will be secured (e.g. description of data
comply with data security x x x x x x x x x x
lifecycle and associated controls, data
requirements
classification, protection of data at rest and
in transit, use of appropriate cryptographic
means, data quality controls).
As all applications, those using ML must
comply with defined policies regarding
identity management (e.g. ensure all users
are integrated in the departure process),
Ensure ML applications authentication (e.g. passwords complexity,
comply with identity use of Multi-Factors Authentication (MFA),
x x x x x x x x x x
management, authentication, access restriction) and access control (e.g.
and access control policies RBAC model, connection context).
Underlying security requirements must be
applied to all ML application components
(e.g. model configuration, host
infrastructures, training data).

19
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Stages of the lifecycle

Data Preprocessing

Model Deployment
Model design and

Model Evaluation
Implementation
Data Collection
Security controls Definition

Model Training

Model Testing
Data Cleaning

Optimisation

Monitoring
As all applications, those using ML must
Ensure ML applications
comply with protection policies (e.g.
comply with protection
hardening, anti-malware policy) and be x x x x x x x x x x
policies and are integrated to
integrated to security operations processes
security operations processes
(e.g. vulnerability management, backups).

Ensure ML applications As all applications, those using ML must

x x x x x x x x x x
comply with security policies comply with existing security policies.

As all applications, those using ML must be

integrated in global processes for detection
and incident response. This implies
Include ML applications into
collecting the appropriate logs, configuring
detection and response to x x x x x x x x x X
relevant detection use cases to detect
security incident processes15
attacks on the application, and giving the
keys to incident response team for efficient
response.
As all applications, those using ML must be
integrated to global processes for asset
Include ML applications in
management to ensure their assets are x x x x x x x x x X
asset management processes
inventoried, their owners are identified, their
information classified.

As any application, ML ones must be

Integrate ML applications into integrated in the overall cyber-resilience
the overall cyber-resilience strategy, to ensure their architecture and x x x x x x x x x x
strategy operational processes (e.g. backups) take
into account cybersecurity scenario.

Specific ML security attention points should

Integrate ML specificities to be integrated in existing security policies and
x x x x x x x x x x
existing security policies guidelines to ensure they are taken into
consideration.

TECHNICAL

Some model designs are more commonly

used or shared than others and, especially
in the ML field; it can be included in their
lifecycle to widely share them (e.g. open
source sharing). These aspects must be
considered in the global application risk
Assess the exposure level of
analysis. For example, two elements can be x x x x x x x x x x
the model used
distinguished:
- Do not reuse models taken directly from
the internet without checking them.
- Use models for which the threats are
clearly identified and for which security
controls exist.

15
Please note that ML components with false positives might have adverse effect.

20
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Stages of the lifecycle

Data Preprocessing

Model Deployment
Model design and

Model Evaluation
Implementation
Data Collection
Security controls Definition

Model Training

Model Testing
Data Cleaning

Optimisation

Monitoring
During the lifecycle of an ML algorithm,
several components (such as software,
programming libraries or even other models)
are used to complete the project. Security
Check the vulnerabilities of checks have to be carried out to ensure that
the components used so that these components offer an adequate level of
x x x x x x x x x x
they have an appropriate security. Moreover, some mechanisms need
security level to be used to prevent tampering with the
components used.
For example: if an open-source library is to
be used, code reviews or check for public
vulnerabilities on it can be done.
A risk analysis of the overall application
should be conducted to take into account the
specificities of its context, including:
- The attacker’s motivations
- The sensitivity of the data handled (e.g.
medical or personal and thus subject to
regulatory constraints, strategic for the
Conduct a risk analysis of the company and should thus be highly
x x x x x x x x x x
ML application protected)
- The application hosting (e.g. through third
parties services, cloud or on premise
environments)
- The model architecture (e.g. its exposition,
learning methods)
- The ML application lifecycle (e.g., model
sharing

Data must be checked to ensure they will

suit the model and limit the ingestion of
malicious data:
- Evaluate the trust level of the sources to
check it's appropriate in the context of the
application
Control all data used by the - Protect their integrity along the whole data x x x x x x x x x x
ML model supply chain
- Their format and consistence are verified
- Their content is checked for anomalies,
automatically or manually (e.g. selective
human control)
- In the case of labeled data, the issuer of
the label is trusted.

ML is a field in which the use of open-source

elements is widespread (e.g., data for
training, including labeled ones, models).
The trust level of the different sources used
Ensure reliable sources should be assessed to prevent using
compromise ones. x x
are used
For example: the project wants to use
labeled images from a public library. Are the
contributors sufficiently trusted to have
confidence in the contained images or the
quality of their labelling?
Removing suspicious samples from the
Use methods to clean the training and testing dataset can help prevent
training dataset from poisoning attacks. Some methods exist to x x x
suspicious samples identify those that could cause strange
behavior of the algorithm.

21
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Stages of the lifecycle

Data Preprocessing

Model Deployment
Model design and

Model Evaluation
Implementation
Data Collection
Security controls Definition

Model Training

Model Testing
Data Cleaning

Optimisation

Monitoring
Define dashboards of key indicators
integrating security indicators (peaks of
Define and monitor indicators
change in model behavior etc.) to follow-up
for proper functioning of the x
the proper functioning of the model
model
regarding the business case, in particular to
allow rapid identification of anomalies.
Test environments must also be secured
according to the sensitivity of the information
Ensure appropriate protection they contain. Special care must be paid to
is deployed for test the data used in these environments, to x x x x x x x x x x
environments ensure their protection (e.g., same
protection measures as for production if not
desensitiser).
As all applications, those using ML must
Ensure ML applications
comply with third parties’ security
comply with third parties’ x x x x x x x x x x
requirements if their context involves
security requirements
suppliers.
As any project, ML projects must comply to
process for integrating security into projects,
including the followings:
- Risk analysis on the whole application
- Check of the integration of cybersecurity
best practices regarding architecture, secure
development.
- Check that the application will be
Ensure ML projects follow the integrated in existing operational security
global process for integrating processes: monitoring and response, patch x x x x x x x x x x
security into projects management, access management, cyber-
resilience.
- Check of the production of adequate
documentation to ensure the sustainability of
the application (e.g., technical architecture,
hardening, exploitation, configuration and
installation documents)
- Security checks before going to production
(e.g. security audit, pen tests)

SPECIFIC ML

Include adversarial examples to the

Add some adversarial algorithm's training to enable it to be more
examples to the training resilient to such attacks. Depending on the x x x
dataset16 application domain and ambient conditions,
such training could be done continuously.
Adding a step to modify the model's inputs
(e.g. data randomisation which consists in
adding random noise to each piece of data),
can improve the robustness of the model to
Apply modifications on attacks. Such steps can make it more
x x
inputs17 difficult for an attacker to understand the
functioning of the algorithm and thus to
manipulate it and reduce the impacts of an
attack. This security control can be applied
during training or model deployment stages.

16
This security control is often referred to as “Robust adversarial training” in the literature.
17
One important thing to keep in mind is that such modifications should not overly impact model performance on benign
inputs.

22
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Stages of the lifecycle

Data Preprocessing

Model Deployment
Model design and

Model Evaluation
Implementation
Data Collection
Security controls Definition

Model Training

Model Testing
Data Cleaning

Optimisation

Monitoring
The ML models should be explainable, even
if it means simplifying them, to enable a
good understanding of their functioning and
decision factors.
It can also be a regulatory requirement (e.g.
GDPR). However, once again, security
Build explainable models x x
interferes with the explainability property of
the model (easier-to-understand decisions
can be easier-to-build adversarial
examples). It is therefore a trade-off
between the need for explainability and
security.
Some model designs can be more robust
than others against attacks. For instance,
ensemble methods like bagging can mitigate
Choose and define a more
the impact of poisoning (during the training x
resilient model design
phase). Another example is defensive
distillation, which may allow deep neural
networks to better deal with evasion attacks.
Using a set of training data expansion
techniques (e.g. data augmentation)
addresses the lack of data and improves the
robustness of the model to poisoning attacks
by diluting their impact. It is notable,
Enlarge the training dataset however, that this security control more x x
specifically addresses poisoning attacks that
aim to reduce the performance of the model
than those that seek to establish a backdoor.
Moreover, one needs to ensure the reliability
of the sources used to augment the dataset.
The introduction of bias in ML algorithms will
not be detailed because it is not the topic of
the publication.
Ensure that models are However, some techniques can be used to
x x x x x x
unbiased mitigate bias: verify the training dataset is
representative enough regarding the
business case, check the relevance of the
attributes used to make decisions etc.

Differential privacy (DP) is a strong,

mathematical definition of privacy in the
context of statistical and ML analysis.
According to this mathematical definition, DP
is a criterion of privacy protection, which
many tools for analysing sensitive personal
Ensure that models respect information have been devised to satisfy. It
differential privacy to a is noticeable that this security control can x x x x x x x x
sufficient degree greatly reduce the performance of the
model. It is therefore important to estimate
the need for data or model protection.
Example: Differential privacy makes it
possible for technology companies to collect
and share aggregate information about user
habits, while maintaining the privacy of
individual users.

Ensure that the model is sufficiently resilient

Ensure that the model is against the environment in which it will
sufficiently resilient to the operate. This includes, for instance, ensure x x x x x x x x x X
environment in which it will that learning process and data are
operate. representative enough of the real conditions
in which the model will evolve.

23
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Stages of the lifecycle

Data Preprocessing

Model Deployment
Model design and

Model Evaluation
Implementation
Data Collection
Security controls Definition

Model Training

Model Testing
Data Cleaning

Optimisation

Monitoring
ML is a rapidly evolving field, especially
regarding its cybersecurity. Regular
Implement processes to checking of new attacks and defenses must
maintain security levels of ML be integrated into the processes for x x x x x x x x x X
components over time maintaining security level applications. The
security level should thus be regularly
assessed too.
Input-based detection tools can be of
interest to identify whether a given input has
Implement tools to detect if a been modified by an attacker or not.
data point is an adversarial One example, in the case of Deep Neural x x x
example or not Networks (DNNs), is to add a neural
subnetwork to an architecture trained to
detect adversarial examples.

ML considerations should be added to

awareness programs for concerned
stakeholders and they must all receive
cybersecurity awareness training:
Integrate ML specificities to - Global cybersecurity awareness training
awareness strategy and including best practices to prevent attackers x x x x x x x x x x
ensure all ML stakeholders compromising the ML application.
are receiving it - Manipulation of potentially sensitive data or
data subject to regulatory restrictions.
- Configurations to prevent applications
being vulnerable
- ML-specific attack awareness
Before moving the model to production and
then on a regular basis, the model should be
evaluated to ensure it has not been
poisoned. This differs from the security
control “Use methods to clean the training
Integrate poisoning control
dataset from suspicious samples”. Indeed,
after the "model evaluation" x
here, it’s the model itself that is evaluated.
phase
For example: deep learning classification
algorithms can be checked for poisoning
using the STRIP18 technique. The principle is
to disturb the inputs and observe the
randomness of the predictions.
This defense consists of limiting the
information about the model when it is not
necessary. More precisely, it aims at taking
the necessary actions in order to reduce the
information available on the model such as
information on the training data set or any
other information that could be used by an
Reduce the available
attacker (e.g., not publishing the model in x x x x x x x x x x
information about the model
open source). Of course, there is a trade-off
between security and the fact that
stakeholders (e.g., users, ML teams)
sometimes want open source models.
However, it remains notable that in many
cases, research has shown that minimal
information is sufficient to mount attacks.

18
See https://arxiv.org/pdf/1902.06531.pdf. It is notable that STRIP (STRong Intentional Perturbatio) may have a huge
runtime overhead and may be infeasible for large dataset.

24
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Stages of the lifecycle

Data Preprocessing

Model Deployment
Model design and

Model Evaluation
Implementation
Data Collection
Security controls Definition

Model Training

Model Testing
Data Cleaning

Optimisation

Monitoring
Controlling the information (like its verbosity)
provided by the model by applying basic
cybersecurity hygiene rules is a way of
limiting the techniques that an attacker can
use to build adversarial examples.
One of the basic rules of hygiene, for
example, is to reduce the information of the
output determined by the model to the
Reduce the information given
maximum, or by profile making the request. x
by the model19
For example: considering a classification
application, it would consist of
communicating only the predicted class to
the users of solution, not the associated
probability. However, it remains notable that
in many cases, research has shown that
minimal information is sufficient to mount
attacks.
Federated learning is a set of training
techniques that trains a model on several
decentraliser servers containing local data
Use federated learning to samples, without exchanging their data
x x
minimize risk of data breaches samples. This avoids the need to transfer
the data and/or entrust it to an untrusted
third party and thus helps to preserve the
privacy of the data.
The transferability property can be used to
force adversarial examples from a
substitution model to evade another. The
ease of transferring an adversarial example
Use less easily transferable
from a model to another depends on the x
models20
family of algorithms. One possible defense is
thus to choose an algorithm family that is
less sensitive to the transferability of
adversarial examples.

19
It is important to keep in mind that, in case of attacks like evasion or oracle, this security control can help. However, in
some cases, it may possible to bypass the security control by using more queries.
20
Some evasion attacks are based on the following principle: train a model with data like the target model used and
generate adversarial examples from this model. Then, present these adversarial examples to the target model to perform
an evasion attack. Whether or not to transfer an adversarial example generated by one model to another depends on their
respective design as shown in the reference 215.

25
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

5. CONCLUSION

Machine Learning algorithms are at the core of modern AI systems and applications. However, There is no silver
they are faced with a series of threats and vulnerabilities. In this report we have identified bullet for ML-
multiple security controls that can be applied to ML applications to address the threats they
face. Some of the security controls are specific to ML algorithms, but others are standard
specific attacks,
technical and organisational cybersecurity controls to mitigate general attacks. It is important to but mitigation
apply both types of controls because AI systems, in addition to ML specific vulnerabilities, there measures can
exist also general type of vulnerabilities, which may also be exploited by adversaries. still raise the bar
for attackers.
Mitigation controls for ML-specific attacks outlined in the report should in general be deployed
during the entire lifecycle of the ML system. This includes measures for assuring the data
Thus, more
quality and protecting its integrity, making the ML algorithms more robust and controlling access attention should
to both the model and the data to ensure their privacy. The report also emphasizes the need for be given to
the explainability of decisions, and the importance of detecting bias that can be present or security controls
injected in a model by an attacker, which can then lead to unethical uses of AI.
to enable
An important point highlighted in the report is that the identified security measures can be
comparability
applied to all algorithms. Nevertheless, their operational implementations (see Annex C) may be and increase
specific to certain types of algorithms. For example, for the security control “Choose and define resilience.
a more resilient model design”, the defensive distillation implementation is specific to neural
networks. It is also notable that with the prevalence of research papers on supervised learning,
there are more examples of operational implementations for this type of algorithms.

This report addresses an emerging subject. Thus, it remains very important to keep an active
watch on threats and security controls in the field of ML in order to understand the latest
innovations both from a technical point of view, or with a view to comply with standards provided
by ISO, IEEE and ETSI21.

When looking ahead and given the complexity of the issue of securing ML, companies and
governments have new responsibilities. For instance, it is increasingly important to raise
cybersecurity awareness within companies, especially regarding the security of ML systems.
For some populations, particularly data science teams, cybersecurity has not been at the
forefront for many years. Moreover, by including data science actors in these actions, they are
also given the opportunity to think of innovative solutions to mitigate the various threats. Thus,
to this end, training and education programs should be organised regularly and the
vulnerabilities of ML should be demonstrated using concrete examples .

Finally, the context in which security controls are applied is crucial and specific use cases
should be considered when conducting targeted risk assessments. All mitigations used should
be proportional to the application-specific threat level and consider specific conditions of the
environment that may either favor or hamper attacks. Moreover, defenders should be aware of
the following points:

1) There is no silver bullet for mitigating ML-specific attacks. Some security controls may
be bypassed by adaptive attackers. However, applied mitigations can still raise the bar
for attackers.

21
https://www.etsi.org/committee/sai

26
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

2) ML-specific mitigation controls are not generally evaluated in a standardised way even
if it is a current and important issue to enable comparability. More research should be
devoted to standardised benchmarks for comparing ML-specific mitigations on a level
playing field. These benchmarks should also be enforced to ensure that the methods
used in practice are the ones that perform best.

3) Deploying security controls often leads to a trade-off between security and

performance and this is a topic of particular importance that should be further pursued
by the research and cybersecurity communities.

27
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

A ANNEX: TAXONOMY OF
ALGORITHMS

Algorithm Main Data Data Learning Accuracy

Definition Explainability Refs
Name domain type environments Paradigm Provided

AdaBoost uses multiple

iterations to generate a
single composite strong
learner by iteratively adding
weak learners. During each
Classic
phase of training, a new Structured Supervised Classification, Globally
AdaBoost Data 38
weak learner is added to the data learning Regression Explainable
Science
ensemble, and a weighting
vector is adjusted to focus
on examples that were
misclassified in previous
rounds.
Adam optimisation is an
extension to Stochastic
gradient decent and can be
used in place of classical
Classic
Adam stochastic gradient descent Structured
Data / Optimisation 24
optimisation to update network weights data
Science
more efficiently, thanks to
two methods: adaptative
learning rate and
momentum
Agglomerative clustering is
a "bottom-up" approach of
hierarchical clustering. Each Classic
Agglomerative Structured Unsupervised
observation starts in its own Data Clustering 32
clustering data Learning
cluster, and pairs of clusters Science
are merged as one moves
up the hierarchy.
Given a time series Xt, the
ARMA/ARIMA model is a
tool to understand and
predict the future values of Classic
ARMA/ARIMA Supervised
this series. The model is Data Time series Regression Fully Explainable 136
model learning
composed of two parts: an Science
autoregressive part (AR)
and a moving average part
(MA)
Bidirectional Encoder
Representations from
Transformers (BERT) is a
NLP &
Transformer-based ML Supervised
BERT Speech Text Classification Not Explainable Yes 5
technique for natural learning
processing
language processing (NLP)
pre-training developed by
Google.
16, 22,
A Convolutional Neural 36, 43,
Network is a deep learning 49, 50,
algorithm which can take in 56, 58,
Computer
Convolutional an input, assign importance Image, 59; 64,
Vision, NLP Supervised
Neural (learnable weights and video, text, Classification Not Explainable Yes 67, 68,
& Speech learning
Network biases) to various time series 69, 70,
processing
aspects/objects in the data 82, 89,
and be able to differentiate 103,
one from the other. 124,
161

28
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Algorithm Main Data Data Learning Accuracy

Definition Explainability Refs
Name domain type environments Paradigm Provided

DBSCAN - Density-Based
Spatial Clustering of
Applications with Noise is a
density-based clustering
non-parametric algorithm:
given a set of points in
some space, it groups 26,
Computer Unsupervised
DBSCAN together points that are Image Clustering 129,
Vision Learning
closely packed together 142
(points with many nearby
neighbours), marking as
outliers points that lie alone
in low-density regions
(whose nearest neighbours
are too far away).
A decision tree is a graph
that uses a branching
Classic
method to illustrate every Structured Supervised Classification, 40, 42,
Decision tree Data Fully Explainable
possible output for a specific data learning Regression 120,
Science
input in order to break down
complex problems.
Deep Q-learning works as
Q-learning algorithm at the
difference that it uses a
Classic
Deep Q- neural network to Reinforcement
Data Time series Rewarding Yes 65, 85
learning approximate the Q-value learning
Science
function to manage big
amount of states and
actions.
EfficientNet is a
Convolutional Neural
Network based on depth
wise convolutions, which
Computer Supervised
EfficientNet makes it lighter than other Image Classification Not Explainable Yes 4
Vision learning
CNNs. It also allows to
scale the model with a
unique lever: the compound
coefficient.
The factorial
correspondence analysis
(CFA) is a statistical method
of data analysis which
Factor allows the analysis and
prioritisation of the Classic
analysis of Structured Unsupervised Dimension
information contained in a Data
corresponde data Learning Reduction
rectangular table of data Science
nces and which is today
particularly used to study
the link between two
variables (qualitative or
categorical).
A GAN is a generative
model where two networks
are placed in competition.
The first model is the
generator, it generates a
sample (e.g. an image),
Computer Image, Unsupervised
GAN while its opponent, the 53, 135
Vision Video Learning
discriminator, tries to detect
whether a sample is real or
whether it is the result of the
generator. Both improve on
the performance of the
other.

29
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Algorithm Main Data Data Learning Accuracy

Definition Explainability Refs
Name domain type environments Paradigm Provided

A Gaussian mixture model

is a probabilistic model that
Computer Text, time
assumes all the data points
Vision, NLP series, Unsupervised
GMM are generated from a Clustering 31, 131
& Speech Image, Learning
mixture of a finite number of
processing video,
Gaussian distributions with
unknown parameters.
Generative Pre-trained
Transformer 3 (GPT-3) is an
NLP &
autoregressive language Supervised
GPT-3 Speech Text Classification Not Explainable Yes 6
model that uses deep learning
processing
learning to produce human-
like text.
Gradient boosting is a
Gradient technique that optimises a Classic 3, 51,
Structured Supervised Classification, Globally
boosting decision tree by combining Data 54, 55,
data learning Regression Explainable
machine weak models to improve Science 140
model prediction.
Gradient descent is a first-
order iterative optimisation
algorithm for finding a local
minimum of a differentiable
function. The idea is to take
Classic
Gradient repeated steps in the Structured
Data / Optimisation 17
descent opposite direction of the data
Science
gradient (or approximate
gradient) of the function at
the current point, because
this is the direction of
steepest descent.
Graph neural networks
(GNNs) are deep learning-
based methods that operate Computer
Graph neural
on graph domain. Graphs Vision, Supervised Regression,
networks Image 20
are a kind of data structure Speech learning classification
(GNNs) which models a set of processing
objects (nodes) and their
relationships (edges)
Hierarchical clustering is a
method of cluster analysis
which seeks to build a
Classic
Hierarchical hierarchy of clusters. The Structured Unsupervised
Data Clustering 32
clustering result is a tree-based data Learning
Science
representation of the
objects, named a
dendrogram.
Hidden Markov Model is a
statistical Markov model in Structured
Hidden Structured
which the system being data, NLP & Reinforcement
Markov data, time Rewarding Yes 29
modelled is assumed to be Speech learning
Model (HMM) series, text
a Markov process with processing
unobservable hidden states.
ICA is a special case of
blind source separation. A
Independent common example Classic
Structured Unsupervised Dimension
component application is the "cocktail Data 2
data Learning Reduction
analysis party problem" of listening in Science
on one person's speech in a
noisy room.

30
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Algorithm Main Data Data Learning Accuracy

Definition Explainability Refs
Name domain type environments Paradigm Provided

The isolation forest returns

the anomaly score of each
sample. It isolates
observations by randomly
Classic
Isolation selecting a feature, and then Structured Unsupervised Anomaly 157,
Data
forest randomly selecting a split data learning detection 161
Science
value between the
maximum and minimum
values of the selected
feature.
K-means clustering is a
method of vector
quantification that aims to
partition n observations into
Classic
k clusters in which each Structured Unsupervised
K-means Data Clustering 129
observation belongs to the data Learning
Science
cluster with the nearest
mean (cluster centres or
cluster centroid), serving as
a prototype of the cluster.
K-Nearest Neighbour is a
simple algorithm that stores
all the available cases and
classifies the new data or Classic
K-Nearest Structured Supervised
case based on a similarity Data Classification Fully Explainable Yes 21, 40,
Neighbour data learning
measure. It is mostly used Science
to classify a data point
based on how its
neighbours are classified.
Linear regression attempts
to model the relationship
between two or more
variables by fitting a linear
equation to observed data.
One variable is considered
to be an explanatory Classic
Linear Structured Supervised 2, 117,
variable, and the other is Data Regression Fully Explainable
regression data learning 221
considered to be a Science
dependent variable. For
example, a modeller might
want to relate the weights of
individuals to their heights
using a linear regression
model.
Logistic regression is used
to classify data by modelling
the probability of a certain
class or event existing such
as pass/fail, win/lose, Classic
Logistic Structured Supervised 2, 120,
alive/dead or healthy/sick. Data Classification Fully Explainable Yes
regression data learning 177
This can be extended to Science
model several classes of
events such as determining
whether an image contains
a cat, dog, lion, etc.
Long short-term memory
(LSTM) is an artificial
recurrent neural network
(RNN) architecture used in 22, 44,
the field of deep learning. NLP & 45, 46,
Unlike standard feedforward Speech Text, 47, 50,
Supervised
LSTM neural networks, LSTM has processing, image, Regression Not Explainable 84, 85,
learning
feedback connections. It computer video 131,
cannot only process single vision 158,
data points (such as 161
images), but also entire
sequences of data (such as
speech or video).

31
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Algorithm Main Data Data Learning Accuracy

Definition Explainability Refs
Name domain type environments Paradigm Provided

Mean shift is a non-

parametric feature-space
analysis technique for Computer Image, Unsupervised
Mean shift Clustering 27
locating the maxima of a Vision video Learning
density function, a so-called
mode-seeking algorithm
Computer
MobileNets are based on a
Vision,
streamlined architecture that Image,
Classic
uses depth-wise separable video, text,
Data Unsupervised
MobileNet convolutions instead of time series, Clustering Yes 4
Science, learning
convolutions, in order to structured
NLP &
build light wFeight deep data
Speech
neural networks.
processing
A Monte Carlo algorithm is a
randomised algorithm Classic
Monte Carlo Structured Reinforcement
whose output may be Data Rewarding 30, 105
algorithm data learning
incorrect with a certain Science
(typically small) probability.
A Multimodal Parallel
Network helps to manage Computer
Multimodal
audio-visual event Vision, Supervised
Parallel Video Classification 18
localisation by processing Speech learning
Network both audio and visual processing
signals at the same time.
Naive Bayes classifiers are
a family of simple
39, 40,
probabilistic classifiers Classic
Naive Bayes Structured Supervised 89,
based on applying Bayes' Data Classification Fully Explainable Yes
classifiers data learning 120,
theorem with strong (naïve) Science
210
independence assumptions
between the features.
A family of policy gradient
methods for Reinforcement
Proximal Learning that alternate Classic Structured
Reinforcement
Policy between sampling data and Data data, time Rewarding Yes 137
learning
Optimisation optimising a surrogate Science series
objective function using
stochastic gradient ascent.
The main idea of principal
component analysis (PCA)
is to reduce the
dimensionality of a data set
Principal Classic
consisting of many variables Structured Unsupervised Dimension
Component Data 2
correlated with each other, data Learning Reduction
Analysis Science
either heavily or lightly,
while retaining the variation
present in the dataset, up to
the maximum extent.
Q-learning is a model-free
reinforcement learning
Classic Structured
algorithm to learn the value Reinforcement
Q-learning Data data, time Rewarding Yes 28
of an action in a particular learning
Science series
state. It does not require a
model of the environment.
Random forests are an
ensemble learning method
that operates by
constructing a multitude of
decision trees at training Classic 51,
Random Structured Supervised Classification, Globally
time and outputting the Data 136,
forests data learning Regression Explainable
class that is the mode of the Science 140
classes (classification) or
mean/average prediction
(regression) of the individual
trees.

32
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Algorithm Main Data Data Learning Accuracy

Definition Explainability Refs
Name domain type environments Paradigm Provided

A recurrent neural network

(RNN) is a class of artificial 14, 17,
neural networks where Computer Time 44, 45,
Recurrent
connections between nodes Vision, NLP series, text, Supervised 46, 47,
neural Regression Not Explainable
form a directed graph along & Speech image, learning 49, 50,
network a temporal sequence. This processing video 52, 89,
allows it to exhibit temporal 13
dynamic behaviour.
A residual neural network
(ResNet) is an artificial
neural network (ANN) that
builds on constructs known
Computer Supervised
ResNet from pyramidal cells in the Image Classification Not Explainable Yes 4, 7, 37
Vision learning
cerebral cortex by utilising
skip connections, or
shortcuts to jump over some
layers.

Spatial Spatial Temporal Graph

Convolutional Networks is a
Temporal
convolutional neural Computer Supervised
Graph Video Classification 25
network that automatically Vision learning
Convolutiona learns both the spatial and
l Networks temporal patterns from data.
Stochastic gradient descent
is an iterative method for
optimising an objective
function with suitable
smoothness properties. It
can be regarded as a
Stochastic stochastic approximation of Classic
Structured
gradient gradient descent Data / Optimisation 17, 24
data
descent optimisation, since it Science
replaces the actual gradient
(calculated from the entire
data set) by an estimate
thereof (calculated from a
randomly selected subset of
the data).
42, 47,
51, 67,
SVM are linear classifiers
69, 87,
which are based on the
89, 92,
margin maximisation
98,
principle. They accomplish
Support Classic 106,
the classification task by Structured Supervised
vector Data Classification Fully Explainable Yes 120,
constructing, in a higher data learning
machine Science 136,
dimensional space, the
139,
hyperplane that optimally
142,
separates data into two
152,
categories.
177,
185
Wavenet is a deep neural
network for generating raw
audio waveforms. The
model is fully probabilistic NLP & 44,
Unsupervised
WaveNet and autoregressive, with the Speech Time series NLP task 45,131,
learning
predictive distribution for processing 132
each audio sample
conditioned on all previous
ones
XGBoost is an extension to
gradient boosted decision
trees (GBM) and specially Classic
Structured Supervised Classification, Globally
XGBoost designed to improve speed Data 3
data learning Regression Explainable
and performance by using Science
regularisation methods to
fight overfitting.

33
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

B ANNEX: MAPPING
SECURITY CONTROLS
TO THREATS

Threats | sub- Threats

Vulnerabilities Security Controls
threats references

Implement tools to detect if a data point is an

adversarial example or not
Lack of detection of abnormal inputs
Include ML applications in detection and
response to security incident processes 13, 34, 37, 48,
Poor consideration of evasion attacks Choose and define a more resilient model 49, 51, 53, 56,
in the model design implementation design 59, 60, 62, 65,
66, 67, 73, 80,
Lack of consideration of attacks to Integrate ML specificities to awareness 81, 82, 83, 84,
which ML applications could be strategy and ensure all ML stakeholders are 90, 95, 97,
exposed receiving it 100, 107, 109,
110, 121, 125,
Evasion Lack of training based on adversarial Add some adversarial examples to the training 139, 144, 154,
attacks dataset 155, 162, 163,
Lack of security process to maintain a good 169, 170, 175,
181, 183, 185,
security level of the components of the ML
199, 200, 201,
Use a widely known model allowing application 202, 204, 205,
the attacker to study it Use less easily transferable models 206, 207, 209,
211, 213, 215
Assess the exposure level of the model used
Inputs totally controlled by the attacker
Apply modifications to inputs
which allows for input-output-pairs
Use of Too much information available on the Reduce the available information about the 34, 35, 48, 51,
adversarial model model 56, 59, 60, 62,
examples 65, 80, 81, 82,
crafted in white 100, 109, 110,
or grey box Too much information about the model 125, 139, 144,
Reduce the information given about the model
conditions (e.g. given in its outputs 154, 170, 204,
FGSM…) 209

Apply a RBAC model, respecting the least

Poor access rights management
privileged principle
The model allows private information
Ensure that models respect differential privacy
to be retrieved
Too much information about the model
Reduce the information given about the model
given in its outputs
Too much information available on the Reduce the available information about the
model model 121, 145, 146,
152, 170, 177,
Oracle Lack of consideration of attacks to Integrate ML specificities to awareness 194, 203, 204,
which ML applications could be strategy and ensure all ML stakeholders are 208, 214
exposed to receiving it
Lack of security process to maintain a
Implement processes to maintain security
good security level of the components
levels of ML components over time
of the ML application
Weak access protection mechanisms Ensure ML applications comply with identity
for ML model components management, authentication, and process
control policies

34
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Threats | sub- Threats

Vulnerabilities Security Controls
threats references

Choose and define a more resilient model

design
Model easy to poison Implement processes to maintain security
levels of ML components over time
Assess the exposure level of the model used
Lack of data for increasing robustness
Enlarge the training dataset
to poisoning
Apply a RBAC model, respecting the least
Poor access rights management
privileged principle
Ensure ML applications comply with data
Poor data management 74, 77, 79, 99,
security requirements
114, 115, 116,
Undefined indicators of proper 117, 118, 121,
Define and monitor indicators for proper
functioning, making complex 126, 140, 142,
functioning of the model
Poisoning compromise identification 143, 162, 167,
170, 171, 172,
Lack of consideration of attacks to Integrate ML specificities to awareness
173, 189, 196,
which ML applications could be strategy and ensure all ML stakeholders are 197, 198, 199,
exposed to receiving it 204, 210
Use of uncontrolled data Control all data used by the ML model
Use of unsafe data or models (e.g with
Ensure reliable sources are used
transfer learning)
Integrate poisoning control after the "model
Lack of control for poisoning
evaluation" phase
No detection of poisoned samples in Use methods to clean the training dataset from
the training dataset suspicious samples
Ensure ML applications comply with identity
Weak access protection mechanisms
management, authentication, and access
for ML model components
control policies
Label
Use of unreliable source to label data Ensure reliable sources are used 125, 140, 204
modification
Apply a RBAC model, respecting the least
Poor access rights management
privileged principle
Existence of unidentified disclosure
Conduct a risk analysis of the ML application
scenarios
Ensure ML applications comply with identity
Weak access protection mechanisms
Model or data management, authentication, and access
for ML model components
disclosure control policies
Lack of security process to maintain a
Implement processes to maintain security
good security level of the components
levels of ML components over time
of the ML application
Unprotected sensitive data on test Ensure appropriate protection are deployed for
environments test environments as well
121, 194, 221,
Integrate ML specificities to awareness
Too much information about the model 222
strategy and ensure all ML stakeholders are
given in its outputs
receiving it
The model can allow private
Data Ensure that models respect differential privacy
information to be retrieved
disclosure
The model can allow private
Reduce the information given by the model
information to be retrieved
Disclosure of sensitive data for ML Use federated learning to minimise the risk of
algorithm training data breaches
Too much information available on the Reduce the available information about the
Model model model
disclosure Too much information about the model
Reduce the information given by the model
given in its outputs

35
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Threats | sub- Threats

Vulnerabilities Security Controls
threats references

Apply a RBAC model, respecting the least

Poor access rights management
privileged principle
Too much information available on the Reduce the available information about the
model model
Existence of several vulnerabilities
because the ML application was not Ensure ML projects follow the global process
integrated into process for integrating for integrating security into projects
security into projects
Check the vulnerabilities of the components
Use of vulnerable components (among
used so that they have an appropriate security
the whole supply chain)
level
Too much information about the model
Reduce the information given by the model
given in its outputs
Existence of unidentified compromise
Conduct a risk analysis of the ML application
scenarios
Undefined indicators of proper
Define and monitor indicators for proper
functioning, making complex
functioning of the model
compromise identification
Compromise of ML
Integrate ML specificities to awareness 121, 164, 183,
application Bad practices due to a lack of
strategy and ensure all ML stakeholders are 189
components cybersecurity awareness
receiving it
Implement processes to maintain security
Lack of security process to maintain a levels of ML components over time
good security level of the components Ensure ML applications comply with protection
of the ML application policies and are integrated to security
operations processes
Ensure ML applications comply with identity
Weak access protection mechanisms
management, authentication, and access
for ML model components
control policies
Existence of several vulnerabilities
Integrate ML specificities to existing security
because ML specificities are not
policies
integrated to existing policies
Ensure ML applications comply with security
Existence of several vulnerabilities policies
because ML application do not comply
with security policies Include ML applications into asset
management processes
Ensure ML applications comply with third
Contract with a low security third party
parties’ security requirements
Existing biases in the ML model or in
Ensure that models are unbiased
the data
Lack of consideration of real-life Ensure that the model is sufficiently resilient to
conditions in training the model the environment in which it will operate.
ML application not integrated in the Integrate ML applications into the overall
cyber-resilience strategy cyber-resilience strategy
Existence of unidentified failure
Failure or Conduct a risk analysis of the ML application
scenarios 121, 164, 183,
malfunction of ML
189, 191
application Undefined indicators of proper
Define and monitor indicators for proper
functioning, making complex
functioning of the model
malfunction identification
Lack of explainability and traceability
Build explainable models
of decisions taken
Lack of security process to maintain a
Implement processes to maintain security
good security level of the components
levels of ML components over time
of the ML application

36
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Threats | sub- Threats

Vulnerabilities Security Controls
threats references

Existence of several vulnerabilities

Ensure ML projects follow the global process
because ML specificities are not
for integrating security into projects
integrated to existing policies
Ensure ML applications comply with third
Contract with a low security third party
parties’ security requirements
Application not compliant with Assess the regulations and laws the ML
applicable regulations application must comply with
Apply a RBAC model, respecting the least
Poor access rights management
privilege principle
Apply documentation requirements to AI
Human error
Lack of documentation on the ML projects
application Include ML applications into asset
management processes
Denial of
service due to
inconsistent
Use of uncontrolled data Control all data used by the ML model
data or a
sponge
example
Cybersecurity
incident not
Integrate ML specificities to awareness
reported to
Lack of cybersecurity awareness strategy and ensure all ML stakeholders are
incident
receiving it
response
teams

37
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

C ANNEX: IMPLEMENTING
SECURITY CONTROLS

Security controls Examples for operational implementation References

The literature provides the following techniques:

- Adversarial Training
Add some adversarial
- Ensemble Adversarial Training 13, 23, 48, 51, 59, 65, 72, 95, 108,
examples to the training
- Cascade Adversarial Training 162, 200, 201, 202, 211, 215
dataset
- Principled Adversarial Training
- Gradient Band Based Adversarial Training

The NIST 800-53 and the ISO 27001/2 provides several

points:
Apply a RBAC model, - Manage access permissions and authorisations,
ISO 27001/2
respecting the least privilege incorporating the principles of least privilege and
NIST 800-53, 162
principle separation of duties
- Manage the identity of the users (Couple lifecycle
management processes and procurement processes etc.)

The NIST 800-53 and the ISO 27001/2 provides several 148
Apply documentation
points:
requirements to Artificial
- Define change management processes, integrating the ISO 27001/2
Intelligence projects
update of the documentation NIST 800-53

The literature provides the following techniques:

- Data randomisation
Apply modifications on inputs 64, 65, 108, 208
- Input transformation
- Input denoising

Assess the exposure level of

the model used

The NIST 800-53 and the ISO 27001/2 provides several

Assess the regulations and points: ISO 27001/2
laws the ML application must
- Identify applicable legislation NIST 800-53, 162
comply with
- Meet the requirements of GDPR for personal data

The literature provides the following techniques:

- Interpret models with some tools
Build explainable models - Use model more explainable like regression instead of 164, 225, 226, 227
Deep Neural Network for supervised learning when it is
necessary

38
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Security controls Examples for operational implementation References

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Manage the exemptions by following it industrially, also
including remediation plans
- Make an inventory of the infrastructure equipment, the
applications (Define, document, improve and review a
Check the vulnerabilities of the
regular process to make inventory)
components used so that they ISO 27001/2
have an appropriate security - Manage the maintenance, the obsolete assets etc.
NIST 800-53
(Define a process and continuously improve it, define a
level
roadmap to replace obsolescent technologies)
- Implement a vulnerability management policy (Control
regularly its implementation)
- Perform and manage vulnerability scans on servers OS,
middleware, database and network infrastructure (Perform
regularly automatics scans)

For poisoning, the literature provides the following

technique:
- Bagging or weight Bagging
- TRIM algorithm

For evasion, the literature provides the following

Choose and define a more technique: 35, 65, 108, 110, 112, 113, 114,
resilient model design 143, 196, 206, 213
- Randomisation
- Stability terms into objective function
- Adversarial perturbation-based regulariser
- Input gradient regularisation
- Defensive distillation
- Random feature nullification

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Coordinate the compliance process with legal and audit
functions
Conduct a risk analysis of the - Identify legal requirements (e.g. GDPR or NIS for ISO 27001/2
ML application European countries) NIST 800-53
- Establish a methodology to manage identified risks
- Establish a formal methodology to analyse risk
- Define and monitor the IT resource availability
(Formalise a capacity management plan)

The literature provides the following techniques:

- Data sanitisation
Control all data used by the ML
- RONI and tRONI technics 114, 116, 118, 142, 162, 197, 228
model
- Point out important data and put a human in the loop
(Human in the loop)

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Formalise a dashboard, bringing together a series of
indicators enabling the state of the information system to
Define and monitor indicators be judged in relation to the objectives set.
ISO 27001/2
for proper functioning of the - Take actions in case of deviation from the objective
model NIST 800-53
- Manage changes on assets, ensure changes will not
impact the production and detect any changes in assets'
baseline configuration
- Guarantee the integrity of the code at all stage (Perform
integrity control etc.)

39
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Security controls Examples for operational implementation References

The literature provides the following technique:

Enlarge the training dataset 68, 150
- Data augmentation

The NIST 800-53 and the ISO 27001/2 provides several

Ensure appropriate protection
points: ISO 27001/2
are deployed for test
- Protect data when there are in non-production NIST 800-53
environments as well
environment (implement desensitisation measures etc.)

Ensure that the model is

sufficiently resilient to the
environment in which it will
operate.

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Apply a methodology for data classification. Review the
classification regularly
Ensure ML applications comply - Implement measures to detect data leakage on the ISO 27001/2
with Data Security Internet (Antivirus, Data right management solution on all
requirements sensitive folders) NIST 800-53
- Secure sensitive data in transit (Deploy mechanisms to
detect bypasses on all networks)
- Deploy security solutions on network points to prevent
data leaks (DLP etc.)

The NIST 800-53 and the ISO 27001/2 provides several

points:
Ensure ML applications comply - Define a policy regarding users’ authentication (Define
with identity management, an authentication policy that considers the sensitivity of ISO 27001/2
authentication and access resources and the connection context for all types of NIST 800-53, 162
control policies account, use Multi-Factor Authentication)
- Define a remote access policy (Verify security
configuration, authenticate connected devices)

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Manage the maintenance, the obsolete assets etc.
Ensure ML applications comply (Define a process and continuously improve it, define a
with protection policies and are roadmap to replace obsolescent technologies) ISO 27001/2
integrated to security - Implement a vulnerability management policy (Control NIST 800-53
operations processes regularly its implementation)
- Perform and manage vulnerability scans on servers OS,
middleware, database and network infrastructure (Perform
regularly automatics scans)

The NIST 800-53 and the ISO 27001/2 provides the

Ensure ML applications comply ISO 27001/2
following point:
with security policies NIST 800-53
- Define policies for information security

The NIST 800-53 and the ISO 27001/2 provides several

points:
Ensure ML applications comply - Integrate the security into contracts (Define a security
ISO 27001/2
with third parties’ security insurance plan for strategic contracts representing high
NIST 800-53
requirements risks for company security. Communicate roles and
responsibilities to every new third party)
- Monitor and review third parties services

40
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Security controls Examples for operational implementation References

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Integrate the security into contracts (Define a security
insurance plan for strategic contracts representing high
risks for company security. Communicate roles and
Ensure ML projects follow the responsibilities to every new third party)
ISO 27001/2
global process for integrating - Define and manage a patch management policy
NIST 800-53
security into projects - Manage the interconnections with external systems
(establish a formal process to regularly review the
exhaustiveness of interconnections inventory)
- Integrate and manage security protection for applications
(firewalls, WAF, reverse proxy)
- Perform security controls on application

The NIST 800-53 and the ISO 27001/2 provides several

points:
Ensure reliable sources are ISO 27001/2
used - Manage the interconnections with external systems
NIST 800-53
(establish a formal process to regularly review the
exhaustiveness of interconnections inventory)

The literature provides the following techniques:

- Classification parity
- Calibration
Ensure that models are
- Anti-classification 217, 229
unbiased
- Having a diverse dataset
- Some other technics: samples bias, measurement
error…

The literature provides the following techniques:

- Model design adapted like PATE for Deep Neural
Ensure that models respect Network based classifier
differential privacy to a 63, 108, 112, 194, 203, 208, 214
- Data randomisation
sufficient degree
- Randomisation
- Objective function perturbation

The NIST 800-53 and the ISO 27001/2 provides the

Implement processes to
following point: ISO 27001/2
maintain security levels of ML
components over time - Perform technical and organisational audits regularly on NIST 800-53
critical scope and develop an action plan after each audit

Implement tools to detect if a

The literature provides the following technique:
data point is an adversarial 65, 207
example or not - Adding detector subnetworks

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Create an inventory of the infrastructure equipment, the
applications (Define, document, improve and review a
Include ML applications into ISO 27001/2
regular process to make inventory)
asset management processes NIST 800-53
- Classify information
- Manage the maintenance, the obsolete assets etc.
(Define a process and continuously improve it, define a
roadmap to replace obsolescent technologies)

41
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

Security controls Examples for operational implementation References

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Include ML projects into the Business Continuity Plan
- Include ML projects into the Cybersecurity Disaster
Include ML applications into
Recovery Plan ISO 27001/2
detection and response to
security incident processes - Define a backup strategy for ML projects (and test it) NIST 800-53
- Define a strategy for public relations during recovery
(Identify and train possible spokespersons, Adapt
communication responses to different categories of
interlocutors)

The NIST 800-53 and the ISO 27001/2 provides several

points:
- Include ML projects into the Business Continuity Plan
- Include ML projects into the Cybersecurity Disaster
Integrate ML applications into
Recovery Plan ISO 27001/2
the overall cyber-resilience
strategy - Define a backup strategy for ML projects (and test it) NIST 800-53, 162
- Define a strategy for public relations during recovery
(Identify and train possible spokespersons, Adapt
communication responses to different categories of
interlocutors)

The NIST 800-53 and the ISO 27001/2 provides the

Integrate ML specificities to ISO 27001/2
following point:
existing security policies NIST 800-53
- Review policies for information security

Integrate ML specificities to The NIST 800-53 and the ISO 27001/2 provides several
awareness strategy and ensure points: ISO 27001/2
all ML stakeholders are - Organise training sessions NIST 800-53
receiving it - Perform locally cyber risks reporting

Integrate poisoning control

The literature provides the following technique:
after the "model evaluation" 198
phase - STRIP technique

The NIST 800-53 and the ISO 27001/2 provides the

Reduce the available ISO 27001/2
following point:
information about the model NIST 800-53
- Implement a classification policy

Reduce the information given The literature provides the following technique:
89, 145
by the model - Gradient Masking

Use federated learning to

194
minimise risk of data breaches

Use less easily transferable

65, 215
models

The literature provides the following techniques:

Use methods to clean the - Data sanitisation
training dataset from suspicious - RONI and tRONI technics 114, 162, 210
samples - Point out important data and put a human in the loop
(Human in the loop)

42
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

D ANNEX: REFERENCES

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
Adversarial Machine https://www.morga
• Yevgeniy
Learning, Synthesis nclaypool.com/doi/
Vorobeychik
1 Lectures on Artificial abs/10.2200/S008 2018 X
• Murat
Intelligence and 61ED1V01Y20180
Kantarcioglu
Machine Learning 6AIM039

https://web.stanfor • Trevor Hastie

The Elements of
2 d.edu/~hastie/Pap 2001 • Robert Tibshirani X X X X X X X X X
Statistical Learning
ers/ESLII.pdf • Jerome Friedman
XGBoost: a scalable https://arxiv.org/pdf • Tianqi Chen
3 2016 X X
tree boosting system /1603.02754.pdf • Carlos Guestrin
EfficientNet: Rethinking
https://proceedings
Model Scaling for • Mingxing Tan
4 .mlr.press/v97/tan1 2019 X X
Convolutional Neural • Quoc V. Le
9a/tan19a.pdf
Networks

BERT: Pre-training of • Jacob Devlin

Deep Bidirectional • Ming-Wei Chang
https://arxiv.org/pdf
5 Transformers for 2019 • Kenton Lee X X
/1810.04805.pdf
Language • Kristina
Understanding Toutanova

• Tom B. Brown
• Benjamin Mann
• Nick Ryder
• Melanie Subbiah
• Jared Kaplan
• Prafulla Dhariwal
• Arvind
Neelakantan
• Pranav Shyam
• Girish Sastry
• Amanda Askell
• Sandhini Agarwal
• Ariel Herbert-
Voss
Language Models are https://arxiv.org/pdf • Gretchen Krueger
6 2020 X X
Few-Shot Learners /2005.14165.pdf • Tom Henighan
• Rewon Child
• Aditya Ramesh
• Daniel M. Ziegler
• Jeffrey Wu
• Clemens Winter
• Christopher
Hesse
• Mark Chen
• Eric Sigler
• Mateusz Litwin
• Scott Gray
• Benjamin Chess
• Jack Clark
• Christopher

43
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
Berner
• Sam McCandlish
• Alec Radford
• Ilya Sutskeve
• Dario Amodei

• Preetum
Nakkiran
Deep Double Descent: • Gal Kaplun
https://arxiv.org/pdf
7 Where Bigger Models 2019 • Yamini Bansal X X
/1912.02292.pdf
and More Data Hurt • Tristan Yang
• Boaz Barak
• Ilya Sutskever
• Kaiming He
Deep Residual
https://arxiv.org/pdf • Xiangyu Zhang
8 Learning for Image 2015 X X
/1512.03385.pdf • Shaoqing Re
Recognition
• Jian Sun
• Kazuki Osawa
• Siddharth
Swaroop
https://papers.nips.
• Anirudh Jain
Practical Deep cc/paper/2019/file/
• Runa
9 Learning with Bayesian b53477c2821c1bf0 2019 X X
Eschenhagen
Principles da5d40e57b870d3
• Richard E. Turner
5-Paper.pdf
• Rio Yokota
• Mohammad
Emtiyaz Khan
Deep Inside
Convolutional • Karen Simonyan
Networks: Visualising https://arxiv.org/pdf • Andrea Vedaldi
10 2014 X X
Image Classification /1312.6034.pdf • Andrew
Models and Saliency Zisserman
Maps
https://web.stanfor
d.edu/class/psych2
Reinforcement learning: • Richard S. Sutton
11 09/Readings/Sutto 1998 X X X X
An introduction • Andrew G. Barto
nBartoIPRLBook2n
dEd.pdf
• Yuri Burda
Exploration by Random https://arxiv.org/pdf • Harrison Edwards
12 2018 X X
Network Distillation /1810.12894.pdf • Amos Storkey
• Oleg Klimov

DeepFool: a simple
• Seyed-Mohsen
and accurate method to
https://arxiv.org/pdf • Moosavi-Dezfooli
13 fool deep neural 2016 X
/1511.04599.pdf • Alhussein Fawzi
networks, in arXiv, July
• Pascal Frossard
2016

• Yingce Xia
• Di He
• Tao Qin
Dual Learning for https://arxiv.org/pdf
14 2016 • Liwei Wang X X
Machine Translation /1611.00179.pdf
• Nenghai Yu
• Tie-Yan Liu
• Wei-Ying Ma

44
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Jingwei Xu
• Huazhe Xu
Video Prediction via https://arxiv.org/pdf
15 2020 • Bingbing Ni X X
Example Guidance /2007.01738.pdf
• Xiaokang Yang
• Trevor Darrell

Context-aware • Ardhendu Behera

Attentional Pooling https://arxiv.org/pdf • Zachary Wharton
16 2021 X X
(CAP) for Fine-grained /2101.06635.pdf • Pradeep Hewage
Visual Classification • Asish Bera
Ilya
On the importance of
http://proceedings. Sutskever/James
initialisation and
17 mlr.press/v28/suts 2013 Martens /George X X X X X
momentum in deep
kever13.pdf Dahl/Geo↵rey
learning
Hinton

MPN: MULTIMODAL
PARALLEL NETWORK • Jiashuo Yu
https://arxiv.org/pdf
18 FOR AUDIO-VISUAL 2021 • Ying Cheng X X X
/2104.02971.pdf
EVENT • Rui Feng
LOCALISATION

Zero-Gradient
Constrained • Kazuki
https://arxiv.org/pdf
19 Optimisation for 2021 Naganuma X X
/2104.02845.pdf
Destriping of 3D • Shunsuke Ono
Imaging Data
• Chen Min
• Jiaolong Xu
Attentional Graph
https://arxiv.org/pdf • Liang Xiao
20 Neural Network for 2021 X X
/2104.02576.pdf • Dawei Zhao
Parking-slot Detection
• Yiming Nie
• Bin Dai
Identity and Posture
• Vandad
Recognition in Smart https://arxiv.org/pdf
21 2019 Davoodnia X X
Beds with Deep /2104.02159.pdf
• Ali Etemad
Multitask Learning

• Abdulaziz M.
A Combined CNN and Alayba
https://arxiv.org/pdf
22 LSTM Model for Arabic 2018 • Vasile Palade X X
/1807.02911.pdf
Sentiment Analysis • Matthew England
• Rahat Iqbal

• Mathias Lechner
• Ramin Hasani
Adversarial Training is
https://arxiv.org/ab • Radu Grosu
23 Not Ready for Robot 2021 X X X
s/2103.08187 • Daniela Rus
Learning
• Thomas A.
Henzinger

http://iwqos2018.ie
Improved Adam ee-
24 Optimizer for iwqos.org/files/201 2018 • Zijun Zhang X X X X X
Deep Neural Networks 8/05/Improved_Ad
am_Optimizer.pdf

https://ieeexplore.i
Vision-Based Fall
eee.org/stamp/sta • Oussema Keskes
25 Detection Using ST- 2021 X X
mp.jsp?tp=&arnum • Rita Noumeir
GCN
ber=9351913

45
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
DBSCAN++: Towards
https://arxiv.org/pdf • Jennifer Jang
26 fast and scalable 2019 X X
/1810.13105.pdf • Heinrich Jiang
density clustering
MeanShift++:
Extremely Fast Mode-
Seeking With https://arxiv.org/pdf • Jennifer Jang
27 2021 X X X
Applications to /2104.00303.pdf • Heinrich Jiang
Segmentation and
Object Tracking
Self-correcting Q- https://arxiv.org/pdf • Rong Zhu
28 2021 X X
Learning /2012.01100.pdf • Mattia Rigotti
• Taha Mansouri
• Mohamadreza
A New Algorithm for https://arxiv.org/ftp/
Sadeghimoghada
29 Hidden Markov Models arxiv/papers/2102/ 2021 X X X X
m
Learning Problem 2102.07112.pdf
• Iman Ghasemian
Sahebi

Deep Reinforcement
Learning Aided Monte • Tz-Wei Mo
https://arxiv.org/pdf
30 Carlo 2021 • Ronald Y. Chang X X X
/2102.00178.pdf
Tree Search for MIMO • Te-Yi Kan
Detection

Hard-Clustering with • Johannes Blomer

https://arxiv.org/pdf
31 Gaussian Mixture 2016 • Sascha Brauer X X X X X X
/1603.06478.pdf
Models • Kathrin Bujna

• Marcel R.
Analysis of Ackermann
https://arxiv.org/pdf
32 Agglomerative 2014 • Johannes Blomer X X
/1012.3697.pdf
Clustering • Daniel Kuntze
• Christian Sohler

Towards Evaluating the

https://arxiv.org/pdf • Nicholas Carlini
33 Robustness of Neural 2017 X X
/1608.04644.pdf • David Wagner
Networks

DELVING INTO
• Yanpei Liu
TRANSFERABLE
https://arxiv.org/pdf • Xinyun Chen
34 ADVERSARIAL 2017 X X
/1611.02770.pdf • Chang Liu
EXAMPLES AND
• Dawn Song
BLACK-BOX ATTACKS

• Chaowei Xiao
• Bo Li
Generating Adversarial
https://arxiv.org/pdf • Jun-Yan Zhu
35 Examples with 2019 X X
/1801.02610.pdf • Warren He
Adversarial Networks
• Mingyan Liu
• Dawn Song

• Roman
Werpachowski
Detecting Overfitting via https://arxiv.org/pdf
36 2019 • András György X X
Adversarial Examples /1903.02380.pdf
• Csaba
Szepesvári

Cybersecurity https://www.enisa. • Georgia Dede

Challenges in the europa.eu/publicati • Ronan Hamon
37 Uptake of Artificial ons/enisa-jrc- 2021 • Rossen X X X
Intelligence in cybersecurity- Naydenov
Autonomous Driving challenges-in-the- • Henrik Junklewitz

46
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
uptake-of-artificial- • Apostolos
intelligence-in- Malatras
autonomous- • Ignacio Sanchez
driving
Rapid Object Detection https://www.cs.cm
using a Boosted u.edu/~efros/cours • Paul Viola
38 2001 X X
Cascade of Simple es/LBMV07/Paper • Michael Jones
Features s/viola-cvpr-01.pdf
Detection of Advanced
https://arxiv.org/ftp/ • Sanjay Sharma
Malware
39 arxiv/papers/1903/ 2019 • C. Rama Krishna X X X
by Machine Learning
1903.02966.pdf • Sanjay K. Sahay
Techniques
https://www.theseu
MACHINE LEARNING
s.fi/bitstream/handl
METHODS FOR
e/10024/123412/T • Kateryna
40 MALWARE 2017 X X X
hesis_final.pdf?seq Chumachenko
DETECTION AND
uence=1&isAllowe
CLASSIFICATION
d=y
• Bojan Kolosnjaji
Adversarial Malware • Ambra Demontis
Binaries: Evading Deep • Battista Biggio
https://arxiv.org/pdf
41 Learning for Malware 2018 • Davide Maiorca X X X
/1803.04173.pdf
Detection in • Giorgio Giacinto
Executables • Claudia Eckert
• Fabio Roli
Generating Adversarial
Malware Examples for https://arxiv.org/pdf • Weiwei Hu
42 2017 X X X
Black-Box Attacks /1702.05983.pdf • Ying Tan
Based on GAN
Exploring Adversarial • Octavian Suciu
https://arxiv.org/pdf
43 Examples in Malware 2018 • Scott E. Coull X X X
/1810.08280.pdf
Detection • Jeffrey Johns
• Felix Kreuk
Deceiving End-to-End • Assi Barak
Deep Learning Malware https://arxiv.org/pdf • Shir Aviv-Reuven
44 2019 X X X
Detectors using /1802.04528.pdf • Moran Baruch
Adversarial Examples • Benny Pinkas
• Joseph Keshet
•Edward Raff
•Jon Barker
Malware Detection by https://arxiv.org/pdf •Jared Sylvester
45 2017 X X X
Eating a Whole EXE /1710.09435.pdf •Robert Brandon
•Bryan Catanzaro
•Charles Nicholas
Black-box attacks
against rnn based https://arxiv.org/pdf •Weiwei Hu
46 2017 X X X
malware detection /1705.08131.pdf •Ying Tan
algorithms

Generic black-box end-

• Shai Rosenberg
to-end attack against
https://arxiv.org/pdf • Asaf Shabtai
47 rnns and other API calls 2018 X X X
/1707.05970.pdf • Lior Rokach
based malware
•Yuval Elovici
classifiers

47
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Raphael Labaca-
Castro
• Luis Muñoz-
González
Universal Adversarial •Feargus
https://arxiv.org/pdf
48 Perturbations for 2021 Pendlebury X X X
/2102.06747.pdf
Malware •Gabi Dreo
Rodosek
•Fabio Pierazzi
• Lorenzo
Cavallaro
MDEA: Malware
• Xiruo Wang
Detection with https://arxiv.org/pdf
49 2020 • Risto X X X
Evolutionary /2002.03331.pdf
Miikkulainen
Adversarial Learning

Attack and Defense of • Jack W. Stokes

Dynamic Analysis- • De Wang, Mady
https://arxiv.org/pdf
50 Based, Adversarial 2017 Marinescu X X X
/1712.05919.pdf
Neural Malware • Marc Marino
Classification Models • Brian Bussone

Robust Android
• Hemant Rathore
Malware Detection
https://arxiv.org/pdf • Sanjay K. Sahay
51 System against 2021 X X X
/2101.12031.pdf • Piyush Nikam
Adversarial Attacks
• Mohit Sewak
using Q-Learning

Binary Black-box • Mohammadreza

Evasion Attacks Ebrahimi
Against Deep Learning- • Ning
https://arxiv.org/pdf
52 based Static Malware 2020 Zhang, James Hu X X X
/2012.07994.pdf
Detectors with • Muhammad Taqi
Adversarial Byte-Level Raza
Language Model • Hsinchun Chen
MalFox: Camouflaged •Fangtian Zhong
Adversarial Malware • Xiuzhen Cheng
Example Generation https://arxiv.org/pdf • Dongxiao YuBei
53 2020 X X X
Based on C-GANs /2011.01509.pdf Gong
Against Black-Box • Shuaiwen Song
Detectors • Jiguo Yu
• Ishai Rosenberg
• Shai
Generating End-to-End
Meir, Jonathan
Adversarial Examples https://arxiv.org/pdf
54 2020 Berrebi X X X
for Malware Classifiers /2009.13243.pdf
• Ilay Gordon
Using Explainability
• Guillaume Sicard
• Eli (Omid)David
Adversarial EXEmples: • Luca Demetrio
A Survey and • Scott E. Coull
Experimental • Battista Biggio
https://arxiv.org/pdf
55 Evaluation of Practical 2020 • Giovanni Lagorio X X X
/2008.07125.pdf
Attacks on Machine • Alessandro
Learning for Windows Armando
Malware Detection • Fabio Roli
Adversarial Machine • Ihai Rosenberg
Learning Attacks and https://arxiv.org/pdf • Asaf Shabtai
56 2020 X X X
Defense Methods in the /2007.02407.pdf • Yuval Elovici
Cyber Security Domain • Lior Rokach

48
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Wei Song
Automatic Generation
• Xuezixiang Li
of Adversarial
https://arxiv.org/pdf • Sadia Afroz
57 Examples for 2020 X X X
/2003.03100.pdf • Deepali Garg
Interpreting Malware
• Dmitry Kuznetsov
Classifiers
• Heng Yin
• Aminollah
Khormali
COPYCAT: Practical
• Ahmed
Adversarial Attacks on https://arxiv.org/pdf
58 2019 Abusnaina X X X
Visualisation-Based /1909.09735.pdf
• Songqing Chen
Malware Detection
• DaeHun Nyang
• Aziz Mohaisen
Effectiveness of
• Robert
Adversarial Examples https://arxiv.org/pdf
59 2019 Podschwadt X X X
and Defenses for /1909.04778.pdf
• Hassan Takabi
Malware Classification
• Shashank Srikant
• Sijia Liu
• Tamara
Generating Adversarial
Mitrovska
Computer Programs https://arxiv.org/pdf
60 2021 • Shiyu Chang X X X
using Optimiser /2103.11882.pdf
• Quanfu Fan
Obfuscations
• Gaoyuan Zhang
• Una-May
O'Reilly
• Ecenaz Erdemir
Adversarial Robustness
https://arxiv.org/pdf • Jeffrey Bickford
61 with Non-uniform 2021 X X X
/2102.12002.pdf • Luca Melis
Perturbations
• Sergul Aydore

• Alexandre Araujo
Robust Neural https://hal.archives
• Laurent Meunier
Networks using -ouvertes.fr/hal-
62 2019 • Rafael Pinot X X
Randomiser 02380184v2/docu
• Benjamin
Adversarial Training ment
Negrevergne

•Rafael Pinot
• Laurent Meunier
Theoretical evidence • Alexandre Araujo
for adversarial https://arxiv.org/pdf • Hisashi Kashima
63 2019 X X
robustness through /1902.01148.pdf • Florian Yger
randomisation • Cédric Gouy-
Pailler
• Jamal Atif

• Cihang Xie
Mitigating Adversarial • Jianyu Wang
https://arxiv.org/pdf
64 Effects Through 2017 • Zhishuai Zhang X X
/1711.01991.pdf
Randomisation • Zhou Ren
• Alan Yuille

•Tong Chen
https://cybersecurit
Adversarial attack and • Jiqiang Liu
y.springeropen.co
defense in • Yingxiao Xiang
65 m/track/pdf/10.118 2019 X X X X X
reinforcement learning- • Wenjia Niu
6/s42400-019-
from AI security view • Endong Tong and
0027-x.pdf
Zhen Han

49
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
TextAttack: A Framework • John X. Morris
for Adversarial Attacks, • Eli Lifland
Data https://arxiv.org/pdf • Jin Yong Yoo
66 2020 X X
Augmentation, and /2005.05909.pdf • Jake Grigsby
Adversarial Training in • Di Jin
NLP • Yanjun Qi
• Anirban
Chakraborty
• Manaar Alam
Adversarial Attacks and https://arxiv.org/pdf • Vishal Dey
67 2018 X X
Defences: A Survey /1810.00069.pdf • Anupam
Chattopadhyay
• Debdeep
Mukhopadhyay
DATA • Ferhat Ozgur
AUGMENTATION Catak
BASED MALWARE https://arxiv.org/pdf • Javed Ahmed
68 2021 X X X
DETECTION USING /2010.01862.pdf • Kevser Sahinbas
CONVOLUTIONAL • Zahid Hussain
NEURAL NETWORKS Khand
https://gala.gre.ac. • Nikolaos
uk/id/eprint/25226/ Pitropakisa
7/25226%20LOUK • Emmanouil
A Taxonomy and
AS_Taxonomy_An Panaousisb
Survey of Attacks
69 d_Survey_Of_Atta 2019 • Thanassis X X
Against Machine
cks_Against_Mach Giannetsosc
Learning
ine_Learning_%28 • Eleftherios
AAM%29_2019.pd Anastasiadisd
f • George Loukase
• Andrew Ilyas
• Shibani Santurkar
Adversarial examples • Dimitris Tsipras
https://arxiv.org/pdf
70 are not bugs, they are 2019 • Logan Engstrom X X
/1905.02175.pdf
features. • Brandon Tran
• Aleksander
Madry
• Rémi Bernhard
• Pierre-Alain
Moellic
Impact of Spatial
• Martial Mermillod
Frequency Based https://arxiv.org/pdf
71 2021 • Yannick Bourrier X X
Constraints on /2104.12679.pdf
• Romain
Adversarial Robustness
Cohendet
• Miguel Solinas
• Marina Reyboz
• Vikash Sehwag
• Saeed
Mahloujifar
Improving Adversarial
https://arxiv.org/pdf • Tinashe Handina
72 Robustness Using 2021 X X
/2104.09425.pdf • Sihui Dai
Proxy Distributions
• Chong Xiang
• Mung Chiang
• Prateek Mittal
• Guoqiu Wang
Improving Adversarial
https://arxiv.org/pdf • Huanqian Yan
73 Transferability with 2021 X X
/2105.04834.pdf • Ying Guo
Gradient Refining
• Xingxing Wei

50
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Guiyu Tian
Poisoning MorphNet for
https://arxiv.org/pdf • Wenhao Jiang
74 Clean-Label Backdoor 2021 X X
/2105.04839.pdf • Wei Liu
Attack to Point Clouds
• Yadong Mu

• Siyue Wang
High-Robustness, Low-
• Xiao Wang
Transferability https://arxiv.org/pdf
75 2021 • Pin-Yu Chen X X
Fingerprinting of Neural /2105.07078.pdf
• Pu Zhao
Networks
• Xue Lin

Detecting Adversarial • Yao Li

Examples with https://arxiv.org/pdf • Tongyi Tang
76 2021 X X
Bayesian Neural /2105.08620.pdf • Cho-Jui Hsieh
Network • Thomas C. M. Lee

• Mingfu Xue
Robust Backdoor
• Can He
Attacks against Deep https://arxiv.org/pdf
77 2021 • Shichang Sun X X
Neural Networks in /2104.07395.pdf
• Jian Wang
Real Physical World
• Weiqiang Liu

secml-malware: A
Python Library for
https://arxiv.org/pdf • Luca Demetrio
78 Adversarial Robustness 2021 X X X
/2104.12848.pdf • Battista Biggio
Evaluation of Windows
Malware Classifiers

• Nicolas M. Müller
Defending against
• Simon
adversarial denial-of- https://arxiv.org/pdf
79 2021 Roschmann X X
service data poisoning /2104.06744.pdf
• Konstantin
attack
Böttinger
• Weiyi Zhang
• Shuning Zhao
Attack on practical
• Le Liu
speaker verification
https://arxiv.org/pdf • Jianmin Li
80 system using universal 2021 X X
/2105.09022.pdf • Xingliang Cheng
adversarial
• Thomas Fang
perturbations
Zheng
• Xiaolin Hu
Exploiting • Faiq Khalid •
Vulnerabilities in Deep Muhammad
https://arxiv.org/pdf
81 Neural Networks: 2021 Abdullah Hanif X X
/2105.03251.pdf
Adversarial and Fault- • Muhammad
Injection Attacks Shafique
Dynamic Defense
• Ruoxi Qin
Approach for Adversarial
https://arxiv.org/ftp/ • Linyuan Wang
Robustness in Deep
82 arxiv/papers/2105/ 2021 • Xingyuan Chen X X
Neural Networks via
2105.02803.pdf • Xuehui Du
Stochastic Ensemble
• Bin Yan
Smoothed Model
• Bangjie Yin
• Wenxuan Wang
Adv-Makeup: A New • Taiping Yao
Imperceptible and https://arxiv.org/pdf • Junfeng Guo
83 2021 X X
Transferable Attack on /2105.03162.pdf • Zelun Kong
Face Recognition • Shouhong Ding
• Jilin Li
• Cong Liu

51
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Micah Goldblum
Adversarial Attacks on
• Avi
Machine Learning https://arxiv.org/pdf
84 2020 Schwarzschild X X X
Systems for High- /2002.09565.pdf
• Ankit B. Patel
Frequency Trading
• Tom Goldstein

M. Hausknecht & Al.,

Deep Recurrent Q- • Matthew
https://arxiv.org/pdf
85 Learning for Partially 2017 Hausknecht X X X
/1507.06527.pdf
Observable MDPs, in • Peter Stone
arXiv, January 2017

• Hyrum S.
Anderson
Learning Malware • Hyrum S.
https://arxiv.org/pdf
86 Models via 2018 Anderson X X X
/1801.08917.pdf
Reinforcement Learnin • Bobby Filar
• David Evans
• Phil Roth
Wild patterns: ten years
after the rise of https://arxiv.org/pdf • Battista Biggioa
87 2018 X X
adversarial Machine /1712.03141.pdf • Fabio Rolia
Learning
https://www.resear
chgate.net/publicat • Qiang Liu
A survey on security ion/323154427_A_ • Pan Li
threats and defensive Survey_on_Securit • Wentao Zhao
88 techniques of machine y_Threats_and_De 2018 • Wei Cai X X X X X X X X X
learning: a data driven fensive_Technique • Shui Yu
view s_of_Machine_Lea • Victor C. M.
rning_A_Data_Driv Leung
en_View
• Han Xu
• Yao Ma
Adversarial attacks and
• Haochen Liu
defenses in images, https://arxiv.org/pdf
89 2020 • Debayan Deb X X X X x X X
graphs and text: a /1909.08072.pdf
• Hui Liu
review
• Jiliang Tang
• Anil K. Jain
https://www.resear
chgate.net/publicat • Daniel Lowd
90 Adversarial Learning 2005 X X
ion/221654486_Ad • Christopher Meek
versarial_learning
• Ling Huang
https://citeseerx.ist. • Anthony D.
psu.edu/viewdoc/d Joseph
Adversarial Machine-
91 ownload?doi=10.1. 2011 • Blaine Nelson X X
Learning
1.360.168&rep=re • Benjamin I. P.
p1&type=pdf Rubinstein
• J. D. Tygar
https://www.resear
chgate.net/publicat
A Survey of
ion/327074374_A_
Adversarial Machine
92 Survey_of_Advers 2018 • Vasisht Duddu X X X X X X X X X X
Learning in Cyber
arial_Machine_Lea
Warfare
rning_in_Cyber_W
arfare

52
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
Intelligence artificielle et https://www.wavest
cybersécurité : protéger one.com/app/uploa • Carole Meziat
93 2019 X X X X X
dès maintenant le ds/2019/09/IA- • Laurent Guille
monde de demain cyber-2019.pdf
• Ian J. Goodfellow
Explaining and
https://arxiv.org/pdf • Jonathon Shlens
94 Harnessing Adversarial 2020 X X
/1412.6572.pdf • Christian
Examples
Szegedy

https://papers.nips.
Feature Cross-
cc/paper/2014/file/ • Yevgeniy
Substitution in
95 8597a6cfa74defcb 2014 Vorobeychik X X
Adversarial
de3047c891d78f90 • Bo Li
Classification
-Paper.pdf

• Wenqi Wei
• Ling Liu
Adversarial Examples • Margaret Loper
in Deep Learning: http://arxiv.org/abs/ • Stacey Truex
96 2018 X X
Characterisation and 1807.00051 • Lei Yu
Divergence • Mehmet Emre
Gursoy
• Yanzhao Wu
Maximal Jacobian-
http://arxiv.org/abs/ • Rey Wiyatno
97 based Saliency Map 2018 X X
1808.07945 • Anqi Xu
Attack
Robustness and • Huan Xu
https://www.jmlr.or
Regularisation of • Constantine
98 g/papers/volume10 2009
Support Vector Caramanis
/xu09b/xu09b.pdf
Machines • Shie Mannor
• Xinyun Chen
Targeted Backdoor
• Chang Liu
Attacks on Deep https://arxiv.org/pdf
99 2017 • Bo Li X X
Learning Systems /1712.05526.pdf
• Kimberly Lu
Using Data Poisoning
• Dawn Song
Towards evaluating the
https://arxiv.org/ab • Nicholas Carlini
100 robustness of neural 2016 X X
s/1608.04644 • David Wagner
networks

• Reuben Feinman
• Ryan R. Curtin
Detecting adversarial https://arxiv.org/ab
101 2017 • Saurabh Shintre X X
samples from artifacts s/1703.00410
• Andrew B.
Gardner

Single Headed
Attention RNN: Stop https://arxiv.org/pdf
102 2019 • Steven Merity X X
Thinking With Your /1911.11423.pdf
Head
• Pauline Luc
Semantic Segmentation
https://arxiv.org/pdf • Camille Couprie
103 using Adversarial 2016 X X
/1611.08408.pdf • Soumith Chintala
Networks
• Jakob Verbeek
• Ming Pang
Unorganiser Malicious https://arxiv.org/pdf • Wei Gao
104 2018
Attacks Detection /1610.04086.pdf • Min Tao
• Zhi-Hua Zhou

53
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• CHRISTOPHE
ANDRIEU
https://link.springer • NANDO DE
An Introduction to
.com/content/pdf/1 FREITAS
105 MCMC for Machine 2003
0.1023/A:1020281 • ARNAUD
Learning
327116.pdf DOUCET
• MICHAEL I.
JORDAN

• Choon Hui Teo

http://people.csail.
• Amir Globerson
Convex Learning with mit.edu/gamir/pubs
106 2007 • Sam Roweis X X
Invariances /TeoGloRowSmo0
• Alexander J.
7.pdf
Smola

Scalable Optimisation
of Randomiser http://proceedings. • Bo Li
107 Operational Decisions mlr.press/v38/li15a 2015 • Yevgeniy X x
in Adversarial .pdf Vorobeychik
Classification Settings

https://www.scienc • Kui Ren

Adversarial Attacks
edirect.com/scienc • Tianhang Zheng
108 and Defenses in Deep 2020 X X X X X
e/article/pii/S20958 • Zhan Qin
Learning,
0991930503X • Xue Liu
Exploring the Space of https://arxiv.org/pdf • Pedro Tabacof
109 2016 X X
Adversarial Images /1510.05328.pdf • Eduardo Valle
• Nicolas Papernot
Distillation as a • Patrick McDaniel
Defense to Adversarial https://arxiv.org/pdf • Xi Wu
110 2016 X X
Perturbations against /1511.04508.pdf • Somesh Jha
Deep Neural Network • Ananthram
Swami
• Xingjun Ma
• Bo Li
• Yisen Wang
• Sarah M. Erfani
Characterizing
• Sudanthi
adversarial subspaces https://arxiv.org/pdf
111 2018 Wijewickrema X X
using local intrinsic /1801.02613.pdf
• Grant
dimensionality
Schoenebeck
• Dawn Song
• Michael E. Houle
• James Bailey
Defense still has a long
way: Adversarial
Examples Are Not https://arxiv.org/pdf • Nicholas Carlini
112 2017 X X
Easily Detected: /1705.07263.pdf • David Wagner
Bypassing Ten
Detection Methods
• Aleksander
Madry
Towards Deep
• Aleksandar
Learning Models https://arxiv.org/pdf
113 2019 Makelov X X
Resistant to Adversarial /1706.06083.pdf
• Ludwig Schmidt
Attacks
• Dimitris Tsipras
• Adrian Vladu

54
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
https://towardsdata
science.com/poiso
Poisoning attacks on
114 ning-attacks-on- 2019 • Ilja Moisejevs X
Machine Learning
Machine-Learning-
1ff247c254db

A Unified Framework • Xuanqing Liu

for Data Poisoning • Si Si
https://arxiv.org/pdf
115 Attack to Graph-based 2019 • Xiaojin Zhu X X X
/1910.14147.pdf
Semi-supervised • Yang Li
Learning • Cho-Jui Hsieh

• Mengchen Zhao
https://personal.ntu
Data Poisoning Attacks • Bo An
.edu.sg/boan/pape
116 on Multi-Task 2018 • Yaodong Yu X X X
rs/AAAI18_MTL.pd
Relationship Learning • Shulin Liu
f
• Sinno Jialin Pan

• Chang Liu
Robust High- • Bo Li
https://arxiv.org/pdf
117 Dimensional Linear 2016 • Yevgeniy X X
/1608.02257.pdf
Regression Vorobeychik
• Alina Oprea
• Jacob Steinhardt
Certified Defenses for https://arxiv.org/pdf
118 2017 • Pang Wei Koh X X X
Data Poisoning Attacks /1706.03691.pdf
• Percy Liang
• Ian J. Goodfellow
• Jean Pouget-
Abadie
• Mehdi Mirza
Generative adversarial https://arxiv.org/pdf • Bing Xu
119 2014 X X
nets /1406.2661.pdf • David Warde-
Farley
• Sherjil Ozair
• Aaron Courville
• Yoshua Bengio
• Qiang Liu
A Survey on Security • Pan Li
https://ieeexplore.i
Threats and Defensive • Wentao Zhao
eee.org/stamp/sta
120 Techniques of Machine 2018 • Wei Cai X X X X
mp.jsp?tp=&arnum
Learning: A Data • Shui Yu
ber=8290925
Driven View • Vctor C. M.
Leung
https://www.enisa.
europa.eu/publicati
Artificial Intelligence • Apostolos
ons/artificial-
121 Cybersecurity 2021 Malatras
intelligence-
Challenges • Georgia Dede
cybersecurity-
challenges

https://ieeexplore.i
A Taxonomy of ML for eee.org/stamp/sta
122 2020 • Martin Maas X X X X
Systems Problems mp.jsp?tp=&arnum
ber=9153088

55
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
Deep Unsupervised
Learning for • Arjun Kaushik
Generaliser • Mehrazin
https://arxiv.org/pdf
123 Assignment Problems: 2021 Alizadeh X X X
/2103.14548.pdf
A Case-Study of • Omer Waqar
User-Association in • Hina Tabassum
Wireless Networks
https://elie.net/blog
Attacks against
/ai/attacks-against-
124 machine learning — an 2018 • Elie Bursztein X
machine-learning-
overview
an-over
https://towardsdata
science.com/how-
How to attack Machine to-attack-machine-
Learning ( Evasion, learning-evasion-
125 2019 • Alex Polyakov X X
Poisoning, Inference, poisoning-
Trojans, Backdoors) inference-trojans-
backdoors-
a7cb5832595c

https://docs.lib.pur • Yingqi Liu

due.edu/cgi/viewco • Shiqing Ma
Trojaning Attack on
126 ntent.cgi?article=2 2017 • Yousra Aafer X X X X
Neural Networks
782&context=cstec • Wen-Chuang Lee
h • Juan Zhai

Evaluating Input
Perturbation Methods • Lukas Brunke
https://arxiv.org/pdf
127 for Interpreting CNNs 2021 • Prateek Agrawal X X
/2101.10977.pdf
and Saliency Map • Nikhil George
Comparison

Capsule Network is Not • Jindong Gu

https://arxiv.org/pdf
128 More Robust than 2021 • Volker Tresp X X
/2103.15459.pdf
Convolutional Network • Han Hu

MagFace: A Universal • Qiang Meng

Representation for https://arxiv.org/pdf • Shichao Zhao
129 2021 X X
Face Recognition and /2103.06627.pdf • Zhida Huang
Quality Assessment • Feng Zhou

Merge and Label: A

https://www.aclweb
novel neural network • Joseph Fisher
130 .org/anthology/P19 2019 X X
architecture for nested • Andreas Vlachos
-1585.pdf
NER
• Soroush Mehrir
• Kundan Kumarrr
SampleRNN: An • Ishaan Gulrajanir
Unconditional End-to- https://arxiv.org/pdf • Rithesh Kumarr
131 2017 X
End Neural Audio /1612.07837.pdf • Shubham Jainr
Generation Model • Jose Sotelor
• Aaron Courviller
• Yoshua Bengio

Generating Black Metal

and Math Rock: https://arxiv.org/pdf • Zack Zukowski
132 2018 X
Beyond Bach, /1811.06639.pdf • CJ Carr
Beethoven, and Beatles

56
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Haotong Qina
• Ruihao Gonga
Binary Neural https://arxiv.org/pdf • Xianglong Liu
133 2020 X X
Networks: A Survey /2004.03333.pdf • Xiao Baie
• Jingkuan Songc
• Nicu Sebed
Towards Accurate • Xiaofan Lin
https://arxiv.org/pdf
134 Binary Convolutional 2017 • Cong Zhao X X
/1711.11294.pdf
Neural Network • Wei Pan
• Yunus Saatchi
https://arxiv.org/pdf
135 Bayesian GAN 2017 • Andrew Gordon X X X X X
/1705.09558.pdf
Wilson

• Neda Tavakoli
A Comparison of
https://par.nsf.gov/ • Sima Siami-
ARIMA and LSTM in
136 servlets/purl/10186 2018 Namini X X
Forecasting Time
768 • Akbar Siami
Series
Namin

• John Schulman
•Filip Wolski
Proximal Policy https://arxiv.org/pdf
137 2017 •Prafulla Dhariwal, X
Optimisation Algorithms /1707.06347.pdf
•Alec Radford,
•Oleg Klimov
https://op.europa.e
u/en/publication-
detail/-
Ethics Guidelines for
138 /publication/d3988 2019
Trustworthy AI
569-0434-11ea-
8c1f-
01aa75ed71a1
• Battista
BiggioIgino
https://link.springer • Igino Corona
Evasion Attacks against .com/content/pdf/1 • Davide Maiorca
139 Machine Learning at 0.1007%2F978-3- 2017 • Blaine Nelson X X
Test time 642-40994- • Nedim Srndie
3_25.pdf • Pavel Laskov
• Giorgio Giacinto
• Fabio Roli
Robustness
Evaluations of
https://www.mdpi.c • Corey Dunn
Sustainable Machine
om/2071- • Nour Moustafa
140 Learning Models 2020 X X
1050/12/16/6434/h • Benjamin
against Data Poisoning
tm Turnbull
Attacks in the Internet
of Things
Defending network
https://www.scienc
intrusion detection • Marek Pawlicki
edirect.com/scienc
141 systems against 2020 • Michał Choraś
e/article/abs/pii/S0
adversarial evasion • Rafał Kozik
167739X20303368
attacks

Defending SVMs
against Poisoning • Hu Ding
https://arxiv.org/pdf
142 Attacks: the Hardness 2021 • Fan Yang X X
/2006.07757.pdf
and DBSCAN • Jiawei Huang
Approach

57
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Battista Biggio
Bagging classifiers for http://pralab.diee.u
• Igino Corona
fighting poisoning nica.it/sites/default/
143 2011 • Giorgio Fumera X X X
attacks in adversarial files/biggio11-
• Giorgio Giacinto
classification tasks mcs.pdf
• Fabio Roli

• Tom B. Brown
• Dandelion Mané
https://arxiv.org/pdf
144 Adversarial Patch 2018 • Aurko Roy X X
/1712.09665.pdf
• Martín Abadi
• Justin Gilmer

Model Inversion Attacks • Matt Fredrikson

https://dl.acm.org/d
that Exploit Confidence • Somesh Jha
145 oi/pdf/10.1145/281 2015 X X X
Information and Basic • Thomas
0103.2813677
Countermeasures Ristenpart

Membership Inference • Reza Shokri

Attacks against https://arxiv.org/pdf • Marco Stronati
146 2017 X X X X
Machine Learning /1610.05820.pdf • Congzheng Song
Models • Vitaly Shmatikov

STRIDE-AI: An
Approach to Identifying https://github.com/
• Lara Mauri
147 Vulnerabilities of LaraMauri/STRIDE 2021
• Ernesto Damiani
Machine Learning -AI
Assets

https://www.ai4eu.
eu/news/meaningf
ul-artificial-
For a meaningful
148 intelligencetowards 2018 • Cedric Villani
Artificial Intelligence
-french-artificial-
and-european-
strategy
https://www.ai4eu.
eu/news/strategic-
Strategic Action Plan action-plan-
149 2019
for Artificial Intelligence artificial-
intelligence-
netherlands
• Hamid Eghdal-
zadeh
• Khaled Koutini
• Paul Primus
• Verena
On Data Augmentation
https://arxiv.org/pdf Haunschmid
150 and Adversarial risk: An 2020 X X
/2007.02650.pdf • Michal
empirical Analysis
Lewandowski
• Werner Zellinger
• Bernhard
A.Moser
• Gerhard Widmer

https://ieeexplore.i
Review of Deep
eee.org/stamp/sta • Ajay Shrestha
151 Learning Algorithms 2019
mp.jsp?tp=&arnum • Ausif Mahmood
and Architectures
ber=8694781

58
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
Learning in a Large • Benjamin I. P.
Function Space: Rubinstein
https://arxiv.org/pdf
152 Privacy-Preserving 2009 • Peter L. Bartlett X
/0911.5708.pdf
Mechanisms for SVM •Ling Huang
Learning • Nina Taft

• Liang Tong
Improving Robustness • Bo Li
of ML Classifiers • Chen Hajaj
https://arxiv.org/pdf
153 against Realizable 2019 • Chaowei Xiao X X
/1708.08327.pdf
Evasion Attacks Using • Ning Zhang
Conserved Features • Yevgeniy
Vorobeychik
• Chaowei Xiao
• Jun-Yan Zhu
Spatially Transformed https://arxiv.org/pdf • Bo Li
154 2018 X X
Adversarial Examples /1801.02612.pdf • Warren He
• Mingyan Liu
• Dawn Song

• Arjun Nitin
Exploring the Space of Bhagoji
https://arxiv.org/pdf
155 Black-box Attacks on 2015 • Warren He X X
/1712.09491.pdf
Deep Neural Networks • Bo Li
• Dawn Song

• Kevin Eykholt
• Ivan Evtimov
• Earlence
Robust Physical-World Fernandes
Attacks on Deep https://arxiv.org/pdf • Bo Li
156 2018 X X
Learning Visual /1707.08945.pdf • Amir Rahmati
Classification • Chaowei Xiao
• Atul Prakash
• Tadayoshi Kohno
• Dawn Song
• PIERRE-
FRANÇCOIS
Hybrid Isolation Forest - MARTEAU
https://arxiv.org/pdf
157 Application to Intrusion 2017 • SAEID SOHEILY- X X X X X X
/1705.03800.pdf
Detection KHAH
• NICOLAS
BÉCHET

• Neda Tavakoli
The Performance of
https://par.nsf.gov/ • Sima Siami-
LSTM and BiLSTM in
158 servlets/purl/10186 2019 Namini
Forecasting Time
554 • Akbar Siami
Series
Namin

• Maurras Togbe
• Mariam Barry
Anomaly Detection for
https://hal.archives • Aliou Boly
Data Streams Based on
-ouvertes.fr/hal- • Yousra
159 Isolation 2020 X X X
02874869/docume Chabchoub
Forest using Scikit-
nt • Raja Chiky
multiflow
• Jacob Montiel
• Vinh-Thuy Tran

59
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Wei Wang
Robust Differentiable https://arxiv.org/pdf • Zheng Dang
160 2021
SVD /2104.03821.pdf • Yinlin Hu
• Pascal Fua
Accurate Stock Price
https://arxiv.org/ftp/
Forecasting Using • Jaydip Sen
161 arxiv/papers/2103/ 2021 X X
Robust and Optimiser • Sidra Mehtab
2103.15096.pdf
Deep Learning Models
• Christian
Vulnerabilities of Berghoff
https://arxiv.org/pdf
ConnectioNIST 800-53
162 /2003.08837.pdf 2020 • Matthias Neu X X X X X X
AI Applications:
Evaluation and Defence • Arndt Von
Twickel

• Zhijie Deng
LiBRe: A Practical • Xiao Yang
https://arxiv.org/pdf
163 Bayesian Approach to 2021 • Shizhen Xu X X
/2103.14835.pdf
Adversarial Detection • Hang Su
• Jun Zhu

• Christian
Berghoff,
• Battista Biggio
• Elisa Brummel
• Vasilios Danos
• Thomas Doms
• Heiko Ehrich
• Thorsten
Gantevoort
• Barbara Hammer
• Joachim Iden
https://www.bsi.bu
nd.de/SharedDocs/ • Sven Jacob
Downloads/EN/BSI • Heidy Khlaaf
Towards Auditable AI /KI/Towards_Audit • Lars Komrowski
164 able_AI_Systems. 2021
Systems • Robert Kröwing
pdf?__blob=public
ationFile&v=4 • Jan Hendrik
Metzen
• Matthias Neu
• Fabian Petsch
• Maximilian
Poretschkin
• Wojciech Samek
• Hendrik Schäbe
• Arndt von
Twickel
• Martin Vechev
• Thomas
Wiegand
• Ian J. Goodfellow
• David Warde-
https://arxiv.org/pdf Farley
165 Maxout network 2013 X X
/1302.4389.pdf • Mehdi Mirza
• Aaron Courville
• Yoshua Bengio

60
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Victor
Shepardson
A Taxonomy of ML https://berryvilleiml.
166 2019 • Gary McGraw
Attacks com/taxonomy/
• Harold Figueroa
• Richie Bonett

BadNets: Identifying • Tianyu Gu

Vulnerabilities in the https://arxiv.org/pdf • Brendan Dolan-
167 2019 X X
Machine Learning /1708.06733.pdf Gavitt
Model Supply Chain • Siddharth Garg

https://towardsdata
AI Security and
science.com/ai-
168 Adversarial Machine 2019 • Alex Polyakov
and-ml-security-
Learning 101
101-6af8026675ff
https://www.resear
• Indira Kalyan
chgate.net/publicat
Generative Adversarial Dutta
ion/344519514_Ge
169 Networks in Security: A 2020 • Bhaskar Ghosh
nerative_Adversari
Survey • Michael Totaro
al_Networks_in_S
• Albert H. Carlson
ecurity_A_Survey
https://www.bsi.bu
nd.de/SharedDocs/
Secure, Robust and Downloads/EN/BSI • Bundesamt für
170 transparent application /KI/Secure_robust_ 2021 Sicherheit in der
of AI and_transparent_a Informationstechnik
pplication_of_AI.pd
f
• Bryant Chen
• Wilka Carvalho
• Nathalie
Detecting Backdoor Baracaldo
http://ceur-
Attacks on Deep Neural • Heiko Ludwig
171 ws.org/Vol- 2018 X X X
Networks by Activation • Benjamin
2301/paper_18.pdf
Clustering Edwards
• Taesung Lee
• Ian Molloy
• Biplav Srivastava

Detection of adversarial • Andrea Paudice

training examples in • Luis Munoz-
https://arxiv.org/pdf
172 Poisoning Attacks 2018 Gonzalez X X X
/1802.03041.pdf
Trough Anomaly • Andras Gyorgy
Detection • Emil C.Lupu

https://blogs.scienti
ficamerican.com/o
When AI Misjudgement
173 bservations/when- 2018 • Douglas Yeung
is not an accident
ai-misjudgment-is-
not-an-accident/

https://www.media. • Samuel
mit.edu/publication G.Finlayson
Adversarial attacks on
s/adversarial- • Jonathan Zittrain
174 medical machine 2029
attacks •Joi Ito
learning
-on-medical- • Andrew L.Beam
machine-learning/ • Isaac S.Kohane

61
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
https://thenewstack
Camouflaged grafiti on
.io/camouflaged-
road signs can fool
175 graffiti-road-signs- 2017 • Kimberley Mok X X
machine learning
can-fool-machine-
models
learning-models/

https://www.idgcon
nect.com/article/35
Deepfakes and deep 83356/deepfakes-
fraud: the new security and-deep-fraud-
176 challenge the-new-security- 2020 • Sadia Sajiad
of misinformation and challenge-of-
impersonation misinformation-
and-
impersonation.html
Stealing
https://arxiv.org/pdf • B.Wang
177 Hyperparameters in 2019 X X X X
/1802.05351.pdf • N.Z.Gong
Machine Learning
Explanation-Guided • Giorgio Severi
https://www.usenix
Backdoor Poisoning • Jim Meyer
178 .org/system/files/se 2020 X X
Attacks Against • Scott Coull
c21-severi.pdf
Malware Classifiers • Alina Oprea
• Gamaleldin F.
Elsayed
Adversarial Examples • Shreya Shankar
that Fool both • Brian Cheung
//arxiv.org/pdf/1802
179 Computer 2018 • Nicolas Papernot X X
.08195.pd
Vision and Time- • Alex Kurakin
Limited Humans • Ian Goodfellow
• Jascha Sohl-
Dickstein
With Great training
https://www.usenix • Bolun Wang
comes great
.org/system/files/co • Bimal Viswanath
vulnerability:
180 nference/usenixse 2018 • Yuanshun Yao
Practical Attacks
curity18/sec18- • Haitao Zheng
against transfer
wang.pdf • Ben Y.Zhao
learning
• Nicolas Papernot
• Patrick McDaniel
Practical black-box • Ian Goodfellow
https://arxiv.org/pdf
181 attacks against 2017 • Somesh Jha X X
/1602.02697.pdf
machine learning • Z. Berkay Celik
• Ananthram
Swami

• Seyed-Mohsen
Moosavi-Dezfooli
Universal adversarial https://arxiv.org/pdf
182 2017 • Alhussein Fawzi X X
perturbations /1610.08401.pdf
• Omar Fawzi
• Pascal Frossard

Benchmarking neural
• Dan Hendrycks
network robustness to https://arxiv.org/pdf
183 2019 • Thomas X X
common corruptions /1903.12261.pdf
Dietterich
and perturbations

62
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
https://www.resear
chgate.net/publicat
ion/341792988_Se
Securing Artificial curing_Artificial_Int
Intelligence, Part 1 The elligence_Part_1_
184 attack surface of The_attack_surfac 2019 • Sven Herping
machine learning and e_of_machine_lear
its implications ning_and_its_impli
cations/link/5ed506
4a299bf1c67d3238
f4/download
Support vector http://proceedings. • Battista Biggio
185 machines under mlr.press/v20/biggi 2011 • Blaine Nelson X X X
adversarial label noise o11/biggio11.pdf • Pavel Laskov
• Avi
Just How Toxic is Data Schwarzschild
Poisoning? A Unified https://arxiv.org/pdf
• Micah Golddblum
186 Benchmark for /2006.12557.pdf 2020 X X
• Arjun Gupta
Backdoor and
• John P.Dickerson
Data Poisoning Attacks
• Tom Goldstein
• Huang Xiao
• Battista Biggio
Is feature selection
https://arxiv.org/pdf • Gavin Brown
187 secure against training 2018
/1804.07933.pdf • Giorgio Fumera
data poisoning?
• Claudia Eckert
• Fabio Roli

You Autocomplete Me: http://pages.cs.wis • Roei Schuster

Poisoning c.edu/~jerryzhu/ssl • Congzheng Song
188 2021 X X
Vulnerabilities in Neural /pub/Mei2015Mach • Eran Tromer
Code Completion ine.pdf • Vitaly Shmatikov

• Yujie Ji
• Xinyang Zhang
Model-reuse attacks on https://arxiv.org/pdf
189 2018 • Shouling Ji X X X X X
deep learning systems /1812.00483.pdf
• Xiapu Luo
• Ting Wang

https://ieeexplore.i
A survey on transfer • Sinno Jialin Pan
190 eee.org/document/ 2009
learning • Qiang Yang
5288526
• Ilia Shumailov
• Yiren Zhao
Sponge examples: https://arxiv.org/pdf
• Daniel Bates
191 Energy-Latency Attacks /2006.03463.pdf 2021 X X
• Nicolas Papernot
on Neural Network
• Robert Mullins
• Ross Anderson
• Jiongcong Chen
. Impact analysis of • Gaoqi Liang
https://link.springer
false data injection • Zexiang CAI
.com/content/pdf/1
192 attacks 2016 • Chunchao Hu
0.1007%2Fs40565
on power system static • Yan Xu
-016-0223-6.pdf
security assessment • Fengji Luo
• Junhua Zhao

63
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
https://www.resear
chgate.net/publicat
The ND2DB attack: ion/250195790_Th
Database content e_ND2DB_attack_ • Ariel Futoransky
193 extraction using timing Database_content 2007 • Damian Saura X
attacks on the indexing _extraction_using_ • Ariel Waissbein
algorithms timing_attacks_on
_the_indexing_alg
orithms
La nouvelle technologie
194 de protection des 2020 • Théo Ryffel
données

Reliable evaluation of
adversarial robustness https://arxiv.org/pdf
• Francesco Croce
195 with an ensemble of /2003.01690.pdf 2020 X X
diverse parameter-free • Matthias Hein
attacks

• M.Jagielski
Manipulating Machine • Alina Oprea
Learning: Poisoning • Battista Biggio
https://arxiv.org/pdf
196 attacks and 2021 • Chang Liu X X
/1804.00308.pdf
countermeasures for • Cristina Nita-
regression learning Rotaru
• Bo Li

Using Machine http://pages.cs.wis

teaching to identify c.edu/~jerryzhu/ma
• Shike Mei
197 Optimal training-set chineteaching/pub/ 2015 X X X
• Xiaojin Zhu
attacks on Machine Mei2015Machine.p
Learners df

• Yansong Gao
• Chang Xu
STRIP: A defence
• Derui Wang
against trojan attacks https://arxiv.org/pdf
198 2020 • Shiping Chen X X
on deep neural /1902.06531.pdf
• Damith C.
networks
Ranasinghe
•Surya Nepal
• Blaine Nelson
• Marco Barreno
• Fuching Jack Chi
https://people.eecs • Anthony
Misleading learners: .berkeley.edu/~tyg D.Joseph
199 Co-opting your spam ar/papers/SML/mis 2009 • Benjamin I.P X X
filter leading.learners.pd Rubinstein
f •Udam Saini
• Charles Sutton
• J.D Tygar
•Kai Xia

Cascade Adversarial • Taesik Na

Machine learning https://arxiv.org/pdf • Jong Hwan Ko
200 2018 X X
Regulariser with a /1708.02582.pdf • Saibal
unified embedding Mukhopadhyay

64
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Tong Chen
Gradient band-based • Wenjia Niu
adversarial training for • Yingxiao Xiang
https://arxiv.org/pdf
201 generaliser attack 2018 • Xiaoxuan Bai X X
/1807.06752.pdf
immunity of A3C path • Jiqiang Liu
finding. •Zhen Han
•Gang Li

Certifying Some • Adam Sinha

Distributional • Hongsoek
https://arxiv.org/pdf
202 Robustness 2020 Namkoong X X
/1710.10571.pdf
with Principled • Riccardo Volpi
Adversarial Training • John Duchi

https://www.jmlr.or • Kamalika
Differentially Private
g/papers/volume12 Chaudhuri
203 Empircal Risk 2011 X X
/chaudhuri11a/cha • Claire Monteleoni
Minimisation
udhuri11a.pdf • Anand D.Sarwate

•Elham Tabassi
•Kevin J. Burns
A Taxonomy and
https://csrc.nist.gov •Michael
Terminology of
204 /publications/detail/ 2019 Hadjimichael
Adversarial Machine
nistir/8269/draft •Andres D.
Learning
Molina-Markham
•Julian T. Sexton
Deep Defense: Training • Ziang Yan
https://arxiv.org/pdf
205 DNNs with improved 2018 • Yiwen Guo X X
/1803.00404.pdf
adversarial Robustness • Changshui Zhang
Improving the
Adversarial Robustness • Andrew Slavin
and Interpretability of https://arxiv.org/pdf Ross
206 2017 X X
Deep Neural Network /1711.09404.pdf • Finale Doshi-
by Regularizing their Velez
input Gradients

• Jan Hendrik
Metzen
On detecting https://arxiv.org/pdf
207 2017 • Tim Genewein X X
Adversarial perturbation /1702.04267.pdf
• Volker Fischer
• Bastian Bischoff

• Nicolas Papernot
https://ieeexplore.i
SoK: Security and • Patrick McDaniel
eee.org/stamp/sta
208 privacy in machine 2017 • Arunesh Sinha
mp.jsp?tp=&arnum
learning • Michael P.
ber=8406613
Wellman

• Qinglong Wang
Random Feature • Wenbo Guo
Nullification for https://arxiv.org/pdf • Kaixuan Zhang
209 2016 X X
Adversary Resistant /1610.01239v1.pdf • Xinyu Xing
Deep Architecture • C. Lee Giles
• Xue Liu

https://people.eecs
• Marco Barreno
.berkeley.edu/~adj/
The security of machine • Blaine Nelson
210 publications/paper- 2010 X X
learning • Anthony D.
files/SecML-
Joseph
MLJ2010.pdf

65
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
• Florian Tramer
• Alexey Kurakin
Ensemble adversarial
https://arxiv.org/pdf • Nicolas Papernot
211 training: attacks and 2018 X X
/1705.07204.pdf • Ian Goodfellow
defenses
• Dan Boneh
• Patrick McDaniel

Improving the • Stephen Zheng

robustness of Deep https://arxiv.org/pdf • Yang Song
212 2016 X X
Neural Networks via /1604.04326.pdf • Thomas Leung
Stability training • Ian Goodfellow

• Nicolas Papernot
Distillation as a defense • Patrick McDaniel
to adversarial https://arxiv.org/pdf • Xi Wu
213 2016 X X
perturbations against /1511.04508.pdf • Somesh Jha
deep neural networks • Ananthram
Swami
• Nicolas Papernot
• Shuang Song
• Ilya Mironov
Scalable Private https://arxiv.org/pdf
214 2016 • Ananth X X
Learning with Pate /1802.08908.pdf
Raghunathan
• Kunal Talwar
• Ulfar Erlingsson
https://seclab.stanf
Gradient masking in ord.edu/AdvML201
215 2017 • Nicolas Papernot
machine learning 7/slides/17-09-aro-
aml.pdf

The Quest for

https://hal.archive
Statistical Significance:
s-ouvertes.fr/hal-
216 Ignorance, Bias and 2018 •Joshua Abah
01758493/docume
Malpractice of
nt
Research Practitioners

The measure and

https://arxiv.org/pdf • Sam Corbett-
mismeasure of fairness
217 / 2018 Davies
A critical review of fair
1808.00023.pdf • Sharad Goel
machine learning
Adversarial Policy • Xian Wu
https://www.usenix
Training Against Deep • Wenbo Guo
218 .org/system/files/se 2021 X X X
Reinforcement • Hua Wei
c21-wu-xian.pdf
Learning • Xinyu Xing
https://link.springer
Early stopping - But .com/
219 2012 • Lutz Prechelt
When? book/10.1007%2F
978-3-642-35289-8
A pitfal and solution in https://icml.cc/Conf
multi-class feature erences
220 2004 • George Forman X X
selection for text /2004/proceedings/
classification papers/107.pdf

• Kamalika
Near-Optimal
Chaudhuri
Algorithms for https://arxiv.org/
221 2013 • Anand D. X X
Differentially-Private pdf/1207.2812.pdf
Sarwate
principal components
• Kaushik Sinha

66
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
Differentially private
model selection via http://proceeding • Adam Smith
222 Stability argument and s.mlr.press/v30/Gu 2013 • Abhradeep X
the robustness of the ha13.pdf Thakurta
Lasso
• Shehzeen
Hussain
WaveGuard:
https://www.usenix • Paarth Neekhara
Understanding and
223 .org/system/files/se 2021 • Shlomo Dubnov X X
Mitigating Audio
c21-hussain.pdf • Julian McAuley,
Adversarial Examples
• Farinaz
Koushanfar

http://researchers.li
lle.inria.fr/abellet/te
Privacy Preserving
224 aching/private_ma 2020 •Aurélien Bellet
Machine Learning
chine_learning_co
urse.html

Interpretability of https://www.wavest
Machine Learning one.com/app/uploa
What are the ds/2019/09/Wavest •Alexandre Vérine
225 2019
challenges in the era of one_Interpretability •Stephan Mir
automated decision- _machine_learning
Making Progresses? .pdf
https://www.analyti
6 Python Libraries to csvidhya.com/blog/
interpret Machine 2020/03/6-python-
226 2020 •Purva Huilgol
Learning Models and libraries-interpret-
Build Trust machine-learning-
models/
• Cristoph Molnar
• Gunnar König
• Julia Herbinger
• Timo Freiesleben
• Susanne Dandl
Pitfalls to avoid when https://arxiv.org/pdf
• Christian
227 Interpreting Machine / 2020
A.Scholbeck
Learning Models 2007.04131.pdf
• Giuseppe
Casalicchio
• Moritz Grosse-
Wentrup
• Bernd Bischl
https://towardsdata
science.com/7-
7 steps to ensure and steps-to-ensure-
228 2019 • Stéphanie Shen
sustain Data quality and-sustain-data-
quality-
3c0040591366

67
 SECURING MACHINE LEARNING ALGORITHMS
December 2021

ed Learning

ent learning
Reinforcem
Unsupervis
Supervised
Learning
Type of data
ingested

Publication date
Index

Title Source Author

Structured Data

Classification
Time series

Regression

Dimension

Rewarding
Clustering

Reduction
Image
Video

Text
https://techcrunch.
com/2018/11/06/3-
ways-to-avoid-
bias-in-machine-
learning/?guccount
er=1&guce_referre
r=aHR0cHM6Ly93
d3cuZ29vZ2xlLmN
vbS8&guce_referr
er_sig=AQAAAIfXh
Three ways to avoid VIDfTYv80Vxw4JV
229 bias in machine aKFZyt_3_2DTapB 2018 • Vince Lynch
learning QQjW8C1vzjPTQq
ViKdAE5O-
BV1Q5J5waGMcY
o4yu2R4QBOr9H1
7RpApdX9vlXDUlo
_MS28Q4GD8qCX
qhogX534lJcR7DP
rzwANTY8WPnJX
5GXVmytlHxM0ZK
9Ym2ANzKGSnae
6QgH

https://www.kdnug
5 Ways to deal with the gets.com/2019/06/
•Alexandre
230 lack of data in Machine 5-ways-lack-d 2019
Gonfalonieri
Learning ata-machine-
learning.html

https://towardsdata
science.com/artifici
Artificial Intelligence is al-intellige
Crucial TO the Success nce-is-crucial-to-
231 2019 • Amit Makhija
of Your Business and the-success-of-
here is why your-business-
learn-why-
d5b96fa3564d
https://www.kaggle
7 Simple Techniques to •Devendra Kumar
232 .com/learn- 2020
Prevent Overfitting Yadav
forum/157623

68
 TP-06-21-153-EN-N
ABOUT ENIS A
The European Union Agency for Cybersecurity, ENISA, is the Union’s agency dedicated to
achieving a high common level of cybersecurity across Europe. Established in 2004 and
strengthened by the EU Cybersecurity Act, the European Union Agency for Cybersecurity
contributes to EU cyber policy, enhances the trustworthiness of ICT products, services and
processes with cybersecurity certification schemes, cooperates with Member States and EU
bodies, and helps Europe prepare for the cyber challenges of tomorrow. Through knowledge
sharing, capacity building and awareness raising, the Agency works together with its key
stakeholders to strengthen trust in the connected economy, to boost resilience of the Union’s
infrastructure, and, ultimately, to keep Europe’s society and citizens digitally secure. More
information about ENISA and its work can be found here: www.enisa.europa.eu.

ISBN: 978-92-9204-543-2
DOI: 10.2824/874249