Journal of Artificial Intelligence Research 1 (Submitted) Submitted 11/21; published

A Framework for Fairness: A Systematic Review of Existing

Fair AI Solutions

Brianna Richardson [email protected]

Juan E. Gilbert [email protected]
University of Florida
Gainesville, FL 32601 USA
arXiv:2112.05700v1 [cs.AI] 10 Dec 2021

Abstract
In a world of daily emerging scientific inquisition and discovery, the prolific launch of
machine learning across industries comes to little surprise for those familiar with the po-
tential of ML. Neither so should the congruent expansion of ethics-focused research that
emerged as a response to issues of bias and unfairness that stemmed from those very same
applications. Fairness research, which focuses on techniques to combat algorithmic bias,
is now more supported than ever before. A large portion of fairness research has gone to
producing tools that machine learning practitioners can use to audit for bias while design-
ing their algorithms. Nonetheless, there is a lack of application of these fairness solutions
in practice. This systematic review provides an in-depth summary of the algorithmic bias
issues that have been defined and the fairness solution space that has been proposed.
Moreover, this review provides an in-depth breakdown of the caveats to the solution space
that have arisen since their release and a taxonomy of needs that have been proposed by
machine learning practitioners, fairness researchers, and institutional stakeholders. These
needs have been organized and addressed to the parties most influential to their implemen-
tation, which includes fairness researchers, organizations that produce ML algorithms, and
the machine learning practitioners themselves. These findings can be used in the future to
bridge the gap between practitioners and fairness experts and inform the creation of usable
fair ML toolkits.

1. Introduction
Today, applications of machine learning (ML) and artificial intelligence (AI) can be found in
nearly every domain (Jordan & Mitchell, 2015): from medicine and healthcare (Goldenberg
et al., 2019; Chang et al., 2018; Lin et al., 2020; Munavalli et al., 2020) to banking and
finance (Sunikka et al., 2011; Choi & Lee, 2018; Moysan & Zeitoun, 2019). At a rate that has
grown exponentially over the last decade, companies are seizing the opportunity to automate
and perfect procedures. In the field of healthcare, machine learning is being used to diagnose
and treat prostate cancer (Goldenberg et al., 2019), perform robotic surgeries (Chang et al.,
2018), organize and schedule patients (Munavalli et al., 2020), and digitize electronic health
record data (Lin et al., 2020). Within banking and finance, machine learning is being used
to personalize recommendations for consumers (Sunikka et al., 2011), detect and prevent
instances of fraud (Choi & Lee, 2018), and provide faster and personalized services through
the use of chatbots (Moysan & Zeitoun, 2019).
While machine learning is praised for its ability to speed up time-consuming processes,
automate mundane procedures, and improve accuracy and performance of tasks, it is also
praised for its ability to remain neutral and void of human bias. This is what Sandvig

©Submitted AI Access Foundation. All rights reserved.

 Richardson & Gilbert

(2014) calls the ‘neutrality fallacy’, which is the common misconception that AI does not
perpetuate the trends of its data which is often clouded with human biases. For this reason,
machine learning continues to undergo heavy scrutiny for the role it plays in furthering
social inequities. Over the past few years, major companies have been headlined for their
AI technologies that cause harms against consumers. Buolamwini and Gebru (2018) assessed
commercially-used facial recognition systems and found that darker-skinned women were the
most misclassified demographic group; Noble (2018) found that search engines perpetuated
stereotypes and contributed substantially to representational harms; and Angwin et al.
(2016) found biases in automated recidivism scores given to criminal offenders.
Circumstances such as these have led to a surplus of contributions to the solution
space for algorithmic bias by stakeholders and researchers, alike (Zhong, 2018). Institu-
tions have been quick to formulate their own ethics codes centered around concepts like
fairness, transparency, accountability, and trust (Jobin et al., 2019). New conferences, like
ACM’s Fairness, Accountability, and Transparency (FAccT) conference1 , and new work-
shops have emerged centered on these concepts. Furthermore, in most machine learning
conferences, new tracks have been added that focus on algorithmic bias research. This
surplus of recognition by stakeholders and computer science researchers has encouraged a
commensurate rise in related contributions to improve the ethical concerns surrounding AI.
One major objective for responsible AI researchers is the creation of fairness tools that
ML practitioners across domains can use in their application of responsible and ethical
AI. These tools translate top-tier research from the responsible AI space into actionable
procedures or functions that practitioners can implement into their pipelines. Despite the
number of currently existing tools, there is still a lack of application of ethical AI found in
industry. Recent research suggests a disconnect between the fairness AI researchers creating
these tools and the ML practitioners who are meant to apply them (Holstein et al., 2019;
Law et al., 2020b; Madaio et al., 2020; Law et al., 2020a; Veale & Binns, 2017). Greene
et al. (2019) emphasizes that the solution to this problem lies in the intersection of technical
and design expertise. Responsible and fair AI must undergo deliberate design procedures to
match the needs of practitioners and satisfy the technical dilemmas found in bias research.
This survey paper will highlight users’ expectations when engaging with fair AI tooling,
and it will summarize both fairness concerns and design flaws discussed in literature. First,
Section 2 focuses on algorithmic bias and the major entry points for bias that perpetuate
the need for fairness research and fairness tools. Section 3 focuses on the solutions that
have been posed thus far by fairness researchers. It will highlight major contributions and
the features offered by them. Section 4 will focus on how these solutions are working in
practice. It will highlight many of the drawbacks and provide recommendations for fairness
experts, organizations, and ML practitioners. This paper is the first of its kind to provide
a comprehensive review of the solution space of fair AI.

2. The Problem Space: Algorithmic Bias

The first recorded case of algorithmic bias was the discrimination suit that was filed against
St. George’s Hospital Medical School in the 1980’s (Lowry & MacPherson, 1988). Leaders of
the admission program had decided to create an algorithm that could mimic the admission
1. ACM FAccT Conference - https://facctconference.org/

2
 A Framework for Fairness

Figure 1: A taxonomy of the currently existing forms of bias.

process by completing the first screening of applicants. Upon completion of the algorithm,
they validated its performance by comparing its results to manually generated decisions and
found a 90-95% similarity. After a few years in circulation, staff began to notice trends in
admission and brought their concerns to the attention of the school’s internal review board
(IRB). When the IRB agreed that the correlation between machine scores and human
scores delegitimized claims of bias, those claims were taken to the U.K. Commission for
Racial Equity. After thorough analysis of the algorithm, the Commission confirmed these
claims were true: the algorithm was placing value on applicant’s names and place of birth,
penalizing individual’s with ‘Non-Caucasian’ sounding names (Lowry & MacPherson, 1988).
The ruling of this committee was pivotal to the start of algorithmic bias research since
it set a precedent that the inclusion of bias in a system, even for the sake of accuracy, was
impermissible. Nonetheless, for the past 40 years, this issue has been recurring. Biases
continue to emerge in the creation and use of machine learning, legitimizing unfair and
biased practices across a multitude of industries (Lum & Isaac, 2016). In the often-cited 2017
presentation at NeurIPs, Crawford (2017) discusses two potential harms from algorithmic
bias: representational and allocative harms. Representational harms are problems that
might arise from the troublesome ways certain populations are represented in the feature
space, and allocative harms are problems that arise from how decisions are allocated to
certain populations (Crawford, 2013). Literature has demonstrated that bias can arise in
all shapes and forms, and the task for fairness experts is to organize and confront those
biases.

3
 Richardson & Gilbert

Friedman and Nissenbaum (1996) separates types of biases into three categories: pre-
existing, technical and emergent biases. They define pre-existing biases as those that are
rooted in institutions, practices, and attitudes. Technical biases are those that stem from
technical constraints or issues stemming from the technical design of the algorithm. Lastly,
emergent biases arise in the context of real use by the user (Friedman & Nissenbaum, 1996).
This section will adopt this categorization to classify influences of bias found in literature.
A visual depiction of the forms of biases that will be explained can be seen in Figure 1.

2.0.1 Pre-existing bias

A large portion of literature attention has gone to pre-existing biases. These are biases
that are associated with an individual or institutions. When human preferences or societal
stereotypes influence the data and/or the model, that is said to be the effect of pre-existing
bias. The saying, ‘garbage in, garbage out’ is well-known in the data science community as
a euphemism for the quality of data you give a system will be the quality of your results.
Fairness researchers translate this to mean that machine learning models serve as a feedback
loop, reflecting the biases it has been fed (Barocas et al., 2019). Bias can come from an
assortium of stages in the machine learning pipeline.
The most prominently discussed form of bias is historical bias, or biases that are per-
petuated in data and are the result of issues that existed at the time (Veale & Binns, 2017;
Calders & Žliobaitė, 2013; Olteanu et al., 2019; Barocas et al., 2019; Rovatsos et al., 2019;
Suresh & Guttag, 2019; Guszcza, 2018; Hellström et al., 2020). Suresh and Guttag (2019)
defines historical biases as those that arise out of the misalignment between the world and
its values and what the model perpetuates. The impact of historical bias can be reflected in
what data was collected, how the data was collected, the quality of the data, and even the
labels given to the samples (Suresh & Guttag, 2019; Rovatsos et al., 2019; Barocas & Selbst,
2018; Calders & Žliobaitė, 2013; Hellström et al., 2020). The subjectivity of labels is a ma-
jor concern because it reproduces and normalizes the intentional or unintentional biases of
the original labeler (Barocas & Selbst, 2018; Kasy & Abebe, 2021). From analyzing a large
collection of ML publications, Geiger et al. (2019) found a dire need for more normalized
and rigorous standards for evaluating datasets and data collection strategies prior to any
machine learning processing, aligning with the recommendations from Crawford (2013).
Besides the historical biases, there also exists data collection biases, or biases that stem
from the selection methods used for data sources (Olteanu et al., 2019). Data collection
bias umbrellas a variety of different biases recognized in literature, including: sampling bias
and representation bias. Sampling bias consists of issues that arise from the non-random
sampling of subgroups that might lead to the under- or over-sampling of certain populations
(Mehrabi et al., 2021; Hellström et al., 2020). Historically, vulnerable populations have often
been undersampled (Veale & Binns, 2017). Mester (2017) discusses two types of sampling
bias: selection bias and self-selection bias. While selection bias focuses on the biases of the
data collector, self-selection bias stems from which individuals were available and willing
to participate in the data collection process (Mester, 2017). Concerns about sampling bias
stem from the fact that inferences made from mis-balanced dataset will inevitably lead to
the disadvantage of populations who were missampled (Asaro, 2019).

4
 A Framework for Fairness

Representation bias, also called data bias (Olteanu et al., 2019), is another form of
data collection bias and consists of bias that arises from the insufficient representation
of subgroups within the dataset (Crawford, 2017; Suresh & Guttag, 2019). This mis-
representation can stem from the subjective selection of unfitting attributes or the incom-
plete collection of data (Calders & Žliobaitė, 2013). Veale and Binns (2017) describes this
issue in detail, stating that subgroups can contain nuanced patterns that are difficult (or
impossible) to quantify in a dataset, resulting in models that misrepresent those popula-
tions.
Another prominent form of pre-existing bias is data processing bias, or bias that is
introduced by data processing procedures (Olteanu et al., 2019). The selection of methods
across the ML pipeline can be incredibly subjective and many of these steps can introduce
unintended biases into the model. For example, the use of sensitive attributes within models
is a well-known taboo in data science. Government regulation has even gone as far as
prohibiting organizations access to protected data (Veale & Binns, 2017). Nonetheless,
the use of proxies, or attributes that encode sensitive information, is prolific (Calders &
Žliobaitė, 2013; Corbett-Davies & Goel, 2018). Besides the use of sensitive data, other
processing steps could incorporate bias, including the categorization of data (Veale & Binns,
2017), feature selection and feature engineering (Veale & Binns, 2017; Barocas & Selbst,
2018), or even how data is evaluated (Suresh & Guttag, 2019). Evaluation bias can occur
from the use of performance metrics that are ill-fitting for the given model (Suresh & Guttag,
2019).
The final category of pre-existing bias is systematic bias, or bias that stems from insti-
tutional or governmental regulations and procedures. While regulations limiting access to
sensitive data can be beneficial for limiting use of sensitive data, they can also prevent the
identification and de-biasing of proxy attributes leading to problematic models (Veale &
Binns, 2017). Funding can also produce bias by manipulating the priorities of institutions
and, therefore, practitioners (Mester, 2017).

2.0.2 Technical bias

While pre-existing biases consider what the model is adopting from the data, technical
biases consider the limitations of computers and how technology insufficiencies might create
bias (Friedman & Nissenbaum, 1996). Baeza-Yates (2018) considers this ‘algorithmic bias’
and defines it as bias that was not present in the data but was added by the algorithm.
Overlapping on pre-existing bias, technical bias can also stem from processing procedures.
Often, the selection of features, models, or training procedures can introduce bias that is not
affiliated with the practitioner but with the insufficiency of those procedures to characterize
the data. For example, hyperparameter tuning may, in an attempt to reduce the sparsity
of the model, end up removing distinct patterns found in sub-populations (Veale & Binns,
2017; Hellström et al., 2020). Furthermore, certain models, such as regression, fail to capture
the correlation between attributes and subgroups (Veale & Binns, 2017; Skitka et al., 1999).
Computer tools bias is another form of technical bias that originates from the limita-
tions of statistics and technology (Friedman & Nissenbaum, 1996). Simpson’s paradox is
a statistical issue that arises from the aggregation of distinct subgroup data that produces
misperformance for one or all subgroups (Blyth, 1972; Suresh & Guttag, 2019).

5
 Richardson & Gilbert

Lastly, Friedman and Nissenbaum (1996) defines formulation of human constructs bias
as that which stems from the inability of technology to properly grasp human constructs.
In situations such as this, fairness experts often insist that practitioners or organizations
reconsider the use of machine learning (Rakova et al., 2020). Selbst et al. (2019) describes
the effects of this bias as the ripple effect trap, where they describe how the use of technology
in areas where human constructs bias exists can potentially change human values.

2.0.3 Emerging bias

This final category of bias, referred to by Suresh and Guttag (2019) as deployment bias, is
defined as biases that result from the deployment of a model. Emerging bias can emerge
in two different forms, via population bias and use bias. Population bias stems from the
insufficiency of a model to represent its population or society after deployment (Olteanu
et al., 2019). This can stem from new knowledge that has emerged and made a model
irrelevant (Friedman & Nissenbaum, 1996) or a mismatch between the sample population
used to train the model and the population who is impacted by its deployment (Friedman
& Nissenbaum, 1996; Calders & Žliobaitė, 2013; Rovatsos et al., 2019). Selbst et al. (2019)
refers to this failure to consider how a model works in context as the “portability trap”.
Use bias umbrellas a variety of different ways in which the use of models can create
biases. In a study conducted by Skitka et al. (1999), participants interacted with an au-
tonomous aid which provided recommendations for a simulated flight task. The results
depicted trends of commission where a user agreed with a recommendation even when it
contradicted their training (Skitka et al., 1999). The neutrality fallacy introduced by Sand-
vig (2014) is a major pitfall to AI application because it forces users to believe the tool is
correct, even when it should be obvious that it is not (Sandvig, 2014). Skitka et al. (1999)
suggests that users would be less willing to challenge a decision if that decision was made
with an algorithm.
A related form of use bias is presentation bias, or bias that stems from how a model
is deployed (Baeza-Yates, 2018). In these situations, users make assumptions about the
presentation format of algorithms and may come to incorrect conclusions. For example, in
a search algorithm, many users attribute the rank of the result to the relevancy, which may
not be accurate in all cases (Baeza-Yates, 2018).
Belief bias is also a form of use bias that occurs when someone is so sure of themselves
they ignore the results (Mester, 2017). This can also impact when they choose to use
the results of a model and when they choose to ignore them (Veale et al., 2018), particu-
larly when the model’s decision counteracts their own beliefs. Veale et al. (2018) discusses
decision-support design that can be used to prevent this selectional belief bias.
Gamification is another use bias in which a person learns how to manipulate an algorithm
(Veale et al., 2018). Ciampaglia et al. (2017) introduces the concept of popularity bias, or
preferences that strongly correlate to popularity, and discusses how these forms of bias are
often gamified on the internet by bots and fake feedback.
The final case of use bias is the curse of knowledge bias: when you assume someone
has the background you do and can use the model appropriately (Mester, 2017). This con-
cern appears in various literature surrounding predictive policing technologies (Asaro, 2019;

6
 A Framework for Fairness

Bennett Moses & Chan, 2018; Ridgeway, 2013) where critics voice concerns about whether
police understand the limitations of these technologies and are using them appropriately.
Guszcza (2018) details the importance of understanding the environment where these
machine learning tools will be used and creating with those environments in mind.

3. The Solution Space: Fairness Technologies

The solution space to algorithmic bias consists of a diverse array of solutions that contribute
to the responsible AI space, including explainability, transparency, interpretability, and
accountability (Cheng et al., 2021). While each solution has diverged from a different
ethical dilemma that emerged from the machine learning production pipeline, the objective
remains the same: to produce fair AI, or AI that is free from unintentional algorithmic bias.
Fairness, as defined by Mehrabi et al. (2021), is

“[T]he absence of any prejudice or favoritism toward an individual or a group

based on their inherent or acquired characteristics.”

Fair AI, for the purpose of this paper, consists of solutions to combat algorithmic bias, which
is often inclusive of top-tier solutions from explainability, transparency, interpretability, and
accountability research.
Since the problem space of algorithmic bias is so large and diverse, a concerted effort has
gone into making fairness tools that practitioners can use to employ state-of-the art fairness
techniques within their ML pipelines. These solutions mainly come in two forms: software
toolkits and checklists. Toolkits serve as functions accessible via programming languages
that can be used to detect or mitigate biases. Checklists are extensive guides created by
fairness experts that ML/AI practitioners can use to ensure the inclusion of ethical thought
throughout their pipelines. While this section won’t provide a complete description of the
solution space, it will highlight the diversity of tools provided by both organizations and
academic institutions.

3.1 Software Toolkits

A variety of software toolkits currently exist, each with overlapping and distinct character-
istics to identify them. While some are accessible via website portals (Saleiro et al., 2018),
many exist as installable packages that can be imported into Python programs (Bird et al.,
2020; Srinivasan, 2020; Johnson et al., 2020; Wexler et al., 2019) or other languages used
by ML practitioners (Bellamy et al., 2018; Vasudevan & Kenthapadi, 2020). The following
selection of toolkits provides a diverse look of available solutions.

3.1.1 Google’s Toolkits

Google provides two relevant toolkits: Fairness Indicators 2 and the What-If toolkit (WIT)
(Wexler et al., 2019). Both toolkits function as interactive widgets where users can provide
their model(s), their performance and fairness metrics of choice, and the attributes on which
slicing and evaluating will take place (Richardson et al., 2021). These toolkits are embedded
2. Tensorflow’s Fairness Evaluation and Visualization Toolkit - https://github.com/tensorflow/fairness-
indicators

7
 Richardson & Gilbert

within the Tensorflow package in Python, but users can build their own custom prediction
functions if their model exists outside of Tensorflow. Unlike other toolkits, Google’s toolkits
require that users provide a model (Seng et al., 2021). Additionally, both toolkits allow
model comparison between, at most, 2 models. While Fairness Indicators works with bi-
nary and multiclass problems and allows intersectional analysis, WIT works with binary
or regression problems. Within Fairness Indicators, users can create their own visualiza-
tions to compare models across different performance and fairness metrics. Users can select
and deselect performance metrics to focus on. The WIT includes a features overview, a
performance chart, and a data point editor. The features overview provides visualizations
to depict the distribution of each feature with some summary statistics. The performance
chart provides simulated results depicting the outcome of select fairness transformations,
including an option to customize a fairness solution. Lastly, the data point editor provides
custom visualizations of data points from the data and allows users to compare counterfac-
tual points, make modifications to points to see how the results change, and plot partial
dependence plots to determine model’s sensitivity to a feature.

3.1.2 UChicago’s Aequitas

Aequitas is an audit report toolkit that can be accessed via command line, as a Python pack-
age, and a web application (Saleiro et al., 2018). Aequitas is a tool built for classification
models and allows for the comparison of models. While Aequitas does not provide fairness
mitigation techniques, it does evaluate models based on commonly used fairness criteria.
Users can label sensitive attributes and Aequitas will calculate group metrics between sen-
sitive subgroups and provide disparity scores that reflect the degree of unfairness. Aequitas
also produces plots that record these fairness disparities and group metrics. Within these
visualizations, users can choose to have groups colored based on whether or not subgroup
disparities pass a set threshold for “fair” or “unfair” (Saleiro et al., 2018). This tool was
built with two types of users in mind: data scientists/AI researchers who are building these
AI tools and policymakers who are approving their use in practice (Saleiro et al., 2018).

3.1.3 IBM’s AI Fairness 360

AI Fairness 360 (AIF360) is a toolkit that provides both fairness detection and mitigation
strategies (Bellamy et al., 2018). It can be used in both Python and R. The fairness metrics
provided include general performance metrics, group fairness metrics, and individual fairness
metrics. Users also have an assortment of mitigation strategies that they can apply to their
models. These mitigation strategies can be applied across a variety of stages in the pipeline,
including: pre-processing, while processing, and post-processing. In order to apply these
strategies, users can distinguish between privileged and unprivileged subgroups on which
to complete analysis. To assist users with learning, their website includes a wide selection
of tutorials and web demos (Bellamy et al., 2018). While this toolkit does not provide its
own visualization functions, it does provide guidance for how its functions can be used in
conjunction with the Local Interpretable Model-Agnostic Explanations toolkit, aka LIME
(Ribeiro et al., 2016). Similar to Aequitas, AIF360 was built with business users and ML
developers, in mind (Bellamy et al., 2018).

8
 A Framework for Fairness

3.1.4 LinkedIn’s Fairness Toolkit (LiFT)

LiFT is a recently released toolkit that provides fairness detection strategies for measuring
fairness across a variety of metrics (Vasudevan & Kenthapadi, 2020). They provide an
assortment of metrics, segregated based on whether they are for the data or the model
outputs. Similar to AI Fairness 360, they do not produce their own visualizations. Unlike
previous toolkits, they are built to be applied in Scala/Spark programs. Furthermore, this
toolkit is unique in its flexibility and scalability, overcoming previous issues with measuring
fairness in large datasets (Vasudevan & Kenthapadi, 2020).

3.1.5 Other Solutions

An assortment of other toolkits exist, including Microsoft’s Fairlearn (Bird et al., 2020),
ML Fairness Gym (Srinivasan, 2020), Scikit’s fairness tool (Johnson et al., 2020), PyMetrics
Audit-AI 3 , and new ones arise often. While these toolkits may differ in the fairness metrics
they consider and the fairness mitigation strategies they may employ, the highlights of their
features can be seen in the toolkits above: some are interactive, some provide visual support,
some allow intersectional analysis, some focus on detection or mitigation alone, etc.

3.2 Checklist
Unlike toolkits, checklists mostly exist as documentation meant to guide readers through
incorporating ethical and responsible AI throughout the lifecycle of their projects. While
some are specifically meant for data scientists or machine learning engineers building the
tools, other checklists are generally written with all relevant parties in mind. While toolkits
provide more statistical support when it comes to strategies to detect and mitigate, check-
lists engage developers with questions and tasks to effectively ensure that ethical thought
occurs from the idea formulation to the auditing after deployment (Patil et al., 2018).

3.2.1 Deon
Deon is a fairness checklist created by DrivenData, a data scientist-led company that uti-
lizes crowdsourcing and cutting-edge technology to tackle predictive problems with societal
impact 4 . The checklist is described as a “default” toolkit, but customizations are made for
practitioners with domain-specific concerns. Deon is split into five areas: data collection,
data storage, analysis, modeling, and deployment. To assist users with utilizing the check-
list, each checklist item is accompanied with use cases to exemplify relevant concerns. One
of Deon’s most unique features is its command line interface that practitioners can use to
interact with the checklist.

3.2.2 Microsoft’s AI Fairness Checklist

Madaio et al. (2020) produced the AI Fairness Checklist. Unlike most checklists, this check-
list was co-designed with the iterative feedback of practitioners. The checklist is split into 6
different parts: Envision, Define, Prototype, Build, Launch, and Evolve. Envision, Define,
and Prototype provide ethical considerations that would fit best in the initial planning and
3. PyMetric’s Audit-AI - https://github.com/pymetrics/audit-ai
4. Driven Data’s Deon tool - https://deon.drivendata.org/

9
 Richardson & Gilbert

designing stages of a project; Build provides guidance for when AI is in the creation phases;
and Launch and Evolve provide guidance for the deployment stages of the product. It also
comes with a comprehensive Preamble that introduces the complexity of fairness, provides
instructions for how the checklist should be used, and encourages practitioners and teams to
personalize and customize the checklist to best fit their environment (Madaio et al., 2020).

3.2.3 Legal and Ethics Checklist

Lifshitz and McMaster (2020) provides a unique ethics checklist that focuses on legal consid-
erations that should be noted in the creation of AI. Unlike related checklists, this checklist is
not sectioned by stages in the AI lifecycle, but by legal priorities, including: human agency
& oversight, security & safety, privacy & data governance, transparency, accessibility, etc.
This checklist provides a unique viewpoint from a lawyer for institutions and practitioners
to use to avoid running into legal issues (Lifshitz & McMaster, 2020).

3.2.4 IBM’s AI FactSheets

AI FactSheets is a unique guide that provides a methodology for incorporating responsibility
and transparency into the AI pipeline via documentation. While AI FactSheets is not de-
scribed as a checklist, it does provide guidance for how to create documentation that depicts
responsible AI practices. The creation of a FactSheet template is very domain-dependent,
and Arnold et al. (2019) provides a detailed methodology for how a FactSheet can be cus-
tomized to the project. In satisfying the components of the FactSheets, institutions and
teams can engender trust with consumers by increasing transparency and ensuring their
products satisfy necessary ethical considerations (Arnold et al., 2019).
These checklists provide an overview of the differing features that can be found across
the checklist landscape: some focus on legality, some are domain specific, and others have
been built with the AI pipeline in mind. Other checklists and guides have been produced
as well (Bradley et al., 2020; Manders-Huits & Zimmer, 2009; Gebru et al., 2018; Mitchell
et al., 2019), similarly structured and intended to assist practitioners with implementing
fair or responsible practices in their pipelines.

4. Shortcomings of Solutions & Recommendations for the Future

While the solutions posed above have been a pivotal starting point for fairness research,
much work has been done addressing the pitfalls of these contributions, as well. This
section aims to discuss the prominent issues that have arisen in the literature and the
corresponding recommendations that have been made for future work. While many of
these recommendations were made to fairness researchers and designers, there also are
recommendations for organizations and ML practitioners.

4.1 Recommendations for Fairness Experts

Towards the goal of universally ethical AI, much of the responsibility lies in the hands of
fair AI researchers to define, design, and translate fairness into a palatable procedure that
can be easily applied by practitioners and institutions. This requires that the process of
producing fair AI undergo human-centered research and design procedure. Nonetheless,

10
 A Framework for Fairness

previous research has found that there exist major gaps between practitioners and fairness
experts, indicating a lack of communication between those producing fair AI and those
intended to apply it (Holstein et al., 2019; Law et al., 2020b; Madaio et al., 2020; Law
et al., 2020a; Veale & Binns, 2017). The results from these works have been thematically
categorized, accompanied by suggestions for improvement.

4.1.1 Conflicting Fairness Metrics

Along with identifying 23 different types of biases and 6 different categories for discrimina-
tion, Mehrabi et al. (2021) identified 10 different types of fairness metrics. Due to the vast
quantity and similarity between these metrics, many authors have tried to categorize them
into related bins. Mehrabi et al. (2021) identified three different types of fairness metrics:
individual, group, and subgroup. Individual fairness gives similar predictions to similar
individuals, group fairness treats different groups equally, and subgroup fairness attempts
to achieve a balance of both group and individual fairness (Mehrabi et al., 2021).
Barocas et al. (2019) also identified 19 proposed fairness metrics and also categorized
them into three categories: independence, separation, and sufficiency. Barocas et al. (2019)
defines these categories through properties of the joint distribution of sensitive attribute
A, the target variable Y, and the classifier or Score R. A random variable (A, R) satisfies
independence if A ⊥ R; a random variable (R, A, Y) satisfies separation if R ⊥ A | Y ; and
a random variable (R, A, Y) satisfies sufficiency if Y ⊥ A | R (Barocas et al., 2019).
Lastly, in a comprehensive review and explanation of fairness metrics, Verma and Rubin
(2018) isolated 20 different fairness metrics and categorized them into 3 different categories:
statistical measures, similarity measures, and causal reasoning. Statistical measures are
those that depend on true positive, false positive, false negative, and true negative; simi-
larity based measures attempt to address issues that are ignored by statistical measures by
focusing on insensitive attributes as well; and causal reasoning uses causal graphs to draw
relations between attributes to determine their influence on the outcome and allow users
to understand where exactly bias is coming from and whether it is permissible (Verma &
Rubin, 2018).
While each author recognized legitimate patterns within and between metrics, these
works highlight a major issue that exists: there are too many metrics for measuring unfair-
ness with too few differences to delineate them (Mehrabi et al., 2021). Furthermore, major
trade-offs exist between fairness metrics, making the selection of metrics highly situation-
dependent (Binns, 2018). Verma and Rubin (2018) concludes that many of these metrics
are advanced and require expert-level input, which in itself produces implicit biases.
Furthermore, many works have identified conflicts between metrics that make them
incompatible with each other, which forces individuals to choose (Berk et al., 2021; Friedler
et al., 2021; Kleinberg et al., 2017; Mittelstadt, 2019; Rovatsos et al., 2019). Different
metrics may emphasize different aspects of performance (Japkowicz & Shah, 2011) and much
work has been done comparing and contrasting these metrics (Garcia-Gathright et al., 2018;
Verma & Rubin, 2018; Chouldechova, 2017; Corbett-Davies & Goel, 2018; Binns, 2018; Fish
et al., 2016; Kilbertus et al., 2017; Simoiu et al., 2017; Cramer et al., 2019; Zliobaite, 2015).
This decision is not one that should be taken lightly because the act of choosing a fairness
metric is very political in that it valorizes one point of view while silencing another (Bowker

11
 Richardson & Gilbert

& Star, 1999; Friedler et al., 2021). According to Rovatsos et al. (2019), the task of choosing
the right metric currently lies in the hands of practitioners, most of whom are unfamiliar
with fairness research, and that is a heavy expectation considering the fact that society as
a whole has not decided which ethical standards to prioritize.
To support this issue, Verma and Rubin (2018) suggests that more work is needed to
clarify which definitions are appropriate for which situations. Friedler et al. (2021) states
that fairness experts must explicitly state the priorities of each fairness metric to ensure
practitioners are making informed choices. Furthermore, Friedler et al. (2018) emphasizes
that new measures of fairness should only be introduced if that metric behaves fundamen-
tally differently from those already proposed.

4.1.2 Other Metric-specific Pitfalls

In addition to the conflicting nature of fairness metrics, there exist several other pitfalls
that have been recognized by researchers. Kilbertus et al. (2017) discussed the insufficiency
of ”observational criteria”, which are the criteria often used as sensitive attributes within
toolkits. They emphasize that such methodologies are unable to confirm that protected
attributes have a direct causal influence on results and instead, they propose two new
metrics of causal reasoning: proxy discrimination and unresolved discrimination (Kilbertus
et al., 2017). Proxy discrimination can be potentially inferred from a causal graph when
there exists a path from a protected attribute to a predicted attribute that includes a proxy
variable, and unresolved discrimination can be inferred from a causal graph if the only path
from a protected attribute to the predicted attribute includes a resolving variable (Kilbertus
et al., 2017).
Furthermore, Friedler et al. (2018) found that many fairness metrics lack robustness.
By simply modifying dataset composition and changing train-test splits, the results of their
study depicted that many fairness criteria lacked stability. They proposed that only mea-
sures that depict stability and success should be used to report fairness.
Hoffmann (2019) discusses the lack of intersectional analysis in fairness criteria. Many
metrics provide analysis for sensitive attributes, but only one-dimensionally. Hoffmann
(2019) emphasizes that proposed metrics should strategically identify multi-dimensional
correlations between attributes and outcomes.
Wagstaff (2012) emphasizes that the use of abstract metrics distract from the problems
specific to datasets and applications. Furthermore, the impact of the metrics cannot be
inferred from the scores, themselves. Furthermore, Kasy and Abebe (2021) demonstrates
that many of the standard metrics legitimize the focus on merit, deterring individuals from
questioning the legitimacy of the status quo.
Additionally, there is a concern about the assumptions made by fairness experts when
producing fairness metrics. Many metrics and available fairness tooling depend on access to
sensitive attributes, which many practitioners do not have (Holstein et al., 2019; Law et al.,
2020a; Rovatsos et al., 2019). Additionally, there are legal restrictions against some fairness
definitions (Xiang & Raji, 2019). For those with (legal) access to sensitive data, some prac-
titioners are concerned with how the public would scrutinize the use of sensitive attributes,
even for the detection of bias (Rovatsos et al., 2019). Furthermore, when studying political
philosophy and how it connects to fairness in machine learning, Binns (2018) notes that

12
 A Framework for Fairness

risks of incomplete fairness analysis arise when users are forced to adhere to a static set of
prescribed protected classes, instead of doing thorough analysis to identify discrimination.
To combat this issue, Law et al. (2020a) suggests that coarse-grained demographic info be
utilized in fairness tools and proposed to practitioners. Furthermore, fairness experts should
utilize and promote fairness metrics that do not rely on sensitive attributes (Holstein et al.,
2019; Law et al., 2020a; Rovatsos et al., 2019).

4.1.3 Oversimplification of Fairness

A major concern in literature is the emphasis on technical solutions to algorithmic bias,
which is a socio-technical problem. Many authors emphasize the need to supplement sta-
tistical definitions with social practices (Veale et al., 2018; Madaio et al., 2020; Verma &
Rubin, 2018; Fazelpour & Lipton, 2020; Jacobs & Wallach, 2021; Birhane, 2021). Madaio
et al. (2020) called the sole use of technical solutions, “ethics washing,” and Selbst et al.
(2019) describes the failure to account for the fact that fairness cannot be solely achieved
through mathematical formulation as the “formalism trap”. Fazelpour and Lipton (2020)
details that the practice of ethics washing can lead to misguided strategies for mitigating
bias, and Harcourt (2007) emphasizes that the perceived success of these technical solutions
stalls pursuits to achieve actual fairness with the aid of social practices. Birhane (2021) calls
for a fundamental shift towards considering the relational factors and impact of machine
learning from a societal standpoint.
Fazelpour and Lipton (2020) recommends that mathematical assessment be supple-
mented with social assessment tools: like a thorough analysis of data collection strategies
and understanding the assumptions made by different ML algorithms. Furthermore, toolkits
and checklists for fair evaluation should avoid solely providing mathematical solutions.

4.1.4 Misguided fairness objectives

In addition to the issue of ethics washing, other concerns have arisen concerning the tra-
jectory of fairness research. Hoffmann (2019) critiques fairness research for its focus on
avoiding disadvantage instead of understanding where advantage stems from. In support
of better understanding where biases exist in the data, Veale and Binns (2017) proposes
that more energy should be placed in data exploration, through the use of unsupervised
learning as a strategy to identify hidden patterns. Futhermore, many fairness metrics and
strategies rely on assumptions about the world that may be problematic in themselves. For
example, Hu and Kohler-Hausmann (2020) discusses how the use of the social concept of
sex can perpetuate problematic and harmful sex discrimination.

4.1.5 Making Fair AI Applicable

Works, such as those done by Holstein et al. (2019), Veale et al. (2018), Rakova et al. (2020),
Richardson et al. (2021), are novel in their inclusion of practitioner feedback into fairness
literature. A common theme that emerged from these papers was the lack of applicability
that most practitioners held toward fairness tools and support. Practitioners interviewed
by Holstein et al. (2019) emphasized the need for domain-specific procedures and metrics.
These participants requested that fairness experts pool knowledge and resources by domain
for easy access. Additionally, practitioners in this study and Veale et al. (2018) had concerns

13
 Richardson & Gilbert

with the scalability of fairness analysis, which has been backed by data scientists in Verma
and Rubin (2018). Practitioners interviewed by Rakova et al. (2020) felt that metrics in
academic-based research had vastly different objectives than metrics in industry, which
required that industry researchers do the additional work to translate their work using
insufficient metrics. To supplement this workload, practitioners felt that fairness research
by academia did not fit smoothly into industry pipelines and required a significant amount
of energy to fully implement (Rakova et al., 2020).
In a study by Richardson et al. (2021), practitioners had the opportunity to interact with
fairness tools and provide comments and feedback. Authors summarized key themes from
practitioners regarding features and design considerations that would make these tools more
applicable. The following lists contains some of the features requested for fairness toolkits:

• Applicable to a diverse range of predictive tasks, model types, and data types (Holstein
et al., 2019)

• Can detect & mitigate bias (Holstein et al., 2019; Olteanu et al., 2019; Mehrabi et al.,
2021)

• Can intervene at different stages of the ML/AI life cycle (Bellamy et al., 2018; Holstein
et al., 2019; Veale & Binns, 2017)

• Fairness and performance criteria agnostic (Corbett-Davies & Goel, 2018; Barocas
et al., 2019; Verma & Rubin, 2018)

• Diverse explanation types (Ribeiro et al., 2016; Dodge et al., 2019; Arya et al., 2019;
Binns et al., 2018)

• Provides recommendations for next steps (Holstein et al., 2019)

• Well-supported with demos and tutorials (Holstein et al., 2019)

The results from these studies collecting practitioner feedback supplement what many
data scientists and social scientists have confirmed. In a study conducted by Corbett-Davies
et al. (2017), when utilizing fairness metrics in a recidivism use case, results depicted that
some metrics had significant trade-offs with public safety. This exemplified the importance
of domain-specific guides and details when it comes to fairness procedures. In similar
landscape summaries of algorithmic bias, Rovatsos et al. (2019), Reisman et al. (2018) also
discuss how concerns of algorithmic bias differ substantially by domain application, and yet
most domains lack thorough and specific algorithmic bias guidance. Lastly, when studying
the use of fair AI checklists, Madaio et al. (2020), Cramer et al. (2018) emphasized the
importance of aligning these checklists with team workflows.

4.1.6 Designing usable Fair AI

Concerns with applicability emphasize an over-arching importance of human-centered design
procedures in the creation of fair AI tools. Efforts should be made to collect practitioner
feedback and incorporate this feedback into the creation of fair AI (Richardson et al., 2021;
Holstein et al., 2019; Rakova et al., 2020).

14
 A Framework for Fairness

A major component of usability when it comes to fair AI tools is the integration of

affordances that support AI fairness. According to Robert et al. (2020), these affordances
include: transparency, explainability, voice, and visualization. In this work, they define
transparency as “making the underlying AI mechanics visible and known to the employee”,
explainability as “describing the AI’s decision/actions to the employee in human terms”,
voice as “providing employees with an opportunity to communicate and provide feedback to
the AI”, and visualization as “representing information to employees via images, diagrams,
or animations” (Robert et al., 2020, p. 26). This work is not the only one of its kind to
discuss the importance of such features in strengthening fairness tools.
Results from studies that interviewed or studied practitioners emphasized the impor-
tance of having the ability to interact with fairness toolkits (Richardson et al., 2021; Holstein
et al., 2019; Cramer et al., 2019), using the tool to compare models and methods (Richard-
son et al., 2021), providing understandable visualizations (Richardson et al., 2021; Veale
et al., 2018; Law et al., 2020a; Ribeiro et al., 2016; Dodge et al., 2019; Arya et al., 2019; Law
et al., 2020b; Veale & Binns, 2017), and receiving feedback from these tools (Ribeiro et al.,
2016; Richardson et al., 2021). Solutions to aid users in understanding bias issues reside
in interpretability and explainability research. Hutchinson and Mitchell (2018) emphasizes
the importance of utilizing explanations in fairness pursuits. In a study by Ribeiro et al.
(2016), participants had the opportunity to interact with LIME, the Local Interpretable
Model-Agnostic Explanations toolkit. Results from this study depict the importance of
explanations as a tool for users to decide whether or not they should trust a classifier or
determine where and how to fix a classifier (Ribeiro et al., 2016). Without concrete expla-
nations, users were either willfully ignorant or unable to rely on the classifier (Ribeiro et al.,
2016).
Furthermore, a study by Dodge et al. (2019) found that different fairness problems
were better explained with different types of explanations. When users were exposed to
different types of fairness explanations, they exhibited very different opinions on the model.
While some explanations were considered inherently less fair meaning users tended to view
their models as biased, others enhanced participants’ confidence in the classifier (Dodge
et al., 2019). Authors in Arya et al. (2019) did a similar study when introducing the
AI Explainability 360 toolkit. They compared and contrasted different explainer types,
including data, directly interpretable, local post-hoc, and global post-hoc explainers. They
provide a taxonomy of explanations that provides guidance for which explainers are best
for which users and future work that needs to be done (Arya et al., 2019).
Law et al. (2020b) ran a simulation with practitioners to depict how a plethora of
visualizations could be used without overloading the participants. Depending on the use
case, they proposed different techniques for organizing visualizations. One technique, which
they referred to as a recommendation list, provides a summary of fairness results and was
best for situations where there were many performance metrics. The other technique, visual
cues, provided high comprehensiveness for a few metrics (Law et al., 2020b).
Despite the indubitable strengths of fairness toolkits, Lakkaraju and Bastani (2019)
and Kaur et al. (2020) also gave some risks that came with the use of visualizations and
explanations. Lakkaraju and Bastani (2019) found that explanations for a black-box model
could still be manipulated to hide issues of unfairness by simply omitting features that
users considered problematic. By doing this, generated explanations increased user trust

15
 Richardson & Gilbert

by nearly 10 fold (Lakkaraju & Bastani, 2019). Furthermore, Kaur et al. (2020) depicted
the importance of training users before providing them with toolkits and visualizations
that they are unfamiliar with. When users were not internalizing tutorials, they made
incorrect assumptions about the data and the model, they could not successfully uncover
issues with the dataset, and had low confidence when interacting with the tool. Nonetheless,
they reported high trust in the fairness tools because it provided visualizations and it was
publicly available (Kaur et al., 2020). These findings should be considered in the creation
and the deployment of these toolkits.

4.1.7 Avoid the over-reliance on Fair AI

As was depicted in the work of Kaur et al. (2020), many practitioners overly trusted toolkits
despite not completely understanding the results of the toolkit. Of particular concern, when
users were asked to select from a dropdown list the functionalities of the toolkit, most of
them grossly overestimated what the toolkit could do. These results depicted a misalignment
between practitioners’ understanding of the toolkits and the toolkits’ intended use (Kaur
et al., 2020). Practitioners interviewed in Veale et al. (2018) depicted similar hesitations
about interacting with fairness tooling: they suspected they would over or under-rely on
them.

4.1.8 Communicating fairness to stakeholders

Across several interview studies where researchers collected practitioner feedback, prac-
titioners requested help communicating fairness concerns to stakeholders (Holstein et al.,
2019; Veale et al., 2018; Law et al., 2020a). Some practitioners discussed how the objectives
of stakeholders were different from their own and requested that resources be provided for
communicating the fairness trade-off as it relates to those objectives (Veale et al., 2018;
Law et al., 2020a). They also requested that these resources be understandable by laymen,
or business-oriented individuals, so that they can understand the cause of bias and the im-
portance of fairness considerations (Law et al., 2020a; Garcia-Gathright et al., 2018). One
practitioner said that fairness toolkits should incorporate the assumption that there will be
tech resistance and many stakeholders and practitioners will reject the output from fairness
toolkits if its not effectively communicated (Veale et al., 2018). While prior discussions
on different explanations for differing users could most definitely be useful in explaining
fairness issues (Arya et al., 2019), fairness experts should consider creating guides for com-
municating fairness results.

4.1.9 The burden of fairness

A common sentiment when it came to discussing or utilizing fairness tools was practitioners
feeling overwhelmed. Whether users were interacting with a toolkit (Law et al., 2020b;
Richardson et al., 2021) or a checklist (Cramer et al., 2019), many felt that either they were
not qualified (Holstein et al., 2019), ethics was too difficult to operationalize (Madaio et al.,
2020), ethics presented more questions than answers (Binns, 2018), or the checklists and
tools were too overwhelming (Cramer et al., 2019; Richardson et al., 2021; Law et al., 2020b).
While some of these concerns require support from institutions, many works discussed what
fairness experts could do to relieve this burden.

16
 A Framework for Fairness

Since there exists so many avenues for bias and so many feasible solutions (Mehrabi
et al., 2021; Olteanu et al., 2019), Cramer et al. (2018) suggests that fairness experts
create taxonomies of existing biases that can help practitioners recognize and counter these
biases. To overcome the information overload that might accompany a visual toolkit (Law
et al., 2020b), Law et al. (2020a), Richardson et al. (2021) suggests that toolkits filter
out biases on the users’ behalf so that alarming outcomes can stand out, but this solution
presents selection biases of the fairness developers and important outcomes may differ in
different contexts. Results from Cramer et al. (2019) suggests that checklist be avoided
as a ‘self-serve’ tool, since they were often found to be overwhelming for users. Instead,
an interactive interface might be beneficial (Cramer et al., 2019). The goal for fairness
practitioners should be to incorporate fairness into pre-existing workflows in such a way
that disruption and chaos are minimized. With this objective in mind, fairness would be
considered substantially less overwhelming for practitioners.

4.1.10 Supplemental Resources for Practitioners

The major challenge presented to fairness experts is translating principles and ethics codes
into actionable items that practitioners and institutions can implement (Mittelstadt, 2019).
This includes providing guidance for users to detect and foresee bias (Madaio et al., 2020;
Law et al., 2020a; Garcia-Gathright et al., 2018; Mehrabi et al., 2021; Cramer et al.,
2018), determine where bias might stem from (Holstein et al., 2019; Garcia-Gathright et al.,
2018), determine how to respond to biases (Holstein et al., 2019; Veale et al., 2018; Law
et al., 2020a; Garcia-Gathright et al., 2018), and how to maintain fairness after deploy-
ment (Veale et al., 2018). Practitioners are requesting taxonomies of potential harms and
biases (Madaio et al., 2020; Cramer et al., 2018), easy-to-digest summaries explaining bi-
ases (Garcia-Gathright et al., 2018; Cramer et al., 2018), and guidelines for best practices
throughout the ML pipeline (Holstein et al., 2019). Users are also requesting a plethora of
additional resources like community forums for fairness, domain-specific guides, and plenty
of tutorials exemplifying how to incorporate fairness (Holstein et al., 2019; Madaio et al.,
2020; Richardson et al., 2021; Gray & Chivukula, 2019). Fairness experts should do as
much as possible to provide these resources for practitioners to support their use of fairness
tools.

4.2 Recommendations for Institutions

While a considerable portion of recommendations were given to fairness researchers, or-
ganizations also have a large portion of responsibility for ensuring the success of fairness
implementation. Several works provide critique and actionable steps for institutions and
organizations to use to assist their practitioners in the implementation of fairness toolkits.

4.2.1 Prioritize Fairness

With the plethora of institutions who have created ethical principles or guidelines centered
on fairness, accountability, and transparency in AI, there is no doubt that organizations
have recognized the importance in prioritizing ethics (Jobin et al., 2019). Nonetheless, the
application of fairness in practice is rare and most ethical codes are not put to action by
engineers and practitioners. Frankel (1989) emphasizes that ethics codes that fail to put val-

17
 Richardson & Gilbert

ues into practice are political tools meant to manipulate the public. One of the easiest ways
to promote a culture of bias awareness is to, at an organizational level, prioritize fairness as
a global objective (Garcia-Gathright et al., 2018; Stark & Hoffmann, 2019). Organizations
can do this by considering fairness in their own hiring practices (Garcia-Gathright et al.,
2018), prioritizing bias correction as they prioritize privacy and accessibility (Madaio et al.,
2020; Garcia-Gathright et al., 2018; Frankel, 1989), providing practitioners with resources or
teams that can provide them with actionable steps (Madaio et al., 2020; Mittelstadt, 2019),
and changing organizational structure to embrace the inclusion of fairness considerations
(Cramer et al., 2019).
Practitioners interviewed in Rakova et al. (2020) provided a plethora of recommenda-
tions for organizations intending to prioritize fairness. Organizations could provide reward
systems to incentivize practitioners to continue educating themselves on ethical AI. Further-
more, they could allow practitioners to work more closely with marginalized communities to
ensure their data is representative and their products are not harmful to them. Lastly, prac-
titioners emphasized the importance of rejecting an AI system if it is found to perpetuate
harms to individuals (Rakova et al., 2020).
Reisman et al. (2018) recommended that organizations practice transparency with the
public as a method for ensuring accountability and, therefore, fairness. They suggest that
the public be given notice of any technology that might impact their lives (Reisman et al.,
2018). Fairness research also emphasizes the importance of public transparency from in-
stitutions when it comes to the existence and the auditing of their AI (Richardson et al.,
2020).

4.2.2 Operationalize Fairness

One method to exhibit a prioritization of fairness is the operationalization of fairness. As

they exist now, most published ethics codes are difficult to operationalize due to their
abstract nature (Madaio et al., 2020; Stark & Hoffmann, 2019). By operationalizing, orga-
nizations force themselves to consider how they want to enact fairness (Robert et al., 2020;
Frankel, 1989) and relieve the burden of ethical responsibility placed on the practitioner
(Stark & Hoffmann, 2019) by making the big decision on things like: what type of demo-
graphics, biases, and stakeholders they intend to consider in their efforts (Cramer et al.,
2018). This will require discussions by top-players within organizations in deciding how the
prioritization of fairness may displace other priorities, and it requires the consultation of
legal-consultants who can ensure new policies satisfy government regulations (Veale et al.,
2018; Mittelstadt, 2019). Furthermore, leaders must make the decision of how much bias
is permissible and be prepared to handle the consequences of these decisions (Barocas &
Selbst, 2018; Barocas et al., 2019).

4.2.3 Avoid dividing practitioner loyalty

According to recent research, many practitioners feel divided by the desire to create ethically-
sound products and the obligation to their role as employees (Stark & Hoffmann, 2019;
Mittelstadt, 2019). Some felt as though conversations of ethics were taboo and could im-
pact their goals of career advancement (Madaio et al., 2020; Rakova et al., 2020). Many

18
 A Framework for Fairness

felt like it was an individual battle, where they were the sole advocate ‘battling’ for fairness
(Holstein et al., 2019; Madaio et al., 2020).
When studying ethics in engineering, it becomes obvious that these concerns are nothing
new to the field of engineering. Works by Frankel (1989) and Davis (1991) discuss the ethical
dilemma that has impacted engineers for decades. When studying the Code of Ethics taught
to engineers, it is clear that engineers should prioritize ethical obligation over institutional
obligations and call out those who do not (Davis, 1991). Nonetheless, this rhetoric is
rarely implemented and those who attempt to follow codes are often referred to as “whistle-
blowers”. Institutions can prevent this by creating clear guidelines for what practitioners
should do when they encounter these issues and rewarding individuals who recognize biases
and follow the appropriate steps to fixing them (Frankel, 1989).

4.2.4 Create Organizational structure around ethics

When interviewed, practitioners stated concerns based on the lack of fairness infrastructure
within organizations. Many practitioners stated that most fairness mitigation was done
independently by practitioners and often their efforts went uncompensated (Madaio et al.,
2020; Rakova et al., 2020). Others said they were unsure of who to share fairness issues with
or if it was even their responsibility to do so (Veale et al., 2018; Rakova et al., 2020; Stark
& Hoffmann, 2019). Lastly, some practitioners had concerns with how to handle fairness
issues when different parts of an AI product are owned and/or handled by different teams
(Cramer et al., 2018).
These comments suggest a dire need for organizations to establish clear delegations
of tasks around fairness. Some practitioners even recommended that fairness teams and
internal review boards be created to be used as a resource for practitioners and as audi-
tors of AI technologies (Rakova et al., 2020; Robert et al., 2020). With these additional
support teams, there should be a formal path for communication that dictates clearly to
practitioners the hierarchy for communicating relevant issues (Veale et al., 2018). If issues
arise as it relates to fairness, organizations should have prepared plans for how these issues
will be dealt with (Rakova et al., 2020). Organizations should consider the use of internal
and external investigation committees, especially if handling sensitive protected attributes
(Rakova et al., 2020; Veale & Binns, 2017). It is critical that these teams be given open
access to the data, the models, and the procedures in order to optimize transparency and
reproducibility (Haibe-Kains et al., 2020). Additionally, education infrastructure, focus-
ing on educating practitioners on recognizing and combating algorithmic bias, should be
made available to practitioners and product managers to ensure a baseline of understanding
amongst practitioners (Rakova et al., 2020).
The prioritization of fairness by organizations is critical for their practitioners, their
global image, and the consumers of their products and services. Organizations can prior-
itize fairness by operationalizing it, defining an explicit infrastructure for it, and ensuring
protection for employees interested in implementing it.

4.3 Recommendations for ML Practitioners

Despite the predominant portion of work that needs to be done by fairness experts and
practitioners, there are still a few recommendations in literature for ML practitioners. In

19
 Richardson & Gilbert

order to ensure the success of fairness, effort must also be made by practitioners to educate
themselves and properly implement fairness when it is made available to them. This section
briefly provides recommendations for how practitioners can correctly implement fairness into
their pipelines.

4.3.1 Implement fairness throughout the pipeline

As stated by Stark and Hoffmann (2019), ethics cannot be considered a final checkpoint but
should be considered from the very start of a project and should be reflected within every
stage of the process. Currently, most practitioners engage with fairness ad-hocly or as a
final performance check (Holstein et al., 2019; Madaio et al., 2020; Rakova et al., 2020). But
the literature suggests that fairness would be much easier to implement if it is considered
from the idea formulation stage of the project (Cramer et al., 2018, 2019; Friedman et al.,
2002). The task becomes much more difficult when a model has been deployed and running
for a while because those unintended biases will have become recursive and, therefore, much
more difficult to manage (Cramer et al., 2018). Furthermore, practitioners should make an
effort in including a wide variety of experts and lay-persons into the design, implementation,
and evaluation of ML (Wagstaff, 2012). Practitioners should make sure that assessing and
addressing bias becomes a normal procedure throughout the pipeline of their projects.

4.3.2 Iteratively implement fairness

While fairness should be considered throughout the process, sometimes it may be over-
whelming to immediately counteract every instance of unfairness found. When Cramer
et al. (2018) allowed practitioners to engage with fairness checklists, they noticed concerns
practitioners had with how these checklists might conflict with the rapid delivery develop-
ment procedures that existed within their organizations. Cramer et al. (2018) concluded
that addressing algorithmic bias would need to be done in short-term narrow steps with
plans for iterative improvement of fairness issues. Cramer et al. (2019) also suggested that
such a method could reduce the overload felt by practitioners. Therefore, practitioners
should iteratively implement small fixes to fairness issues between deployments and not try
to detect and handle every issue immediately.

5. Related works
While this work is the first of its kind to provide a comprehensive review of the solution
space of fairness AI, there are a few related reviews. Section 2’s summative breakdown
of biases that exists in machine learning are similar to that of (Friedman & Nissenbaum,
1996; Mehrabi et al., 2021; Olteanu et al., 2019; Hutchinson & Mitchell, 2018; Hellström
et al., 2020). Mehrabi et al. (2021) provides a comprehensive list of 23 types of biases,
which are not organized in any format. Friedman and Nissenbaum (1996) and Olteanu
et al. (2019) both provided a hierarchy of biases, creating their own structure for how
these biases should be organized. While Olteanu et al. (2019) summarizes the different
biases that are associated with social data, Friedman and Nissenbaum (1996) defines a
new set of biases around data in machine learning. This work adopts part of Friedman
and Nissenbaum (1996)’s hierarchical structure and utilizes thematic analysis to categorize

20
 A Framework for Fairness

different biases introduced across a comprehensive list of works. Hutchinson and Mitchell
(2018) discusses the history of fairness, including the differing notions of fairness and gaps in
fairness work. Lastly, Hellström et al. (2020) creates their own taxonomy of bias, discusses
the relation between forms of bias, and provides examples to support differing definitions.
Their taxonomy, however, is depicted based on chronological influences for bias.
While, to our knowledge, there are no reviews that exists that are similar to our Section
3, there are some works that do a user-focused comparative analysis of different toolkits.
Richardson et al. (2021) is a paper that allows practitioners to engage with two toolkits,
UChicago’s Aequitas (Saleiro et al., 2018) and Google’s What-if (Wexler et al., 2019) and
collects their feedback. Seng et al. (2021) is a similar work that compares six different
toolkits: Scikit’s fairness tool (Johnson et al., 2020), IBM Fairness 360 (Bellamy et al.,
2018), UChicago’s Aequitas (Saleiro et al., 2018), Google’s What-if (Wexler et al., 2019),
PyMetrics Audit-AI, and Microsoft’s Fairlearn (Bird et al., 2020). To our knowledge, there
is no other work that reviews both fairness toolkits and checklists.
Furthermore, our contributions from Section 4 are entirely novel. While Richardson
et al. (2021) does a brief literature review of related works that take practitioner feedback
into consideration, this paper is the first of its kind to combine recommendations from
practitioners, fairness experts, and institutions alike and presents a comprehensive review
of necessary work for fairness research.

6. Conclusion
There exists a seemingly endless list of potential influences of bias when it comes to machine
learning algorithms. In order to respond to the prolific nature of bias, fairness researchers
have produced a wide variety of tools that practitioners can use, mainly in the form of
software toolkits and checklists. Companies, academic institutions, and independent re-
searchers alike have created their own versions of these resources, each with unique features
that those developers thought were critical. Nonetheless, there still exists a disconnect be-
tween the fairness developers and the practitioners that prevents the use of these resources
in practice. The literature suggests that fairness in practice can be normalized at the effort
of fairness developers, institutions who create AI, and practitioners who build AI.
This review provides a summary of algorithmic bias issues that have arisen in literature,
highlights some unique contributions in the fairness solution space, and summarizes recom-
mendations for the normalization of fairness practices. To our knowledge, there exists no
other work that provides such a comprehensive review of the problem and solution space of
fairness research. The literature suggests that there is still much work to be done and this
review highlights those major needs.

Acknowledgments

The authors wish to thank Hans-Martin Adorf, Don Rosenthal, Richard Franier, Peter
Cheeseman and Monte Zweben for their assistance and advice. We also thank Ron Musick
and our anonymous reviewers for their comments. The Space Telescope Science Institute is
operated by the Association of Universities for Research in Astronomy for NASA.

21
 Richardson & Gilbert

References
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine Bias. Tech. rep., ProP-
ublica.
Arnold, M., Piorkowski, D., Reimer, D., Richards, J., Tsay, J., Varshney, K. R., Bellamy,
R. K., Hind, M., Houde, S., Mehta, S., Mojsilovic, A., Nair, R., Ramamurthy, K. N.,
& Olteanu, A. (2019). FactSheets: Increasing trust in AI services through supplier’s
declarations of conformity. IBM Journal of Research and Development, 63 (4-5).
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde,
S., Liao, Q. V., Luss, R., Mojsilovic, A., Mourad, S., Pedemonte, P., Raghavendra,
R., Richards, J. T., Sattigeri, P., Shanmugam, K., Singh, M., Varshney, K. R., Wei,
D., & Zhang, Y. (2019). One explanation does not fit all: A toolkit and taxonomy of
ai explainability techniques. ArXiv, abs/1909.03012.
Asaro, P. M. (2019). AI ethics in predictive policing: From models of threat to an ethics of
care. IEEE Technology and Society Magazine, 38 (2), 40–53.
Baeza-Yates, R. (2018). Bias on the web. Communications of the ACM, 61 (6), 54–61.
Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and machine learning. fairml-
book.org.
Barocas, S., & Selbst, A. D. (2018). Big Data’s Disparate Impact. SSRN Electronic Journal,
104, 671–732.
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia,
P., Martino, J., Mehta, S., Mojsilovic, A., Nagar, S., Ramamurthy, K. N., Richards,
J., Saha, D., Sattigeri, P., Singh, M., Varshney, K. R., & Zhang, Y. (2018). AI
Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating
Unwanted Algorithmic Bias. Advances in Neural Information Processing Systems,
2017-Decem(Nips), 5681–5690.
Bennett Moses, L., & Chan, J. (2018). Algorithmic prediction in policing: assumptions,
evaluation, and accountability. Policing and Society, 28 (7), 806–822.
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021). Fairness in Criminal
Justice Risk Assessments: The State of the Art. Sociological Methods & Research,
50 (1), 3–44.
Binns, R. (2018). Fairness in Machine Learning: Lessons from Political Philosophy. Journal
of Machine Learning Research, 81, 1–11.
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018). ’It’s
Reducing a Human Being to a Percentage’; Perceptions of Justice in Algorithmic
Decisions. Conference on Human Factors in Computing Systems - Proceedings, 2018-
April.
Bird, S., Dudı́k, M., Edgar, R., Horn, B., Lutz, R., Milan, V., Sameki, M., Wallach, H., &
Walker, K. (2020). Fairlearn: A toolkit for assessing and improving fairness in AI *.
Tech. rep., Microsoft.
Birhane, A. (2021). Algorithmic injustice: a relational ethics approach. Patterns, 2 (2),
100205.

22
 A Framework for Fairness

Blyth, C. R. (1972). On Simpson’s Paradox and the Sure-Thing Principle. Journal of the
American Statistical Association, 67 (338), 366.
Bowker, G. C., & Star, S. L. (1999). Sorting Things Out: Classification and Its Conse-
quences. MIT Press.
Bradley, T., Ambrose, K., Bernstein, M., DeLoatch, I., Dreisigmeyer, D., Gonzales, J.,
Grubb, C., Haralampus, L., Hawes, M., Johnson, B., Kopp, B., Krebs, J., Marsico, J.,
Morgan, D., Osatuke, K., & Vidrine, E. (2020). Data Ethics Framework. Tech. rep.,
United Kingdom Department of Digital Culture Media and Sport.
Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in
Commercial Gender Classification *. In Proceedings of Machine Learning Research,
Vol. 81, pp. 1–15.
Calders, T., & Žliobaitė, I. (2013). Why unbiased computational processes can lead to
discriminative decision procedures. In Studies in Applied Philosophy, Epistemology
and Rational Ethics, Vol. 3, pp. 43–57. Springer International Publishing.
Chang, K., Raheem, A., & Rha, K. (2018). Novel robotic systems and future directions..
Cheng, L., Varshney, K. R., & Liu, H. (2021). Socially Responsible AI Algorithms: Issues,
Purposes, and Challenges. Journal of Artificial Intelligence Research, 71, 1137–1181.
Choi, D., & Lee, K. (2018). An Artificial Intelligence Approach to Financial Fraud Detection
under IoT Environment: A Survey and Implementation..
Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidi-
vism prediction instruments. Big data, 5 (2), 153–163.
Ciampaglia, G. L., Nematzadeh, A., Menczer, F., & Flammini, A. (2017). How algorithmic
popularity bias hinders or promotes quality. Scientific Reports, 8 (1).
Corbett-Davies, S., & Goel, S. (2018). The measure and mismeasure of fairness: A critical
review of fair machine learning. ArXiv, abs/1808.00023.
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017). Algorithmic De-
cision Making and the Cost of Fairness. In Proceedings of the 23rd ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining, New York, NY,
USA. ACM.
Cramer, H., Garcia-Gathright, J., Springer, A., & Reddy, S. (2018). Assessing and address-
ing algorithmic bias in practice. Interactions, 25 (6), 58–63.
Cramer, H., Reddy, S., Bouyer, R. T., Garcia-Gathright, J., & Springer, A. (2019). Transla-
tion, tracks & Data: An algorithmic bias effort in practice. In Conference on Human
Factors in Computing Systems - Proceedings. Association for Computing Machinery.
Crawford, K. (2013). The Hidden Biases in Big Data. Harvard Business Review.
Crawford, K. (2017). The Trouble with Bias..
Davis, M. (1991). Thinking Like an Engineer: The Place of a Code of Ethics in the Practice
of a Profession. Philosophy and Public Affairs, 20 (2), 150–167.
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019). Explaining
Models: An Empirical Study of How Explanations Impact Fairness Judgment. In

23
 Richardson & Gilbert

International Conference on Intelligent User Interfaces, Proceedings IUI, Vol. Part

F1476, pp. 275–285. Association for Computing Machinery.
Fazelpour, S., & Lipton, Z. C. (2020). Algorithmic Fairness from a Non-ideal Perspective.
AIES 2020 - Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society,
57–63.
Fish, B., Kun, J., & Lelkes, Á. D. (2016). A Confidence-Based Approach for Balancing
Fairness and Accuracy. 16th SIAM International Conference on Data Mining 2016,
SDM 2016, 144–152.
Frankel, M. S. (1989). Professional codes: Why, how, and with what impact?. Journal of
Business Ethics, 8 (2-3), 109–115.
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., &
Roth, D. (2018). A comparative study of fairness-enhancing interventions in machine
learning. FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability,
and Transparency, 329–338.
Friedler, S. A., Scheidegger, C. E., & Venkatasubramanian, S. (2021). The (im)possibility
of fairness. Communications of the ACM, 64, 136 – 143.
Friedman, B., Kahn, P., & Borning, A. (2002). Value sensitive design: Theory and methods.
University of Washington technical report, 2 (12).
Friedman, B., & Nissenbaum, H. (1996). Bias in Computer Systems. ACM Transactions
on Information Systems, 14 (3), 330–347.
Garcia-Gathright, J., Springer, A., & Cramer, H. (2018). Assessing and Addressing Al-
gorithmic Bias - But Before We Get There. In Proceedings ofthe AAAI2018 Spring
Symposium: Designing the User Experience ofArtificial Intelligence.
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., &
Crawford, K. (2018). Datasheets for Datasets. arXiv.
Geiger, R. S., Yu, K., Yang, Y., Dai, M., Qiu, J., Tang, R., & Huang, J. (2019). Garbage In,
Garbage Out? Do Machine Learning Application Papers in Social Computing Report
Where Human-Labeled Training Data Comes From?. In Proceedings of the 2020
Conference on Fairness, Accountability, and Transparency, pp. 325–336. Association
for Computing Machinery, Inc.
Goldenberg, S. L., Nir, G., & Salcudean, S. E. (2019). A new era: artificial intelligence and
machine learning in prostate cancer. Nature Reviews Urology, 16 (7), 391–403.
Gray, C. M., & Chivukula, S. S. (2019). Ethical mediation in UX practice. In Proceedings
of the 2019 CHI Conference on Human Factors in Computing Systems. Association
for Computing Machinery.
Greene, D., Hoffman, A. L., & Stark, L. (2019). Better, Nicer, Clearer, Fairer: A Critical
Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning.
In Proceedings of the 52nd Hawaii International Conference on System Sciences.
Guszcza, J. (2018). Smarter Together: Why artificial intelligence needs human-centric de-
sign. Tech. rep. 22, Deloitte Insights.

24
 A Framework for Fairness

Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Shraddha, T., Kusko, R.,
Sansone, S. A., Tong, W., Wolfinger, R. D., Mason, C. E., Jones, W., Dopazo, J.,
Furlanello, C., Waldron, L., Wang, B., McIntosh, C., Goldenberg, A., Kundaje, A.,
Greene, C. S., Broderick, T., Hoffman, M. M., Leek, J. T., Korthauer, K., Huber,
W., Brazma, A., Pineau, J., Tibshirani, R., Hastie, T., Ioannidis, J. P., Quackenbush,
J., & Aerts, H. J. (2020). Transparency and reproducibility in artificial intelligence.
Nature, 586 (7829), E14–E16.
Harcourt, B. (2007). Against Prediction: Profiling, Policing, and Punishing in an Actuarial
Age. University of Chicago Press, Chicago, IL.
Hellström, T., Dignum, V., & Bensch, S. (2020). Bias in Machine Learning – What is it
Good for?. In CEUR Workshop Proceedings, Vol. 2659, pp. 3–10. CEUR-WS.
Hoffmann, A. L. (2019). Where fairness fails: data, algorithms, and the limits of antidis-
crimination discourse. Information Communication and Society, 22 (7), 900–915.
Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudı́k, M., & Wallach, H. (2019).
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners
Need. In CHI Conference on Human Factors in Computing Systems. ACM.
Hu, L., & Kohler-Hausmann, I. (2020). What’s sex got to do with machine learning?. In
Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency,
FAT* ’20, p. 513, New York, NY, USA. Association for Computing Machinery.
Hutchinson, B., & Mitchell, M. (2018). 50 Years of Test (Un)fairness: Lessons for Machine
Learning. In Proceedings of the 2019 Conference on Fairness, Accountability, and
Transparency, pp. 49–58. Association for Computing Machinery, Inc.
Jacobs, A. Z., & Wallach, H. (2021). Measurement and fairness. In Proceedings of the
2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, p.
375–385, New York, NY, USA. Association for Computing Machinery.
Japkowicz, N., & Shah, M. (2011). Evaluating Learning Algorithms: A Classification Per-
spective. Cambridge University Press.
Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines.
Nature Machine Intelligence, 1 (9), 389–399.
Johnson, B., Bartola, J., Angell, R., Keith, K. A., Witty, S., Giguere, S., & Brun, Y. (2020).
Fairkit, fairkit, on the wall, who’s the fairest of them all? supporting data scientists
in training fair models. ArXiv, abs/2012.09951.
Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and
prospects..
Kasy, M., & Abebe, R. (2021). Fairness, equality, and power in algorithmic decision-making.
In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans-
parency, Vol. 11, pp. 576–586. Association for Computing Machinery, Inc.
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020).
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability
Tools for Machine Learning. In CHI Conference on Human Factors in Computing
Systems.

25
 Richardson & Gilbert

Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., & Schölkopf,
B. (2017). Avoiding Discrimination through Causal Reasoning. In Proceedings of the
2017 Advances in Neural Information Processing Systems, Vol. 30.
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017). Inherent trade-offs in the fair
determination of risk scores. Leibniz International Proceedings in Informatics, LIPIcs,
67, 1–23.
Lakkaraju, H., & Bastani, O. (2019). ”How do I fool you?”: Manipulating User Trust via
Misleading Black Box Explanations. AIES 2020 - Proceedings of the AAAI/ACM
Conference on AI, Ethics, and Society, 79–85.
Law, P.-M., Malik, S., Du, F., & Sinha, M. (2020a). Designing Tools for Semi-Automated
Detection of Machine Learning Biases: An Interview Study. In Proceedings of the
CHI 2020 Workshop on Detection and Design for Cognitive Biases in People and
Computing Systems.
Law, P.-M., Malik, S., Du, F., & Sinha, M. (2020b). The impact of presentation style on
human-in-the-loop detection of algorithmic bias. In Graphics Interface.
Lifshitz, L., & McMaster, C. (2020). Legal and Ethics Checklist for AI Systems. SciTech
Lawyer, 17 (1), 28–34.
Lin, W. C., Chen, J. S., Chiang, M. F., & Hribar, M. R. (2020). Applications of artificial
intelligence to electronic health record data in ophthalmology. Translational Vision
Science and Technology, 9 (2).
Lowry, S., & MacPherson, G. (1988). A blot on the profession. British Medical Journal
(Clinical research ed.), 296 (6623), 657–658.
Lum, K., & Isaac, W. (2016). To predict and serve?. Significance, 13 (5), 14–19.
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020). Co-Designing
Checklists to Understand Organizational Challenges and Opportunities around Fair-
ness in AI. In CHI Conference on Human Factors in Computing Systems, Honolulu.
ACM.
Manders-Huits, N., & Zimmer, M. (2009). Values and Pragmatic Action: The Challenges
of Introducing Ethical Intelligence in Technical Design Communities. International
Review of Information Ethics, 10, 37–44.
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on
bias and fairness in machine learning. ACM Comput. Surv., 54 (6).
Mester, T. (2017). Statistical Bias Types Explained (with examples)..
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E.,
Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. In Proceedings
of the 2019 Conference on Fairness, Accountability, and Transparency, pp. 220–229.
Association for Computing Machinery, Inc.
Mittelstadt, B. (2019). AI Ethics – Too principled to fail?. Nature Machine Intelligence.
Moysan, Y., & Zeitoun, J. (2019). Chatbots as a lever to redefine customer experience in
banking. Journal of Digital Banking, 3 (3), 242–249.

26
 A Framework for Fairness

Munavalli, J. R., Rao, S. V., Srinivasan, A., & van Merode, G. G. (2020). An intelligent real-
time scheduler for out-patient clinics: A multi-agent system model. Health Informatics
Journal, 26 (4), 2383–2406.
Noble, S. U. (2018). Algorithms of Oppression: How Search Engines Reinforce Racism (First
edition). NYU Press.
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019). Social Data: Biases, Methodolog-
ical Pitfalls, and Ethical Boundaries. Frontiers in Big Data, 2 (13).
Patil, D., Mason, H., & Loukides, M. (2018). Of oaths and checklists..
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2020). Where Responsible AI meets
Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices..
Reisman, D., Schultz, J., Crawford, K., & Whittaker, M. (2018). Algorithmic Impact As-
sessments: A Practical Framework for Public Agency Accountability. Tech. rep., AI
Now.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?” Explaining the
Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining, pp. 1135–1144.
Richardson, B., Garcia-Gathright Spotify, J., Way, S. F., Jennifer Thom, S., Henriette
Cramer, S., Garcia-Gathright, J., Thom, J., & Cramer, H. (2021). Towards Fairness
in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits; Towards
Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits.
In CHI Conference on Human Factors in Computing Systems (CHI ’21),, Yokohama,
Japan.
Richardson, B., Prioleau, D., Alikhademi, K., & Gilbert, J. E. (2020). Public Account-
ability : Understanding Sentiments towards Artificial Intelligence across Dispositional
Identities. In IEEE 2020 International Symposium on Technology and Society.
Ridgeway, G. (2013). The Pitfalls of Preduction. Tech. rep. 271, NIJ Journal.
Robert, L. P., Pierce, C., Marquis, E., Kim, S., & Alahmad, R. (2020). Designing Fair AI.
In Human-Computer Interaction.
Rovatsos, M., Mittelstadt, B., & Koene, A. (2019). Landscape Summary: Bias in Algorith-
mic Decision-Making What is bias in algorithmic decision-making, how can we identify
it, and how can we mitigate it?. Tech. rep., Centre for Data Ethics and Innovation.
Saleiro, P., Kuester, B., Stevens, A., Anisfeld, A., Hinkson, L., London, J., & Ghani, R.
(2018). Aequitas: A bias and fairness audit toolkit. ArXiv, abs/1811.05577.
Sandvig, C. (2014). Seeing the Sort: The Aesthetic and Industrial Defense of “The Algo-
rithm”. Journal of the New Media Caucus, 10.
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019).
Fairness and abstraction in sociotechnical systems. In FAT* 2019 - Proceedings of
the 2019 Conference on Fairness, Accountability, and Transparency, pp. 59–68, New
York, NY, USA. Association for Computing Machinery, Inc.

27
 Richardson & Gilbert

Seng, M., Lee, A., & Singh, J. (2021). The Landscape and Gaps in Open Source Fairness
Toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing
Systems. ACM.
Simoiu, C., Corbett-Davies, S., & Goel, S. (2017). The Problem of Infra-marginality in
Outcome Tests for Discrimination. Annals of Applied Statistics, 11 (3), 1193–1216.
Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making?.
International Journal of Human Computer Studies, 51 (5), 991–1006.
Srinivasan, H. (2020). ML-fairness-gym: A Tool for Exploring Long-Term Impacts of Ma-
chine Learning Systems. Google AI Blog.
Stark, L., & Hoffmann, A. L. (2019). Data Is the New What? Popular Metaphors &
Professional Ethics in Emerging Data Culture. Journal of Cultural Analytics.
Sunikka, A., Bragge, J., & Kallio, H. (2011). The effectiveness of personalized marketing
in online banking: A comparison between search and experience offerings. Journal of
Financial Services Marketing, 16 (3-4), 183–194.
Suresh, H., & Guttag, J. V. (2019). A framework for understanding unintended consequences
of machine learning. ArXiv, abs/1901.10002.
Vasudevan, S., & Kenthapadi, K. (2020). LiFT: A Scalable Framework for Measuring
Fairness in ML Applications. In Proceedings of the 29th ACM International Conference
on Information and Knowledge Management.
Veale, M., & Binns, R. (2017). Fairer machine learning in the real world: Mitigating dis-
crimination without collecting sensitive data. Big Data & Society, 4 (2).
Veale, M., Van Kleek, M., & Binns, R. (2018). Fairness and Accountability Design Needs
for Algorithmic Support in High-Stakes Public Sector Decision-Making. Conference
on Human Factors in Computing Systems - Proceedings, 2018-April.
Verma, S., & Rubin, J. (2018). Fairness Definitions Explained. IEEE/ACM International
Workshop on Software Fairness, 18.
Wagstaff, K. L. (2012). Machine Learning that Matters. In Proceedings of the 29th Inter-
national Conference on Machine Learning, Vol. 1, pp. 529–534.
Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Vı́, F., & Wilson, J. (2019). The
What-If Tool: Interactive Probing of Machine Learning Models. IEEE Transactions
on Visualization and Computer Graphics, 26 (1), 56–65.
Xiang, A., & Raji, I. D. (2019). On the Legal Compatibility of Fairness Definitions. arXiv.
Zhong, Z. (2018). A Tutorial on Fairness in Machine Learning..
Zliobaite, I. (2015). On the relation between accuracy and fairness in binary classification. In
The 2nd Workshop on Fairness, Accountability, and Transparency in Machine Learn-
ing.

28