Algorithmic Accountability:

A Primer

Prepared for the Congressional Progressive Caucus:

Tech Algorithm Briefing: Cannon House Office Building

27 Independence Ave., SE
How Algorithms Perpetuate Racial Room 421
Bias and Inequality Washington, DC 20003

April 18, 2018

JOAN DONOVAN ROBYN CAPLAN

Research Lead, PhD, Media Manipulation/ Researcher
Platform Accountability Initiative

JEANNA MATTHEWS LAUREN HANSON

Fellow, PhD Research Analyst

Data & Society is a 501(c)(3) research institute focused on the social and
cultural issues arising from data-centric technological development.

datasociety.net
 Algorithmic Accountability: A Primer 1

Table of Contents
What Is an Algorithm? ………………………………………………………………….……………………. 2
How Are Algorithms Used to Make Decisions? ……………………………….…………….………. 2
Example: Racial Bias in Algorithms of Incarceration ……………………….……………….…… 4
Complications with Algorithmic Systems ………………………………………….…………….……. 6
Fairness and Bias ………………………………………………………………….………….……… 6
Opacity and Transparency ……………………………………………………….…….…………. 7
Repurposing Data and Repurposing Algorithms ……………………………….……..…. 7
Lack of Standards for Auditing ……………………………………………………….………. 8
Power and Control ……………………………………………………………………….…………. 8
Trust and Expertise ……………………………………………………………….…………………. 9
What is Algorithmic Accountability? ………………………………………………….……………….. 10
Auditing by Journalists ………………………………………………………….……………….. 10
Enforcement and Regulation ………………………………………………….……………….. 10

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 2

What Is an Algorithm?
An algorithm is a set of instructions for how a computer should accomplish a par-
ticular task. Algorithms are used by many organizations to make decisions and allocate re-
sources based on large datasets. Algorithms are most often compared to recipes, which take a
specific set of ingredients and transform them through a series of explainable steps into a pre-
dictable output. Combining calculation, processing, and reasoning, algorithms can be excep-
tionally complex, encoding for thousands of variables across millions of data points. Critically,
there are few consumer or civil rights protections that limit the types of data used to build data
profiles or that require the auditing of algorithmic decision-making. Standards and enforce-
ment for fairness, accountability, and transparency are long overdue for algorithms that allocate
housing, healthcare, hiring, banking, social services, as well as goods and service delivery.1 Al-
gorithmic accountability is the process of assigning responsibility for harm when algorithmic
decision-making results in discriminatory and inequitable outcomes.

How Are Algorithms Used to Make Decisions?

Algorithmic decision-making is becoming more common every day. Increasingly, important de-
cisions that affect people’s lives are governed by datasets that are too big for an individual to
process. People have become accustomed to algorithms making all manner of recommenda-
tions, from products to buy, to songs to listen to, to social network connections. But, algorithms
are not just recommending, they are also being used to make big decisions about people’s lives.
Among many applications, algorithms are used to:
• Sort résumés for job applications;
• Allocate social services;
• Decide who sees advertisements for open positions, housing, and products;
• Decide who should be promoted or fired;
• Estimate a person’s risk of committing crimes or the length of a prison term;
• Assess and allocate insurance and benefits;
• Obtain and determine credit; and
• Rank and curate news and information in search engines.

While algorithmic decision making can offer benefits in terms of speed, efficiency, and even
fairness, there is a common misconception that algorithms automatically result in unbiased
decisions. While it may appear like algorithms are unbiased calculations because they take in
objective points of reference and provide a standard outcome, there remain many problems with
those inputs and the outputs. As Frank Pasquale, law professor at the University of Maryland,
points out, algorithmic decision-making is “black boxed,” which means that while we may

1Eubanks, Virginia. 2018. Automating Inequality: How High-Tech Tools Profile, Police, and Punish
the Poor. New York, NY: St. Martin’s Press.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 3

know what goes into the computer for processing and what the outcome is, there are currently
no external auditing systems or regulations for assessing what happens to the data during pro-
cessing.2
Algorithms are attractive because they promise neutrality in decision making—they take
in data and deliver results. But algorithms are not “neutral.” In the words of mathematician
Cathy O’Neil, an algorithm is an “opinion embedded in mathematics.”3 And like opin-
ions, all algorithms are different. Some algorithms privilege a certain group of people over an-
other. O’Neil argues that across a range of occupations, human decision makers are being en-
couraged to defer to software systems even when there is evidence that a system is making in-
correct, unjust, or harmful decisions.
When an algorithm’s output results in unfairness, we refer to it as bias. Bias can find
its way into an algorithm in many ways. It can be created through the social context where an
algorithm is created, as a result of technical constraints, or by the way the algorithm is used in
practice.4 When an algorithm is being created, it is structured by the values of its designer, which
might not be neutral. And after an algorithm is created, it must be trained—fed large amounts
of data on past decisions—to teach it how to make future decisions. If that training data is itself
biased, the algorithm can inherit that bias. For these reasons and others, decisions made by
computer are not fundamentally more logical and unbiased than decisions made by people.
Black-boxed algorithms can unfairly limit opportunities, restrict services, and even pro-
duce “technological redlining.” As Safiya Noble, professor of communication at University
of Southern California, writes, technological redlining occurs when algorithms produce inequi-
table outcomes and replicate known inequalities, leading to the systematic exclusion of Blacks,
Latinos, and Native Americans.5 Technological redlining occurs because we have no control over
how data is used to profile us. If bias exists in the data, it is replicated in the outcome. Without
enforceable mechanisms of transparency, auditing, and accountability, little can be known
about how algorithmic decision-making limits or impedes civil rights.
Noble writes, “technological redlining is a form of digital data discrimination, which uses
our digital identities and activities to bolster inequality and oppression. It is often enacted with-
out our knowledge, through our digital engagements, which become part of algorithmic, auto-
mated, and artificially intelligent sorting mechanisms that can either target or exclude us. It is
a fundamental dimension of generating, sustaining, or deepening racial, ethnic, and gender dis-
crimination, and it is centrally tied to the distribution of goods and services in society, like ed-
ucation, housing, and other human and civil rights. Technological redlining is closely tied to

2 Pasquale, Frank. 2015. The Black Box Society: The Secret Algorithms That Control Money and Infor-
mation. Harvard University Press.
3 O’Neil, Cathy. 2016. Weapons of Math Destruction. Crown.
4 Batya Friedman and Helen Nissenbaum, “Bias in Computer Systems,” ACM Transactions on Infor-

mation Systems 14, no. 3 (1996): 330-347.

5 Noble, Safiya Umoja. 2018. Algorithms of Oppression: How Search Engines Reinforce Racism. 1 edi-

tion. New York: NYU Press.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 4

longstanding practices of ‘redlining,’ which have been consistently defined as illegal by the
United States Congress, but which are increasingly elusive because of their digital deployments
through online, internet-based software and platforms, including exclusion from, and control
over, individual participation and representation in digital systems.”6 Important examples of
technological redlining were uncovered by ProPublica, who showed how Facebook’s targeted
advertising system allowed for discrimination by race and age.7 These decisions embedded in
design have significant ramifications for those who are already marginalized.
In this memo, we begin by showcasing one example to illustrate how racial bias mani-
fests in an algorithmic system. We then address the trade-offs between and debates about algo-
rithms and accountability across several key ethical dimensions: fairness and bias; opacity and
transparency; the repurposing of data and algorithms; lack of standards for auditing; power and
control; as well as trust and expertise. From there, we provide an overview of algorithmic ac-
countability by highlighting how news coverage and self-governance have further exacerbated
problems related to unfair, unethical, and possibly illegal applications of algorithmic systems.

Example: Racial Bias in Algorithms of Incarceration

One of the most important examples of algorithmic bias comes from the justice system, where
a newly-created algorithmic system has imposed stricter jail sentences on black defendants. For
decades, the company Northpointe has developed algorithmic systems for justice system rec-
ommendations. One such system is the Correctional Offender Management Profiling for Alter-
native Sanctions (COMPAS), which is used across the country to assess the risk of recidivism
for defendants in pretrial hearings. The system operates on numerous points of data, such as
questions about whether parents had separated and how many friends had been arrested, to
make sentencing recommendations to judges. The goal of the system is to help balance protect-
ing public safety while also eliminating the possible bias of human judges.8
While the exact details of how COMPAS computes scores is proprietary information, the
system has been built and tested across several dimensions by Northpointe’s own team of

6 Noble wrote this definition of “technological redlining” specifically for this publication.
7 Julia Angwin, Ariana Tobin. 2017. “Facebook (Still) Letting Housing Advertisers Exclude….” ProPub-
lica. November 21, 2017. https://www.propublica.org/article/facebook-advertising-discrimination-
housing-race-sex-national-origin.
Angwin, Julia, Noam Scheiber, and Ariana Tobin. 2017. “Facebook Job Ads Raise Concerns About Age
Discrimination.” The New York Times, December 20, 2017, sec. Business Day. https://www.ny-
times.com/2017/12/20/business/facebook-job-ads.html.
8 Christin, Angele, Alex Rosenblat, and danah boyd. “Courts and Predictive Algorithms.” CRIMINAL

Justice Policy Program 38 (2015). http://www.datacivilrights.org/pubs/2015-1027/Courts_and_Pre-

dictive_Algorithms.pdf.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 5

computer scientists9,10 and externally validated by researchers at Florida State University.11

Their analysis consistently showed that the system met a very commonly accepted definition of
fairness within the field of statistics:12 for defendants of different races, it correctly predicted
recidivism at about the same rate.13 ,14
In 2016, however, ProPublica, a nonprofit news organization known for its investigative
journalism, ran an analysis on how the system was being used in Broward County, Florida.15
Their analysis revealed that even though the system predicted recidivism equally well for white
and black defendants, it made different kinds of systematic mistakes for the two populations.
The system was more likely to mistakenly predict that black defendants were high-
risk, while making the opposite type of mistake for white defendants. This meant
that black defendants who would never go on to recidivate were being treated more harshly by
the law, while white defendants who would go on to commit more crimes were being treated
more leniently. To ProPublica, this was clear evidence of algorithmic bias.16 Northpointe’s re-
sponse was to reassert the statistical merit of the COMPAS system. In the end, there were no
public announcements made about changes to the COMPAS system, and it continues to be
widely used within courts.
The COMPAS conflict hinges on two key factors: there are no standard definitions for
algorithmic bias, and there is no mechanism for holding stakeholders accountable. Northpointe
and ProPublica both agreed that COMPAS should meet some definition of racial fairness but
neither agreed about what that meant. Because there was no public standard, Northpointe was
free to create its own definition of fairness. When a challenge was made, Northpointe was not
accountable to any particular set of values. Because of this lack of governance around the tech-
nologies of algorithmic risk assessment tools, the courts that continue to use the COMPAS sys-
tem are not accountable either. Recently, the New York City Council passed a bill to determine
a process for auditing the selection, use, and implementation of algorithms used by the city that

9 Tim Brennan, Bill Dieterich, Beate Ehret, “Research Synthesis: Reliability and validity of COMPAS,”
Northpointe Inc., September, 2007.
10 Brennan, Tim, William Dieterich, and Beate Ehret. “Evaluating the Predictive Validity of the Compas

Risk and Needs Assessment System.” Criminal Justice and Behavior 36, no. 1 (January 2009): 21–40.
https://doi.org/10.1177/0093854808326545.
11 Blomberg, Thomas, William Bales, Karen Mann, Ryan Meldrum, and Joe Nedelec. “Validation of the

COMPAS Risk Assessment Classification Instrument.” College of Criminology and Criminal Justice,
Florida State University, Tallahassee, FL, 2010. http://criminology.fsu.edu/wp-content/uploads/Vali-
dation-of-the-COMPAS-Risk-Assessment-Classification-Instrument.pdf.
12 Chouldechova, Alexandra. “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Pre-

diction Instruments.” arXiv Preprint arXiv:1610.07524, 2016. https://arxiv.org/abs/1610.07524.

13 Brennan, Tim, William Dieterich, and Beate Ehret. “Evaluating the Predictive Validity of the Compas

Risk and Needs Assessment System.” Criminal Justice and Behavior 36, no. 1 (January 1, 2009): 21–
40. https://doi.org/10.1177/0093854808326545.
14 Blomberg, Thomas, William Bales, Karen Mann, Ryan Meldrum, and Joe Nedelec. “Validation of the

COMPAS Risk Assessment Classification Instrument.”

15 Julia Angwin et. al., “Machine Bias,” ProPublica, May 23, 2016, https://www.propublica.org/arti-

cle/machine-bias-risk-assessments-in-criminal-sentencing.
16 Julia Angwin et. al., “Machine Bias.”

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 6

directly affect people’s lives.17 The bill highlights a need for assessment of disproportionate im-
pacts across protected categories as well as a procedure for redress if harms are found.

Complications with Algorithmic Systems

The COMPAS controversy demonstrates just how many different factors can complicate the de-
sign, use, assessment, and governance of algorithmic systems. Algorithms can be incredibly
complicated and can create surprising new forms of risk, bias, and harm.18 Here, we lay out how
complications in assessing fairness and bias are a result of stakeholders keeping algorithms in-
tentionally opaque amidst calls for transparency. There is a need for greater reflection on mod-
els of power and control, where the sublimation of human decision-making to algorithms erodes
trust in experts. Ultimately, regulators and researchers are ill-equipped to audit algorithms or
enforce any regulation under these conditions.

Fairness and Bias

Algorithms are often deployed with the goal of correcting a source of bias in decisions made by
humans. However, many algorithmic systems either codify existing sources of bias or introduce
new ones. Additionally, bias can exist in multiple places within one algorithm.
An algorithmic system can take on unintended values that compete with designed val-
ues. In the case of COMPAS, the algorithm delivered discriminatory results because of the bias
19

embedded in the training data. Because black people have historically been arrested at a higher
rate than white people, COMPAS learned to predict that a black person is more at risk of being
re-arrested than a white person. When implemented, this system reflects this learning back into
the criminal justice system at a large scale, injecting a source of racial bias into steps of the
judicial process that come after arrest.
By transferring values from one particular political and cultural moment to a different
context, algorithms create a certain moral rigidity. Unless algorithms are consistently moni-
tored and adjusted as time passes, they reinforce the values they were created with and can
become rapidly outdated. For example, in terms of apportionment of healthcare, service deliv-
ery by insurance companies and hospitals depends on algorithmic decision-making, yet some
doctors and caregivers do not agree with the standardized treatment models because these data
are not robust enough to assess variables unavailable to the computer model, such as the un-
steady living conditions of those in poverty.

17 “The New York City Council - File #: Int 1696-2017.” Accessed April 15, 2018. http://legistar.coun-
cil.nyc.gov/LegislationDetail.aspx?ID=3137815&GUID=437A6A6D-62E1-47E2-9C42-461253F9C6D0.
18 Suresh Venkatsburamanian. “When an algorithm isn’t,” Medium, October 1, 2015, https://me-

dium.com/@geomblog/when-an-algorithm-isn-t-2b9fe01b9bb5
19 Batya Friedman, Peter H. Kahn Jr, and Alan Borning, “Value Sensitive Design and Information Sys-

tems,” in Human-Computer Interaction and Management Information Systems: Foundations, ed.

Ping Zhang and Dennis F. Galletta (Abingdon: Routledge, 2006): 348-372.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 7

Opacity and Transparency

Many algorithms are unable to be scrutinized because the data, process, or outcomes they rely
on are kept behind closed doors. According to Jenna Burrell, this can happen for three reasons:
• Intentional corporate or state secrecy, such as a trade secrets;
• Inadequate education on the part of auditors; or
• Overwhelming complexity and scale on the part of the algorithmic system.
The more complex and sophisticated an algorithm is, the harder it is to explain, even by a knowl-
edgeable algorithmic engineer.
Without some level of transparency, it is difficult to know whether an algorithm does
what it says it does, whether it is fair, or whether its outcomes are reliable. For example, there
is a clear-cut need for transparency around risk assessment tools like COMPAS, but this need is
challenged by upholding trade secrets laws. Also, in some cases, transparency may lead to
groups and individuals “gaming the system.” For example, even the minimal openness sur-
rounding how the trending feature on Twitter surfaces topics has allowed it to be manipulated
into covering certain topics by bots and coordinated groups of individuals. Therefore, different
contexts may call for different levels of transparency.

Repurposing Data and Repurposing Algorithms

Algorithms are expensive and difficult to build from scratch. Hiring computer scientists, finding
training data, specifying the algorithm’s features, testing, refining, and deploying a custom al-
gorithm all cost time and money. Therefore, there is a temptation to take an algorithm that
already exists and either modify it or use it for something it wasn’t designed to do. However,
accountability and ethics are context specific. Standards that were set and ethical issues that
were dealt with in an algorithm’s original context may be problems in a new application.
PredPol, a predictive policing service, uses an algorithm designed to predict earthquakes
to find and assign police to hotspots.20 Crime data isn’t the same as earthquake data, though,
and civil rights organizations have criticized PredPol for using biased data to overpolice certain
areas.21 For a variety of reasons, crime data, especially that for arrests, is racially biased, which
has an impact on any algorithm that uses it as training data. This type of approach is also per-
formed at an interpretive level, where the same data is interpreted to apply to a different con-
text. For instance, credit history reports, which are designed to be evidence of financial respon-
sibility, are often used as an input in hiring decisions, even though connections between credit
history and work capability are dubious at best. In order to deal with such algorithmic creep, we

20 Huet, Ellen. “Server and Protect: Predictive Policing Firm PredPol Promises to Map Crime Before It
Happens.” Forbes. Accessed April 10, 2018. https://www.forbes.com/sites/ellenhuet/2015/02/11/pred-
pol-predictive-policing/.
21 Lartey, Jamiles. “Predictive Policing Practices Labeled as ‘flawed’ by Civil Rights Coalition.” The

Guardian, August 31, 2016. http://www.theguardian.com/us-news/2016/aug/31/predictive-policing-

civil-rights-coalition-aclu.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 8

may need new, more cost-effective systems for creating algorithms or more standards in place
for evaluating when an algorithm can be successfully adapted from one application to another.

Lack of Standards for Auditing

Since the 1970s in the financial sphere, independent auditing has been used to detect instances
of discrimination. While independent auditing could be used to detect bias in algorithmic sys-
tems, so far independent auditing is underutilized because of a lack of industry standards or
guidelines for assessing social impact. One set of standards proposed by the Association for
Computing Machinery US Public Policy Council seeks to ensure that automated decision-mak-
ing is held to the same standards as equivalent human decision-making.22 According to the
ACM, these principles should be applied by algorithm designers at every stage in the creation
process, putting the primary responsibility for their adoption in the hands of industry. Another
set of guidelines, put forward by a coalition of industry and university researchers, advocates
for social impact statements to accompany the sale and deployment of algorithmic products.23
In the wake of the Facebook hearings, Russian disinformation campaigns, and the tar-
geted harassment of civil rights organizers, civil society organizations, such as Color of Change
and Muslim Advocates, are calling for independent audits of platforms and internet compa-
nies.24 Data for Black lives has called for a “data public trust,” where they ask Facebook to share
anonymized datasets for public good.25 Data for Black Lives are also drafting a data code of eth-
ics that would focus on data protections and limit digital profiling. Facebook reacted to Cam-
bridge Analytica by deleting pages and limiting access to data, which forecloses the possibility
of outside review.26 As a result, it is imperative to create an organizational structure for inde-
pendent auditing that is open and accessible to researchers and organizations.

Power and Control

One of the primary decisions made by algorithms is that of relevance of each dataset to other
data points. What values, categories, and pieces of information are relevant to customers? Com-
panies? States? Tarleton Gillespie, a professor at Cornell University and principal researcher at
Microsoft, states that algorithms are treated as trusted, objective sources of information.

22 “Statement on Algorithmic Transparency and Accountability,” ACM US Public Policy Council, Janu-
ary 12, 2017, https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_al-
gorithms.pdf.
23 Fairness, Accountability, and Transparency in Machine Learning. n.d. “Principles for Accountable

Algorithms and a Social Impact Statement for Algorithms :: FAT ML.” Accessed April 11, 2018.
https://www.fatml.org/resources/principles-for-accountable-algorithms.
24 Simpson, Scott. “Muslim Advocates and Color Of Change Demand Independent Civil Rights Audit of

Facebook.” Muslim Advocates, April 3, 2018. https://www.muslimadvocates.org/muslim-advocates-

and-color-of-change-demand-independent-civil-rights-audit-of-facebook/.
25 Milner, Yeshimabeit, “An Open Letter to Facebook from the Data for Black Lives Movement.” Me-

dium (blog), April 4, 2018. https://medium.com/@YESHICAN/an-open-letter-to-facebook-from-the-

data-for-black-lives-movement-81e693c6b46c.
26 Facebook. “An Update on Our Plans to Restrict Data Access on Facebook | Facebook Newsroom.”

Published April 4 2018. https://newsroom.fb.com/news/2018/04/restricting-data-access/.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 9

However, their decisions about relevance are choices shaped by a political agenda, whether that
agenda is implicit or explicit to even the algorithm’s own designers.27 This is especially im-
portant for algorithms that perform a gatekeeping role. Algorithms replicate social values but
also embed them into systems, creating new standards and expectations for what is important
in a given context. While there are laws prohibiting the sharing or sale of health and financial
data by hospitals and banks, discrimination occurs because there are few protections in place
for consumer data brokering, where discrete data points act as proxies for protected categories
that are then assembled into profiles that are sold. This can lead to technological redlining.

Trust and Expertise

Trust means many things in different disciplines, but one sociological perspective holds that
trust is the belief that the necessary conditions for success are in place. Those who are pro-
algorithm suggests that humans are too trusting of other humans and some algorithms can out-
perform experts. Humans are accepting of error in other humans, but hold algorithms to a
higher standard. In a series of studies conducted at the University of Chicago, researchers found
that a subject’s likelihood to use output from an algorithm dropped significantly after they saw
evidence that the algorithm can make errors, even if it was still more accurate than their own
responses. From this point of view, humans’ lack of trust in algorithms is irrational. However,
as Eubanks’s and Noble’s research shows, algorithms are just as capable of bias as humans, as
they are embedded with subjective values.
Who is being endowed with trust has a direct relationship with where liability for deci-
sion making should fall. One way of avoiding responsibility is to keep an air of mystery around
who is ultimately accountable through a lack of specification. In the COMPAS case, it wasn’t
clear who was liable for decisions so no one was held accountable for bias in the system. How-
ever, this can lead to a “moral crumple zone,” where one entity is held legally liable for errors,
even if they aren’t in full control of the system.28 For example, airplane pilots are held liable for
the behavior of planes, even though many of the decisions are regularly made by computerized
systems. Determining who is the trusted decision-maker between algorithmic engineers, algo-
rithms, and users requires careful consideration of what the algorithm claims to do and who
suffers from the consequences of mistakes. When an algorithm is making decisions or helping
an expert make decisions, it becomes unclear who is ultimately responsible for the effects of
those decisions.

27 Gillespie, Tarleton. “The Relevance of Algorithms.” Media Technologies: Essays on Communication,

Materiality, and Society 167. (2014).
28 Madeleine Elish and Tim Hwang, “When Your Self-Driving Car Crashes, You Could Still be the One

Who Gets Sued,” Quartz, July 25, 2015, https://qz.com/461905/when-your-self-driving-car-crashes-

you-could-still-be-the-one-who-gets-sued/.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 10

What is Algorithmic Accountability?

Algorithmic accountability ultimately refers to the assignment of responsibility for how an
algorithm is created and its impact on society; if harm occurs, accountable systems include a
mechanism for redress. Algorithms are products that involve human and machine learning.
While algorithms stand in for calculations and processing that no human could do on their own,
ultimately humans are the arbiters of the inputs, design of the system, and outcomes. Im-
portantly, the final decisions to put an algorithmic system on the market belongs to the tech-
nology’s designers and company. Critically, algorithms do not make mistakes, humans do. Es-
pecially in cases of technological redlining, assigning responsibility is critical for quickly reme-
diating discrimination and assuring the public that proper oversight is in place. In addition to
clearly assigning responsibility for the implementation of decisions made by algorithms, ac-
countability must be grounded in enforceable policies that begin with auditing in pre- and post-
marketing trials as well as standardized assessments for any potential harms. Currently, it is
difficult to get technology corporations to answer for the harms their products have caused.

Below we outline how journalists, in consultation with academics and whistleblowers, have
taken up the role of auditing algorithms, while also showing how the lack of enforceable regu-
lation led to a deficit in consumer protections.

Auditing by Journalists
Currently, journalists are an important watchdog for algorithmic bias. Data journalism blends
investigative methods from journalism with technical know-how to provide clear and accurate
reporting on computational topics. While many algorithms are proprietary information, skilled
journalists can use techniques of “reverse-engineering” to probe what’s inside the black box by
pairing inputs with outputs. A second approach facilitated by journalists is that of collaborative
research with academics and whistleblowers. Particularly for personalization algorithms, which
can be difficult or impossible to parse from the perspective of an individual user’s account, peer-
sourced research can reveal patterns that give clues about how the underlying algorithms work.

Enforcement and Regulation

The governance of algorithms is played out on an ad hoc basis across sectors. In some cases,
existing regulations are reinterpreted to apply to technological systems and guide behavior, as
with Section 230 of the Communications Decency Act. These instances can be hotly contested
as algorithmic systems bring up new issues not before properly covered by the logic of existing
precedents. In other cases, specific governing bodies are convened in order to set standards. For
example, the Internet Governance Forum has been convened annually by the United Nations
since 2006 and attempts to set non-binding guidance around such facets of the internet as the
diversity of media content.

DATA & SOCIETY datasociety.net

 Algorithmic Accountability: A Primer 11

However, for accountability to be meaningful, it needs to come with the appropriate gov-
ernance structures. According to Florian Saurwein, Natascha Just, and Michael Latzer, govern-
ance is necessary because algorithms impose certain risks, such as the violation of privacy rights
and social discrimination.29 These risks need to be dealt with by the appropriate governance
structure, which currently involves little oversight by states. Governance can occur by market
and design solutions, such as product innovation that mitigates risk or consumers’ ability to
substitute risky products for ones they deem safer. Governance can also come from industry
self-regulation, where company principles and collective decision-making favor public interest
concerns. Last is traditional state intervention through mechanisms such as taxes and subsidies
for certain kinds of algorithmic behavior. The appropriate structure must be matched with the
context at hand to ensure the accountability mechanisms are effective.
Because of the ad hoc nature of self-governance by corporations, few protections are in
place for those most affected by algorithmic decision-making. Much of the processes for obtain-
ing data, aggregating it, making it into digital profiles, and applying it to individuals are corpo-
rate trade secrets. This means they are out of the control of citizens and regulators. As a result,
there is no agency or body currently in place that develops standards, audits, or enforces neces-
sary policies.
While law has always lagged behind technology, in this instance technology has become
de facto law affecting the lives of millions—a context that demands lawmakers create policies
for algorithmic accountability to ensure these powerful tools serve the public good.

29Saurwein, Florian, Natascha Just, and Michael Latzer. “Governance of Algorithms: Options and Limi-
tations.” Info 17, no. 6 (September 14, 2015): 35–49. https://doi.org/10.1108/info-05-2015-0025.

DATA & SOCIETY datasociety.net