DEEPFAKES AND

CHEAP FAKES
THE MANIPULATION OF
AUDIO AND VISUAL EVIDENCE

Britt Paris
Joan Donovan
DEEPFAKES AND CHEAP FAKES -1-

CONTENTS
02 Executive Summary
05 Introduction
10 Cheap Fakes/Deepfakes: A Spectrum
17 The Politics of Evidence
23 Cheap Fakes on Social Media
25 Photoshopping
27 Lookalikes
28 Recontextualizing
30 Speeding and Slowing
33 Deepfakes Present and Future
35 Virtual Performances
35 Face Swapping
38 Lip-synching and Voice Synthesis
40 Conclusion
47 Acknowledgments

Author: Britt Paris, assistant professor of Library and Information Science,

Rutgers University; PhD, 2018,Information Studies, University of California,
Los Angeles.

Author: Joan Donovan, director of the Technology and Social Change Research
Project, Harvard Kennedy School; PhD, 2015, Sociology and Science Studies,
University of California San Diego.

This report is published under Data & Society’s Media Manipulation research
initiative; for more information on the initiative, including focus areas,
researchers, and funders, please visit https://datasociety.net/research/
media-manipulation
DATA & SOCIETY -2-

EXECUTIVE SUMMARY

Do deepfakes signal an information apocalypse? Are they

the end of evidence as we know it? The answers to these
questions require us to understand what is truly new
about contemporary AV manipulation and what is
simply an old struggle for power in a new guise.

The first widely-known examples of amateur,AI-manipulated,

face swap videos appeared in November 2017. Since then,
the news media, and therefore the general public, have
begun to use the term “deepfakes” to refer to this larger
genre of videos—videos that use some form of deep or
machine learning to hybridize or generate human bodies
and faces. News coverage claims that deepfakes are poised
to assault commonly-held standards of evidence, that they
are the harbingers of a coming “information apocalypse.”
But what coverage of this deepfake phenomenon often
misses is that the “truth” of audiovisual content has
never been stable—truth is socially, politically, and
culturally determined.

Deepfakes which rely on experimental machine

learning represent one end of a spectrum of audio-
visual AV manipulation. The deepfake process is both
the most computationally-reliant and also the least publicly
accessible means of creating deceptive media. Other forms
of AV manipulation – “cheap fakes” – rely on cheap,
accessible software, or no software at all. Both deepfakes
DEEPFAKES AND CHEAP FAKES -3-

and cheap fakes are capable of blurring the line between

expression and evidence. Both can be used to influence
the politics of evidence: how evidence changes and
is changed by its existence in cultural, social, and
political structures.

Locating deepfakes and cheap fakes in the longer

history of the politics of evidence allows us to see:

• decisions over what counts as “evidence” have

historically been a crucial tool in defending
the privilege of the already powerful;

• the violence of AV manipulation can only be

addressed by a combination of technical and
social solutions;

• public agency over these technologies cannot be

realized without addressing structural inequality;
and

• the violence of AV manipulation will not be

curtailed unless those groups most vulnerable
to that violence are able to influence public
media systems.
DATA & SOCIETY -4-

“AV manipulation includes

both the cutting edge,
AI-reliant techniques of deep-
fakes, as well as “cheap fakes”
that use conventional
techniques like speeding,
slowing, cutting, re-staging, or
re-contextualizing footage.”
DEEPFAKES AND CHEAP FAKES -5-

INTRODUCTION
In June 2019, artists Bill Posters and Daniel Howe posted a
fake video of Mark Zuckerberg to Instagram. Using a pro-
1

prietary video dialogue replacement model (VDR) made by

Canny, an Israeli advertising company, Posters and Howe
produced a video of Zuckerberg talking about amassing
power and control through a nefarious organization called
Spectre. The video was created using a suite of machine
learning and artificial intelligence techniques that enable
the sophisticated manipulation of visual data, specifically
the movement of bodies. While some are fascinated by the
expressive possibilities of these technologies, others see
dire consequences in the ability to put words and actions
in the mouths and bodies of others. If video can no longer
be trusted as proof that someone has done something, what
happens to evidence, to truth?

The first widely known examples of amateur,AI-manipulated,

face-swap videos appeared in November 2017, when a Reddit
user with the username “deepfakes” uploaded a series of
videos with the faces of famous female actors, including Gal
Gadot and Scarlett Johansson, grafted onto other actors’
bodies in pornography. Since then, the news media, and
therefore the general public, have begun to use the term
“deepfakes” to refer to this genre of videos that use some
form of “deep” or machine learning to hybridize or generate
human bodies and faces.

Deepfakes, however, are just one component of a larger field

of audiovisual (AV) manipulation. AV manipulation includes
any sociotechnical means for influencing the interpretation

1 https://www.instagram.com/p/ByaVigGFP2U/
DATA & SOCIETY -6-

of media. AV manipulation includes both the cutting edge,

AI-reliant techniques of deepfakes, as well as “cheap fakes”
that use conventional techniques like speeding, slowing,
cutting, re-staging, or re-contextualizing footage.

How will deepfakes complicate the larger field of AV manip-

ulation? Many of the journalists who cover deepfakes have
declared them the harbingers of a coming “information
apocalypse.” Journalists, politicians, and others allege deep-
2

fakes’ ability to destroy democracy: to tamper with elections,

compromise national security, or foment widespread
violence. News coverage claims that deepfakes are poised
3

to destroy video’s claim to truth by permanently blurring

the line between evidentiary and expressive video. But what
coverage of this deepfake phenomenon often misses is that
the “truth” of AV content has never been stable—truth is
socially, politically, and culturally determined. And people
are able to manipulate truth with deepfakes and cheap
fakes alike.

2 Charlie Warzel, “He Predicted the 2016 Fake News Crisis. Now He’s Worried About An
Information Apocalypse.,” BuzzFeed News (blog), February 11, 2018, https://www.buzz-
feednews.com/article/charliewarzel/the-terrifying-future-of-fake-news; Franklin Foer,
“The Era of Fake Video Begins,” The Atlantic, April 8, 2018, https://www.theatlantic.
com/magazine/archive/2018/05/realitys-end/556877/; Samantha Cole, “AI-Assisted
Fake Porn Is Here and We’re All Fucked,” Motherboard (blog), December 11, 2017,
https://motherboard.vice.com/en_us/article/gydydm/gal-gadot-fake-ai-porn; Joshua
Rothman, “In the Age of A.I., Is Seeing Still Believing?,” The New Yorker, November 5,
2018, https://www.newyorker.com/magazine/2018/11/12/in-the-age-of-ai-is-seeing-
still-believing; Jennifer Finney Boylan, “Will Deep-Fake Technology Destroy Democ-
racy?,” The New York Times, October 19, 2018, sec. Opinion, https://www.nytimes.
com/2018/10/17/opinion/deep-fake-technology-democracy.html.
3 Warzel, “He Predicted the 2016 Fake News Crisis. Now He’s Worried About An Informa-
tion Apocalypse”; James Vincent, “US Lawmakers Say AI Deepfakes ‘Have the Potential
to Disrupt Every Facet of Our Society,’” The Verge, September 14, 2018, https://www.
theverge.com/2018/9/14/17859188/ai-deepfakes-national-security-threat-lawmak-
ers-letter-intelligence-community; Daniel Funke, “A Potential New Marketing Strategy
for Political Campaigns: Deepfake Videos,” Poynter, June 6, 2018, https://www.poynter.
org/news/potential-new-marketing-strategy-political-campaigns-deepfake-videos;
Boylan, “Will Deep-Fake Technology Destroy Democracy?”
DEEPFAKES AND CHEAP FAKES -7-

We contextualize the phenomenon of deepfakes with the

history of the politics of AV evidence to show that the ability
to interpret the truth of evidence has been the work of insti-
tutions—journalism, the courts, the academy, museums, and
other cultural organizations. Every time a new AV medium
4

has been decentralized, people have spread content at new

speeds and scales; traditional controls over evidence have
been upset until trusted knowledge institutions weighed in
with a mode of dictating truth. In many cases, panic around
a new medium has generated an opening for experts to gain
juridical, economic, or discursive power.

However, there are two related phenomena that are truly new
today. First, deepfakes are being applied to the digitization
of bodies, including one’s voice and likeness in increasingly
routine ways. These are techniques that require training data
of only a few hundred images. With thousands of images of
many of us online, in the cloud, and on our devices, anyone
with a public social media profile is fair game to be faked.
And we are already seeing that the unbounded use of tools

4 Jean Baudrillard et al., In the Shadow of the Silent Majorities, trans. Paul Foss et
al. 1979. (Los Angeles: Cambridge, Mass: Semiotext, 2007); Jean Baudrillard,
Simulacra and Simulation. 1980, trans. Sheila Faria Glaser, 14th Printing edition (Ann
Arbor: University of Michigan Press, 1994); John Fiske and Kevin Glynn, “Trials of
the Postmodern,” Cultural Studies 9, no. 3 (October 1, 1995): 505–21, https://doi.
org/10.1080/09502389500490541; Ian Hacking, “Prussian Numbers 1860-1882,”
in The Probabilistic Revolution, Volume 1 (Cambridge, MA: MIT Press, 1987), 377–94;
Herbert Schiller, Information Inequality, 1 edition (New York, NY: Routledge, 1995); Jean
Baudrillard, The Gulf War Did Not Take Place. 1993. (Indiana University Press, 1995); Tal
Golan, Laws of Men and Laws of Nature: The History of Scientific Expert Testimony in
England and America (Cambridge, MA; London: Harvard University Press, 2007); Sarah
E. Igo, The Averaged American: Surveys, Citizens, and the Making of a Mass Public
(Cambridge, MA: Harvard University Press, 2008); Randall C. Jimerson, Archives Power:
Memory, Accountability, and Social Justice (Society of American Archivists, 2009);
Saloni Mathur, “Social Thought & Commentary: Museums Globalization,” Anthropologi-
cal Quarterly 78, no. 3 (2005): 697–708.trans. Sheila Faria Glaser, 14th Printing edition
(Ann Arbor: University of Michigan Press, 1994
DATA & SOCIETY -8-

for AV manipulation is most likely to have a negative impact

on women, people of color, and those questioning powerful
systems. 5

Second, AV manipulations of any type, deepfakes or

cheap, can be transmitted at the speed and scale of today’s
online platforms. This increase in distribution provides
a real challenge to traditional counters to manipulation
and makes efforts like moderation or fact-checking
harder to accomplish. Further, in some applications,
like WhatsApp, encrypted messages are circulated along
private connections, achieving a type of hidden virality in
6

which these fake videos can spread to a wide network of

viewers while avoiding content moderation or mainstream
media coverage.

Currently, technologists. policymakers, and journalists are

responding to deepfakes with calls for what scholars call
technical and legal closures —that is, regulations, design
7

features, and cultural norms that will determine the role

of this technology. Venture capitalists, technologists, and

5 Jessie Daniels, Cyber Racism: White Supremacy Online and the New Attack on Civil
Rights (Lanham, MD: Rowman & Littlefield Publishers, 2009); Safiya Umoja Noble, Algo-
rithms of Oppression: How Search Engines Reinforce Racism, 1 edition (New York: NYU
Press, 2018); Mary Anne Franks, “Unwilling Avatars: Idealism and Discrimination in Cy-
berspace,” Columbia Journal of Gender and Law 20, no. (2011) (May 9, 2019), https://
cjgl.cdrs.columbia.edu/article/unwilling-avatars-idealism-and-discrimination-in-cyber-
space?article=unwilling-avatars-idealism-and-discrimination-in-cyberspace&post_
type=article&name=unwilling-avatars-idealism-and-discrimination-in-cyberspace;
Franks, “The Desert of the Unreal: Inequality in Virtual and Augmented Reality,” U.C.D.
L. Rev., January 1, 2017, 499; Danielle Citron, “Addressing Cyber Harassment: An
Overview of Hate Crimes in Cyberspace,” Journal of Law, Technology, & the Internet 6,
no. 1 (January 1, 2015): 1.
6 Recently Facebook’s Zuckerberg made a statement that Facebook’s suite of plat-
forms, of which WhatsApp is one, should follow WhatsApp’s mode of encryption and
private messaging.
7 Trevor J. Pinch and Wiebe E. Bijker, “The Social Construction of Facts and Artefacts:
Or How the Sociology of Science and the Sociology of Technology Might Benefit Each
Other,” Social Studies of Science 14, no. 3 (1984): 399–441.
DEEPFAKES AND CHEAP FAKES -9-

entrepreneurs in particular have called for new forms of

technical redress, from the automated identification of fakes
to the mandatory registration of content creators. But such 8

strategies do not address the thornier, and more important,

issues of social, cultural, and historical context. There is a
risk that these technical and legal closures will be directed
by those who already hold economic and political power in
ways that, following the political history of evidence, further
consolidate truth for the status quo.

8 Antonio García Martínez, “The Blockchain Solution to Our Deepfake Problems,” Wired,
March 26, 2018, https://www.wired.com/story/the-blockchain-solution-to-our-deep-
fake-problems/; Siwei Lyu, “The Best Defense against Deepfake AI Might Be... Blinking,”
Fast Company, August 31, 2018, https://www.fastcompany.com/90230076/the-best-
defense-against-deepfakes-ai-might-be-blinking; Tristan Greene, “Researchers Devel-
oped an AI to Detect DeepFakes,” The Next Web, June 15, 2018, https://thenextweb.
com/artificial-intelligence/2018/06/15/researchers-developed-an-ai-to-detect-deep-
fakes/; Facebook, “Expanding Fact-Checking to Photos and Videos,” Facebook
Newsroom, September 13, 2018, https://newsroom.fb.com/news/2018/09/expand-
ing-fact-checking/; TruePic, “Photos and Videos You Can Trust,” accessed December
10, 2018, https://truepic.com/about/; Sam Gregory and Eric French, “OSINT Digital
Forensics,” WITNESS Media Lab (blog), accessed June 20, 2019, https://lab.witness.
org/projects/osint-digital-forensics/; Matt Turek, “Media Forensics,” Defense Advanced
Research Projects MediFor, 2018, https://www.darpa.mil/program/media-forensics.
DATA & SOCIETY DEEPFAKES AND CHEAP FAKES

THE This spectrum charts specific examples of audiovisual (AV) manipulation that The deepfake process is both the most computationally reliant and also the least
illustrate how deepfakes and cheap fakes differ in technical sophistication, publicly accessible means of manipulating media. Other forms of AV
DEEPFAKES/ barriers to entry, and techniques. From left to right, the technical sophistication manipulation rely on different software, some of which is cheap to run, free to
CHEAP FAKES of the production of fakes decreases, and the wider public’s ability to produce download, and easy to use. Still other techniques rely on far simpler methods, like
SPECTRUM fakes increases. Deepfakes—which rely on experimental machine learning— mislabeling footage or using lookalike stand-ins.
are at one end of this spectrum.

Recurrent
TECHNOLOGIES

Neural
After
Network (RNN); Generative Video Free Relabeling/
Effects, Free speed
Hidden Markov Adversarial Dialogue FakeApp / Sony real-time In-camera Reuse of
Adobe alteration
Models (HMM) Networks Replacement After Effects Vegas Pro filter effects extant
Premiere applications
and Long Short (GANs) (VDR) model applications video
Pro
Term Memory
Models (LTSM)

DEEPFAKES More expertise and technical resources required Less expertise and fewer technical resources required CHEAP FAKES
TECHNIQUES

Virtual Virtual Voice Face Lip-synching Face Speeding Face altering/ Speeding and Lookalikes Recontextualizing
performances performances synthesis swapping (page 38) swapping: and slowing swapping slowing (page 27) (page 28)
(page 35) (page 38) (page 35) Rotoscope (page 30)

Rana Ayyub
Deepfakes: Gal Gadot (not pictured
(not pictured because because of
EXAMPLES

of image content) image content)

Suwajanajorn et al. Face2Face: Posters and Howe’s: Mark Zuckerberg Paul Joseph Watson: Acosta Video Belle Delphine: Hit or
Synthesizing Obama Miss Choreography

Mario Klingemann: AI Art Jordan Peele and BuzzFeed: Obama PSA Huw Parkinson: Uncivil War SnapChat: Amsterdam Fashion Institute Unknown: BBC NATO newscast
DATA & SOCIETY - 12 -

THE CHEAP FAKES-

DEEPFAKES SPECTRUM
The upper end of the spectrum (page 10) contains
experimental methods of computationally intensive image
production, exclusive to a small set of graphics researchers
and professional media producers.The deepfake process was
initially developed by those working in the graphics processing
industry. NVIDIA, the market leader in the production of
9

graphics processing units (GPUs), has been at the forefront

10

of “deep learning”—a type of machine learning that uses

layers of algorithms called “neural networks” to sort through
visual data to make predictions. In recent years, these treat-
11

ments have included weather prediction, cancer detection,

9 The graphical processing unit (GPUs) are found in chips in a computer’s motherboard,
with the CPU, or as a plug-in unit. It renders images, animations and video for the
computer's screen. A GPU can perform parallel processing and pixel decoding much
faster than a CPU. The primary benefit is that it decodes pixels to render smooth 3D
animations and video.
10 Jensen Huang, “Accelerating AI with GPUs: A New Computing Model,” The Official
NVIDIA Blog, January 12, 2016, https://blogs.nvidia.com/blog/2016/01/12/acceler-
ating-ai-artificial-intelligence-gpus/; Moor Insights and Strategy, “A Machine Learning
Landscape: Where AMD, Intel, NVIDIA, Qualcomm And Xilinx AI Engines Live,” Forbes,
accessed June 30, 2018, https://www.forbes.com/sites/moorinsights/2017/03/03/a-
machine-learning-landscape-where-amd-intel-nvidia-qualcomm-and-xilinx-ai-engines-
live/; “Intel, Nvidia Trade Shots Over AI, Deep Learning,” eWEEK, accessed June 30,
2018, https://www.eweek.com/servers/intel-nvidia-trade-shots-over-ai-deep-learning.
11 Machine learning is a set of complicated algorithms used to train machines to perform
a task with data, such as sorting a list of names in alphabetical order. The first few
times, the lists would be mostly correct, but with a few ordering errors. A human would
correct these errors and feed that data back into the system. As it sorts through
different hypothetical lists of names, the machine does it faster with fewer errors. Ma-
chine learning requires a human for error correction. Deep learning, on the other hand,
is a subset of machine learning that layers these algorithms, called a neural network,
to correct one another. It still uses complicated algorithms to train machines to sort
through lists of names, but instead of a human correcting errors, the neural network
does it.
DEEPFAKES AND CHEAP FAKES - 13 -

and responsive navigation systems for self-driving vehicles. 12

Computer scientists are also experimenting with ever more

sophisticated strains of deep learning neural networks
such as recurrent neural networks (RNNs) or generative
adversarial networks (GANs), to turn audio or audiovisual 13

clips into realistic, but completely fake, lip-synced videos.

14 15

With these techniques, media producers are able to create

entirely virtual performances by recognizable figures, as in
the cases of synthesizing various performances of Obama
in prototypes produced by the University of Washington, 16

Stanford, and at the State University of New York at

17

Albany. It is these spectacular, high-profile experiments

18

12 Economist Special Report, ”From Not Working to Neural Networking,” The Economist,
June 25, 2016, https://www.economist.com/special-report/2016/06/25/from-not-
working-to-neural-networking.; Jon Markman, “Deep Learning, Cloud Power Nvidia,”
Forbes, November 22, 2016, https://www.forbes.com/sites/jonmarkman/2016/11/22/
deep-learning-cloud-power-nvidia-future-growth/; Mario Casu et al., “UWB Mi-
crowave Imaging for Breast Cancer Detection: Many-Core, GPU, or FPGA?,” ACM
Trans. Embed. Comput. Syst. 13, no. 3s (March 2014): 109:1–109:22, https://doi.
org/10.1145/2530534.
13 Supasorn Suwajanakorn, Steven Seitz, and Ira Kemelmacher-Shlizerman, “Synthesizing
Obama: Learning Lip Sync from Audio,” in ACM Transactions on Graphics, vol. 36. 4, Ar-
ticle 95 (SIGGRAPH 2017, New York, NY: Association for Computing Machinery, 2017),
http://grail.cs.washington.edu/projects/AudioToObama/.
14 Aayush Bansal et al., “Recycle-GAN: Unsupervised Video Retargeting,” ArXiv:
1808.05174 [Cs], August 15, 2018, http://arxiv.org/abs/1808.05174.
15 In 2011, NVIDIA began partnering with deep learning computer scientists at leading
research universities to develop image and pattern recognition technology. https://
blogs.nvidia.com/blog/2016/01/12/accelerating-ai-artificial-intelligence-gpus/
16 Suwajanakorn, et al., “Synthesizing Obama: Learning Lip Sync from Audio.”
17 Justus Thies, Michael Zollhfer, Marc Stamminger, Christian Theobalt, and Matthias
Neissner, “Face2Face: Real-Time Face Capture and Reenactment of RGB Videos,” ac-
cessed June 27, 2019, https://cacm.acm.org/magazines/2019/1/233531-face2face/
fulltext.
18 Gregory Barber, “Deepfakes Are Getting Better, But They’re Still Easy to Spot,” Wired,
May 26, 2019, https://www.wired.com/story/deepfakes-getting-better-theyre-easy-
spot/.
DATA & SOCIETY - 14 -

that have driven much of the press coverage of deepfakes. 19

Computer scientists are not the only ones with access to

deepfake techniques. Consumer-grade animation software
like Adobe After Effects enables anyone with average
computing power and a little time to create similar
audiovisual fakes through machine learning. Individual pro-
grammers have also created open source projects for deepfake
creation—projects like FakeApp, FaceSwap, and Deep-
20 21

Face Lab. Most of these are hosted in public repositories

22

like GitHub and can produce output ranging from face-

mapped performance videos to artificial lip-synchs.

Even without deep and machine learning, some video pro-

ducers use a technique called digital rotoscoping to produce
similar effects to deepfake face projection. This technique
requires creators to manually outline target areas in every
frame of a video and is therefore both difficult and time
consuming. For instance, Australian artist Huw Parkinson
used this technique to create an Avengers-themed political
parody,manually mapping politicians’faces onto superheroes’
bodies. Many other video producers manipulate media
with simple methods of slowing, speeding, or cutting. In

19 Artists have also used deep learning methods to produce abstract images that sell for
great amounts of money and exhibit in the most renowned galleries across the world,
for example, new media artist Mario Klingeman’s work shown in the typology above
has shown at MoMA Le Centre Pompidou. Rama Allen, “AI Will Be the Art Movement
of the 21st Century,” Quartz, March 5, 2018, https://qz.com/1023493/ai-will-be-
the-art-movement-of-the-21st-century/; “Artificial Intelligence and the Art of Mario
Klingemann,” Sothebys.com, February 8, 2019, https://www.sothebys.com/en/articles/
artificial-intelligence-and-the-art-of-mario-klingemann.
20 “FakeApp 2.2.0 - Download for PC Free,” Malavida, accessed June 1, 2018, https://
www.malavida.com/en/soft/fakeapp/.
21 deepfakes, Deepfakes/Faceswap, Python, 2019, https://github.com/deepfakes/
faceswap.
22 iperov, DeepFaceLab Is a Tool That Utilizes Machine Learning to Replace Faces in Vid-
eos. Includes Prebuilt Ready to Work Standalone Windows 7,8,10 Binary (Look Readme.
Md), Iperov/DeepFaceLab, Python, 2019, https://github.com/iperov/DeepFaceLab.
DEEPFAKES AND CHEAP FAKES - 15 -

“By using lookalike stand-ins, or

relabeling footage of one event
as another, media creators can
easily manipulate an audience’s
interpretations.”

2018 and 2019, respectively, videos of CNN reporter Jim

Acosta and Speaker of the House of Representatives Nancy
Pelosi were the subject of widespread media coverage; both
of these had been altered by speeding or slowing using
consumer editing software.

Countless free mobile apps now offer limited versions of

the above techniques: speeding and slowing, but also forms
of facial tracking and manipulation. Apps like SnapChat
and TikTok offer dramatically lowered computational and
expertise requirements that allow users to generate various
manipulations in real time. SnapChat offers elements that
mimic rotoscoping simply by clicking on filters. TikTok has
a time filter that allows people to speed and slow video both
as they capture it and afterward.

Finally, the most accessible forms of AV manipulation are

not technical but contextual. By using lookalike stand-ins,
or relabeling footage of one event as another, media creators
can easily manipulate an audience’s interpretations. One
such example is a video emailed to Indian journalist Rana
Ayyub as a form of blackmail. According to Ayyub, the
manipulators had staged a pornographic video with a body
double in an attempt to silence her critiques of the Indian
DATA & SOCIETY - 16 -

government. Similarly accessible techniques were used in

23

another Indian disinformation effort in 2018. Manipulators

circulated a video via WhatsApp that showed footage of a
nerve gas attack in Syria from 2013. By misidentifying the
time and events of the video, manipulators used it to support
false claims of child kidnapping and motivated violence in
rural India. These types of staging and re-contextualizing
24

are possible for nearly anyone to reproduce, and technical

forensic differences are even harder to detect, because there
are no pixels out of order. 25

Regardless of where a particular example rests on the

spectrum of techniques, it is critical to attend to the ways in
which these examples work as both expression and evidence.
Both technically sophisticated and exceedingly simple tech-
niques can be used in works of art or fiction. But problems
arise when these techniques are used to create works that
are interpreted as evidence. When that happens, despite the
techniques or intentions behind production, manipulated
media becomes a tool for changing or maintaining the
distribution of power.

23 Rana Ayyub, “I Was the Victim of a Deepfake Porn Plot Intended to Silence Me,”
HuffPost UK, November 21, 2018, https://www.huffingtonpost.co.uk/entry/deep-
fake-porn_uk_5bf2c126e4b0f32bd58ba316; Rana Ayyub, “In India, Journalists Face
Slut-Shaming and Rape Threats,” The New York Times, May 22, 2018, https://www.
nytimes.com/2018/05/22/opinion/india-journalists-slut-shaming-rape.html.
24 Timothy McLaughlin, “How WhatsApp Fuels Fake News and Violence in India,” Wired,
December 12, 2018, https://www.wired.com/story/how-whatsapp-fuels-fake-news-
and-violence-in-india/.
25 Gregory and French, “OSINT Digital Forensics.”
DEEPFAKES AND CHEAP FAKES - 17 -

THE POLITICS OF EVIDENCE

In the past year, much news coverage has claimed that
AI-generated video signals an incoming “information
apocalypse.” For instance, the MIT Technology Review
26

declared that “AI could set us back 100 years when it comes
to how we consume news.” The Atlantic characterized the
27

advent of AI-video production capacity as the “collapse

of reality.” And The New Yorker questioned “In the age
28

of A.I., is seeing still believing?” Across these panicked

29

summations of AV manipulation there is an underlying

assumption: that video and photography, before our current
moment, worked as objective, accurate evidence.

However, reviewing the history of visual evidence, it becomes

clear that the relationship between media and truth has never
been stable. There has always been a politics of evidence
around audiovisual media—how evidence changes and is
changed by its existence in cultural, social, and political
structures.

The treatment of visual media as an objective documentation

of truth is a 19th century legal construct. Science studies
scholar Tal Golan has documented how photographic
30

26 Warzel, “He Predicted the 2016 Fake News Crisis. Now He’s Worried About an Informa-
tion Apocalypse.”
27 Jackie Snow, “AI Could Send Us Back 100 Years When It Comes to How We Consume
News,” MIT Technology Review, November 7, 2017, https://www.technologyreview.
com/s/609358/ai-could-send-us-back-100-years-when-it-comes-to-how-we-
consume-news/.
28 Foer, “The Era of Fake Video Begins.”
29 Joshua Rothman, “In the Age of A.I., Is Seeing Still Believing?,” The New Yorker, Novem-
ber 5, 2018, https://www.newyorker.com/magazine/2018/11/12/in-the-age-of-ai-is-
seeing-still-believing.
30 Tal Golan, Laws of Men and Laws of Nature: The History of Scientific Expert Testimony
in England and America (Cambridge, MA: Harvard University Press, 2007).
DATA & SOCIETY - 18 -

evidence has been made admissible in courts on a case-

by-case basis since the 1850s. In the 19th century, witness
testimony had long been the gold standard for courts, despite
the fact that personal accounts were not always reliable. 31

Similarly, written historical records were also taken as fact

in courts and by historians, because they were seen as less
32

mysterious than newer technologies, such as photography. 33

“There has always been a politics of evi-

dence around audiovisual media—how
evidence changes and is changed by its
existence in cultural, social, and
political structures.”

Golan describes that, as a result of the widespread distrust

in image-capture technology, careful pains had to be taken
to provide juries and judges with transparency into the pro-
cess of image capture. For example, submitting an X-ray
into evidence in court in the 1890s required the testimony
from an expert technician to explain how a machine could
see inside a body, that certain lines were evidence of a
broken bone and not an artifact of developing the image.
Op-ed columnists reacting to the introduction of X-rays

31 John Durham Peters, “Witnessing,” Media, Culture & Society 23, no. 6 (November 1,
2001): 707–23, https://doi.org/10.1177/016344301023006002.
32 Sarah E. Igo, The Averaged American: Surveys, Citizens, and the Making of a Mass
Public (Cambridge, MA: Harvard University Press, 2008); Matthew S. Hull, Government
of Paper: The Materiality of Bureaucracy in Urban Pakistan, First edition (Berkeley:
University of California Press, 2012); Durba Ghosh, “Optimism and Political History: A
Perspective from India,” Perspectives on History 49, no. 5 (2011): 25–27; Donald Fran-
cis McKenzie, Oral Culture, Literacy & Print in Early New Zealand: The Treaty of Waitangi
(Victoria University Press, 1985); Harold Innis, Empire and Communications (Toronto,
Canada: Dundurn Press Limited, 1948); Marshall McLuhan, The Gutenberg Galaxy,
1962. Centennial Edition (Toronto: University of Toronto Press, Scholarly Publishing
Division, 2011); Walter J. Ong, Orality and Literacy, 1980. 30th Anniversary Edition
(London ; New York: Routledge, 2012).
33 Golan, Laws of Men and Laws of Nature.
DEEPFAKES AND CHEAP FAKES - 19 -

expressed fear that all doctors might have to become skilled

X-ray technicians, to carry X-ray equipment with them on
house visits.

At left, taking an X-ray image, late 1800s, published in the medical journal “Nouvelle Iconographie
de la Salpetrière.” Center, a hand deformity; at right, the same hand seen using X-ray technology.
(Wikimedia Commons)

The use of expertise to determine what media is evidence of

still happens today. Even simple, unsophisticated manipu-
lations can have dire impact on how media is interpreted
as evidence. The mutability of video in constructing legal
evidence was clearly demonstrated in 1991, during the trial
of four Los Angeles police officers for the beating of Rodney
King. The trial hinged on a central piece of video evidence
– captured by onlooker George Holliday – that shows King
being brutally beaten by the group of officers. During the 34

defense, the officers’ lawyers played the Holliday video,

slowed to a fraction of its normal speed. As they did, they
asked the officers, “Was he complying here with your order
to stay down?...” Slowed to a crawl, the video made King’s
35

involuntary physical reactions to blows appear as if he were

attempting to get up.

34 Charles Goodwin, “Professional Vision,” American Anthropologist, 1994.

35 John Fiske and Kevin Glynn, “Trials of the Postmodern,” Cultural Studies 9, no. 3 (Octo-
ber 1, 1995): 505–21, https://doi.org/10.1080/09502389500490541.
DATA & SOCIETY - 20 -

The jurors decided the police officers were not guilty of

using excessive force on King and said that the slow video in
the courtroom “made all the difference.” The slowed video 36

captured the events of March 3, 1991, but its manipulation

allowed new interpretations, a new evidentiary function—
one that maintained police sovereignty to brutalize members
of the public. 37

Visual evidence is not only interpreted in formal courtroom

settings. Within journalism, visual evidence plays a key role
in the construction of public opinion and the arrangement
of political power. Journalists also serve as experts in the
politics of evidence, deciding how to frame media as
representative of truth to the public.

Media theorist Jean Baudrillard used the term “simulacra”

to describe how broadcast journalism coverage of the First
Gulf War defined how the West perceived the conflict –
turning the daily news into around-the-clock reality TV. 38

Audiences tuned in to the spectacle of night-vision missile

launches, in retaliation to less powerful but equally visually
stunning Scud missiles launched by Iraqis, which provided
evidence that a war was raging. News media turned the
39

Gulf War into fight between evenly matched opponents,

36 Brian Stonehill, “Real Justice Exists Only in Real Time: Video: Slow-Motion Images
Should Be Barred from Trials.,” Los Angeles Times, May 18, 1992, http://articles.
latimes.com/1992-05-18/local/me-15_1_real-time.
37 Meredith D. Clark, Dorothy Bland, Jo Ann Livingston, “Lessons from #McKinney: Social
Media and the Interactive Construction of Police Brutality,” (McKinney, 2017), https://
digital.library.unt.edu/ark:/67531/metadc991008/; Kimberlé Crenshaw, “Mapping the
Margins: Intersectionality, Identity Politics, and Violence against Women of Color,”
Stanford Law Review 43, no. 6 (July 1991): 1241–99. Both these pieces are founda-
tional to the thinking of the first author in her work on how evidence is wielded by law
enforcement and by groups pushing for police accountability and frame how evidence
from official statistics to images online are used to reify structural inequality.
38 Jean Baudrillard, The Gulf War Did Not Take Place. 1993. (Indiana University Press,
1995).
39 Baudrillard, 42–45.
DEEPFAKES AND CHEAP FAKES - 21 -

but did not mention or show the uneven damage and

death toll: 20,000 dead Iraqi soldiers and an estimated
75,000 civilian casualties, compared with 286 deaths of
American soldiers. According to Baudrillard, US media
40

mythologized the conflict through the stylized misrepresen-

tation of events, justifying a colonization effort by a country
with a strong and well-resourced military. These images 41

were real images. What was manipulative was how they were
contextualized, interpreted, and broadcast around the clock
on cable television.

Gulf War coverage on CNN in January 1991 featured infrared night time reporting of missile fire,
heightening the urgency of US operations in the Persian Gulf. (Wikimedia Commons)

Simulacra describes how those who are already privileged

in the shaping of social structures make media into
evidence. From these examples, we see that evidentiary
media’s effect on society is not only produced by the tech-
nical specifics of its ability to capture the world. Instead, the

40 Baudrillard, 71-73; Robert Fisk, The Great War for Civilisation: The Conquest of the
Middle East, Reprint edition (New York: Vintage, 2007).“DCAS Reports - Persian Gulf
War Casualties - Desert Storm,” accessed April 9, 2019, https://dcas.dmdc.osd.mil/
dcas/pages/report_gulf_storm.xhtml.
41 Baudrillard, 78–83.
DATA & SOCIETY - 22 -

expert interpreter’s expression – their creation of a frame

– shapes the evidence and its effect on society. When that 42

expert interpreter works in service of the powerful, power

is consolidated, regardless of what “reality” is captured
in media.

This history shows that evidence does not speak for itself.
That is, it is not simply the representational fidelity of
media that causes it to function as evidence. Instead, media
requires social work for it to be considered as evidence. It
requires people to be designated as expert interpreters. It
requires explicit negotiations around who has the status to
interpret media as evidence. And because evidence serves
such a large role in society – it justifies incarcerations, wars,
and laws – economically, politically, and socially powerful
actors are invested in controlling those expert interpre-
tations. Put another way, new media technologies do not
inherently change how evidence works in society. What
they do is provide new opportunities for the negotiation of
expertise, and therefore power.

It is critical to understand the current panic around deep-

fakes and the drive toward technical solutions as part of this
historical trajectory. While many understand the evidence
they commonly use to inform their daily decisions as neutral,
it has never been so. Supposed evidentiary neutrality has
often worked to uphold and justify the privilege of those
who benefit from social inequality. Moving forward, it is
important for us to question whether those who claim or
construct expertise around deepfakes will do so in a way
that reinforces economic, political, and social privilege.

42 Pinch and Bijker, “The Social Construction of Facts and Artefacts.”

DEEPFAKES AND CHEAP FAKES - 23 -

“When that expert interpreter

works in service of the powerful,
power is consolidated, regardless
of what “reality” is captured
in media.”

CHEAP FAKES ON
SOCIAL MEDIA
Today, social media is experiencing a sped-up version of
the cycles of hype, panic, and closure that still and moving
images went through in the last 200 years. The camera
function became a competitive feature of mobile phones in
the mid-2000s. At the same time, social media platforms,
43

like Facebook, began amassing popularity. The combination

of mobile cameras and social media radically expanded the
possibilities of who could distribute photographic media, to
whom, and at what speeds.

Cultural scholar Henry Jenkins has responded to one aspect

of the activity around social media by identifying what he
calls “Photoshop for democracy.” He analyzes a number of
online communities that used image manipulation software
and social media to make “participatory culture [become]
participatory government.” Jenkins illustrates this with a
44

set of images that criticize political figures. He refers to an

image with faces of Howard Dean, John Kerry, and John

43 Nathan Jurgenson, The Social Photo: On Photography and Social Media (New York:
Verso, 2019).
44 Henry Jenkins, “Photoshop for Democracy,” MIT Technology Review, June 4, 2004,
https://www.technologyreview.com/s/402820/photoshop-for-democracy/.
DATA & SOCIETY - 24 -

Edwards grafted onto the Three Stooges. Another depicts

the Stooges as George W. Bush, Colin Powell, and Donald
Rumsfeld. Jenkins argues these uses of Photoshop are a
meaningful measure taken by the public to hold elected
officials accountable. Indeed, Jenkins put into language an
important component of the history of evidence we have
constructed thus far: While the already-dominant shapers
of discourse centralize power through expert interpretation,
members of the public can redistribute that power when
they have the ability to spread expressive messages at larger
scales, with less expert oversight than ever before.

But while Jenkins helps us understand what happens to

the horizon of political expression with increased scale
and speed, evidentiary media carries a different valence.
Clearly, while manipulated, the “Stooges” images make no
evidentiary claim. They are, like political cartoons or protest
signs, an expression of political beliefs. Therefore, to Jenkins’
excitement for participatory democracy, we must add that
Photoshop and other image manipulation tools can have
real consequences for individuals when expression is not
clearly distinguishable from evidence.

The remainder of this section discusses techniques of

contemporary cheap fakes: photoshopping, lookalikes,
recontextualization, and speeding and slowing moving
images. Each technical tool set described was previously
only available to experts, but in the context of technological
advancement and widespread social media use, these
technical tool sets are more accessible to amateurs and their
outputs reach larger scales at higher speeds. These examples
show how the structural power of manipulators and subjects
determine how media can be read as evidence that justifies
physical, sexual, and political violence.
DEEPFAKES AND CHEAP FAKES - 25 -

Photoshopping

Photoshop and similar image manipulation software can

be used to manipulate photographic media in countless
ways, but the stakes of such manipulations are most clear
in cases of faked pornography. People have manipulated
pornographic photos and films since the very inventions of
those technologies. The distribution of faked pornographic
45

images has become more widespread since mid-2000s 46

as Photoshop and related digital editing tools became

more accessible, and aggregated websites dedicated to
pornography became more common. Today, pornographic
cheap fakes are often played off as expression even as they
work as evidence in image-based violence against women
and other vulnerable groups. Feminist legal scholars Claire
47

McGlynn, Erica Rackley, and Ruth Hoffman refer to the act

of superimposing images of an individual’s head or body part
into pornographic content, without their consent, to make
it appear as if an individual is engaged in sexual activity,
as “sexualized photoshopping.” McGlynn et al. call this
48

type of gendered, sexualized abuse of images “image-based

sexual abuse.” The authors suggest that if a photo is
originally shared publicly, it becomes a private image when
it has been edited to depict the person in the image in a sex-
ualized way. When the image is manipulated, the individual

45 Richard Abel, Encyclopedia of Early Cinema (Taylor & Francis, 2004); John Pultz, The
Body and the Lens: Photography 1839 to the Present, First edition (New York: Harry N
Abrams Inc., 1995).
46 Clare McGlynn, Erika Rackley, and Ruth Houghton, “Beyond ‘Revenge Porn’: The Contin-
uum of Image-Based Sexual Abuse,” Feminist Legal Studies 25, no. 1 (April 1, 2017):
25–46, https://doi.org/10.1007/s10691-017-9343-2.
47 McGlynn, Rackley, and Houghton, “Beyond ‘Revenge Porn’”; Danielle Keats Citron and
Mary Anne Franks, “Criminalizing Revenge Porn,” SSRN Scholarly Paper (Rochester,
NY: Social Science Research Network, May 19, 2014), https://papers.ssrn.com/
abstract=2368946.
48 McGlynn, Rackley, and Houghton, “Beyond ‘Revenge Porn,’” 33.
DATA & SOCIETY - 26 -

pictured loses agency over their own likeness and the “por-
trayal of their sexual self.” 49

For instance, in 2016 a 17-year-old Australian woman,

Noelle Martin, found her face photoshopped onto por-
nographic images circulating on the internet, which caused
her anguish over diminished job prospects. Martin is hardly 50

alone. Countless examples of image-based violence online

link pornographic expression to existing power relations
in society. The internet at once facilitates access to por-
51

nographic materials made with and without consent, and

provides multiple unchecked and unregulated environs that
“normalize misogyny, hurtful sexuality, racism, and even
seeking revenge on female ex-partners.” The debate and 52

legislation over this phenomenon continues today and sets

the background for most legal discussion around deepfake
pornography.

Lookalikes

Image-based sexual violence can be used to inflict other types

of harm, such as the suppression of the press, civil society,

49 McGlynn, Rackley, and Houghton, “Beyond ‘Revenge Porn,’” 33.

50 “Noelle Martin Fights to Have Harmless Selfie Removed from ‘parasite’ Porn Sites - ABC
News (Australian Broadcasting Corporation),” accessed February 18, 2019, https://
www.abc.net.au/news/2016-10-12/womans-fight-to-have-harmless-selfie-removed-
from-porn-site/7924948.
51 McGlynn, Rackley, and Houghton, “Beyond ‘Revenge Porn’”; Citron and Franks, “Crimi-
nalizing Revenge Porn”; Feona Attwood, “No Money Shot? Commerce, Pornography and
New Sex Taste Cultures,” Sexualities 10, no. 4 (October 1, 2007): 441–56, https://doi.
org/10.1177/1363460707080982.
52 Walter S. DeKeseredy and Martin D. Schwartz, “Thinking Sociologically About Im-
age-Based Sexual Abuse: The Contribution of Male Peer Support Theory,” Sexualiza-
tion, Media, & Society 2, no. 4 (October 20, 2016): 237462381668469, https://doi.
org/10.1177/2374623816684692.p 6.
DEEPFAKES AND CHEAP FAKES - 27 -

or political opposition. Many of the most public examples of

this have been directed toward female politicians or activists,
often simply using similar-looking actors to depict them in
sexualized or pornographic footage. This strategy is a type of
in-camera editing that requires no technical skill, simply the
desire and ability to find a lookalike for the target.

In the Philippines in 2016, the legal counsel of President

Rodrigo Duterte used a faked sex video of Senator Leila
De Lima to justify her imprisonment, in seeming retribution
for her critique of his authoritarian rule. Similar tactics 53

have been used with female journalists who call out abuses
of power as in the case of Rana Ayyub, mentioned earlier,
who critiqued the nationalist Bharatiya Janata Party (BJP)
that ran the Indian government in Gujarat responsible for
extrajudicial murders following the Gujarat riots in 2002. 54

Since her book was originally published in 2010, and as

her reporting continues to follow abuses of power, she has
experienced ongoing harassment. In 2018, she discovered
that a pornographic video of “herself” circulated through
WhatsApp had been posted onto a BJP fan page and
circulated widely in order to ruin her credibility and force
her into silence. In reality, the pornographic fake simply
55

featured an actor who resembled her.

53 “De Lima on Sex Video: It Is Not Me,” philstar.com, accessed March 29, 2019, https://
www.philstar.com/headlines/2016/10/06/1630927/de-lima-sex-video-it-not-me.
54 Rana Ayyub, Gujarat Files: Anatomy of a Cover Up (India: CreateSpace Independent
Publishing Platform, 2016).
55 Ayyub, “I Was The Victim of a Deepfake Porn Plot Intended To Silence Me.”
DATA & SOCIETY - 28 -

Re-contextualizing

Perhaps the easiest mode of producing a cheap fake is

simply cutting together existing footage and spreading
it under false pretenses. For example, in April 2018, a
falsified BBC report began circulating on WhatsApp,
presenting a false story of nuclear escalation between
NATO and Russia. The shared clip was four minutes
56

long, and featured nuclear mushroom clouds, the Queen’s

evacuation from Buckingham Palace, a Russian ship firing
missiles, and NATO aircraft being launched in retaliation.
As the clip spread, alarmed viewers began sending messages
to the BBC, which issued a statement explaining that the 57

clip was cut together from YouTube footage uploaded in

2016 by the Benchmarking Assessment Group, an Irish
marketing company. The original footage was a 30-minute
video designed as a psychometric test for reactions during a
disaster scenario. Outside of the focus group, the video was
never intended to be taken seriously. But, a still-unknown
manipulator was able to re-contextualize and disseminate it
through WhatsApp as an imminent threat.

WhatsApp’s secure encryption model gives the service an

air of authenticity—if one is only receiving information from
verified contacts, then everything should be trustworthy.
This feeling of authenticity is precisely what gets exploited
by manipulators. Once a video like BBC NATO does make

56 Martin Coulter, “BBC Issues Warning after Fake News Clips Claiming NATO and Russia
at War Spread through Africa and Asia,” London Evening Standard, accessed April 25,
2018, https://www.standard.co.uk/news/uk/bbc-issues-warning-after-fake-news-
clips-claiming-nato-and-russia-at-war-spread-through-africa-and-a3818466.html;
“BBC Forced to Deny Outbreak of Nuclear War after Fake News Clip Goes Viral,” The
Independent, April 20, 2018, https://www.independent.co.uk/news/media/bbc-forced-
deny-fake-news-nuclear-war-viral-video-russia-nato-a8313896.html.
57 Chris Bell, “No, the BBC Is Not Reporting the End of the World,” April 19, 2018, https://
www.bbc.com/news/blogs-trending-43822718.
DEEPFAKES AND CHEAP FAKES - 29 -

“Platforms are not equipped to

handle fake videos even when
these videos are cheap fakes—not
generated with sophisticated
artificial intelligence.”

it onto a node of one of these hidden social networks, new

recipients can feel a political, emotional, or social duty to
share information received from their family or friend
groups, regardless what judgment they may have about the
factual content of the information. 58

This and other more prominent recent examples where

WhatsApp has been implicated in spreading disinformation
illustrate that in the contemporary climate, platforms
struggle to moderate problematic content traveling at large
scales. Platforms are not equipped to handle fake videos
59

even when these videos are cheap fakes—not generated with

sophisticated artificial intelligence. One method platforms
have used to attempt to address this is to create encrypted
social media networks like WhatsApp that allow sharing

58 Daniel Kreiss, “The Fragmenting of the Civil Sphere: How Partisan Identity Shapes the
Moral Evaluation of Candidates and Epistemology,” American Journal of Cultural Sociol-
ogy 5, no. 3 (October 2017): 443–59, https://doi.org/10.1057/s41290-017-0039-5;
Santanu Chakrabarti, Lucile Stengel, and Sapna Solanki, “Fake News and the Ordinary
Citizen in India,” Beyond Fake News (London: British Broadcasting Corp., November 12,
2018), http://downloads.bbc.co.uk/mediacentre/duty-identity-credibility.pdf.
59 Jason Koebler, Derek Mead, and Joseph Cox, “Here’s How Facebook Is Trying to Moder-
ate Its Two Billion Users,” Motherboard (blog), August 23, 2018, https://motherboard.
vice.com/en_us/article/xwk9zd/how-facebook-content-moderation-works; Sarah T.
Roberts, Behind the Screen, Yale University Press (New Haven, CT: Yale University
Press, 2019), https://yalebooks.yale.edu/book/9780300235883/behind-screen.
DATA & SOCIETY - 30 -

between small groups. They claim this will promote

60

authentic communication between trusted parties, and

thus this technical solution would discourage the spread of
disinformation.However, this has also enabled disinformation
to enjoy the cover of hidden virality in which trusted parties
in private networks link quickly to wider networks of users
while remaining impenetrable to outside oversight.

Speeding and Slowing

Recalling how the video of Rodney King’s beating was

slowed in court to attempt to sow doubt in jurors’ minds,
we can see that the speed of a video has a large effect on
its interpretation. We were recently reminded of this in
May 2019, when a faked video appeared to depict House
Speaker Nancy Pelosi drunkenly talking about Donald
Trump caused waves of public outrage. It was amplified by
61

Trump and other prominent pundits. Placing the “drunk 62

Pelosi” clip side by side with the original C-SPAN footage 63

make the manipulation clear: a roughly 75% decrease in

speed, which could have easily been accomplished with any
consumer video editing program, from Adobe Premiere
to iMovie.

60 Mark Zuckerberg, “A Privacy-Focused Vision for Social Networking,” Facebook (blog),

March 6, 2019, https://www.facebook.com/notes/mark-zuckerberg/a-privacy-fo-
cused-vision-for-social-networking/10156700570096634/.
61 The Guardian, “Real v Fake: Debunking the ‘drunk’ Nancy Pelosi Footage,” The Guardian,
May 24, 2019, https://www.theguardian.com/us-news/video/2019/may/24/real-v-
fake-debunking-the-drunk-nancy-pelosi-footage-video.
62 Contemptor, Team Trump Goes All In on Pushing Doctored “Drunk” Pelosi Videos,
accessed June 7, 2019, https://www.youtube.com/watch?time_continue=43&v=FGB-
c3PJ40oM.
63 The New York Times, “Edited Pelosi Video vs. the Original: A Side-by-Side Compari-
son,” The New York Times, May 24, 2019, https://www.nytimes.com/video/us/poli-
tics/100000006525055/pelosi-video-doctored.html.
DEEPFAKES AND CHEAP FAKES - 31 -

Not so long ago, on November 7, 2018, another widely

publicized struggle over evidentiary media came from the
social media account of the White House press secretary.
Following a heated exchange between Donald Trump and
CNN reporter Jim Acosta, the White House announced
that it would be revoking Acosta’s press credentials after
he struck an intern while reaching for a microphone. Along
with the announcement, White House press secretary Sarah
Sanders posted an altered version of the original C-SPAN
footage that showed the interaction between Acosta and
the intern. 64

Within hours social media users identified the edited

version of the Acosta video as one created by Paul Joseph
65

Watson, editor and writer for conspiracy site Infowars.

66

The video Watson produced was taken from the original

C-SPAN footage, but zoomed in on the moment of contact,
apparently spliced in with a low-frame rate GIF. To many 67

viewers and experts, the resulting interaction between Acosta

64 Sarah Sanders, "We Stand by Our Decision to Revoke This Individual’s Hard Pass.
We Will Not Tolerate the Inappropriate Behavior Clearly Documented in This
Video," Twitter, accessed November 8, 2019, https://twitter.com/PressSec/sta-
tus/1060374680991883265.
65 Mike Stuchbery, "Oh Lord, in His Fevered Attempts to Claim He Didn’t Edit the Acosta
Video, @PrisonPlanet Posted a Picture of His Vegas Timeline — That Shows He Added
Frames...," Twitter, accessed November 8, 2019, https://twitter.com/MikeStuchbery_/
status/1060535993525264384; Rafael Shimunov, "1) Took @PressSec Sarah Sand-
ers’’ Video of Briefing 2) Tinted Red and Made Transparent over CSPAN Video 3) Red
Motion Is When They Doctored Video Speed 4) Sped up to Make Jim Acosta’s Motion
Look like a Chop 5) I’ve Edited Video for 15+ Years 6) The White House Doctored
It," Twitter, accessed November 8, 2019, https://twitter.com/rafaelshimunov/sta-
tus/1060450557817708544.
66 Paul Joseph Watson, "Sarah Sanders Claims Jim Acosta ’Placed His Hands on
a Woman.’ Jim Acosta Says This Is a ‘Lie’. Here’s the Video. You Be the Judge,"
Twitter, accessed November 8, 2019, https://twitter.com/PrisonPlanet/sta-
tus/1060344443616800768.
67 Paris Martineau, “How an InfoWars Video Became a White House Tweet,” Wired,
November 8, 2018, https://www.wired.com/story/infowars-video-white-house-cnn-jim-
acosta-tweet/.
DATA & SOCIETY - 32 -

and the intern appeared more forceful in the Watson/Sand-

ers version.

All of the examples above – whether photoshopping, stand-ins,

or conventional editing – show that politically dangerous
AV manipulation in the form of cheap fakes already exists.
Society is already struggling to mitigate the harm of
“traditional” techniques: changes in speed, cutting apart or
together, circulating in new, vulnerable contexts. It is into
this landscape that so-called deepfakes emerge.

Deepfakes Present and Future

Much of the current coverage of deepfakes strikes the same

tone as Franklin Foer’s assessment of the phenomenon in
early 2018: “We’ll shortly live in a world where our eyes
routinely deceive us … we’re not so far from the collapse
of reality.” While video has always been susceptible to
68

manipulation, deepfakes have produced a refreshed panic.

But it is important to not let this general panic shape the
discourse around AV manipulation. Instead, we should pay
close attention to the full range of such manipulations, to
understand how these technologies are actually being inter-
preted and acted upon.

The range of these newly widespread, technically advanced

modes of AV manipulation include virtual performances,
face swapping, and lip-synching or voice synthesis. Some
instances may include all of these techniques, and others
may only use one. While at present amateurs may certainly
participate in the production of video content, the technical

68 Foer, “The Era of Fake Video Begins.”

DEEPFAKES AND CHEAP FAKES - 33 -

barriers to entry are still high, as is the expertise needed to

produce the faked videos.

During our survey of audiovisual manipulation online, 69

and background interviews with producers of these videos, 70

we found evidence of a range of communities of practice 71

represented among those producing and discussing these

deepfake videos. There are professional effects artists who
use sophisticated and expensive equipment to generate
material for Hollywood movies and other big budget media
spectacles. Then there is a loose collection of hobbyist and
tech enthusiasts who organize online to reproduce these
expensive AI techniques with free and consumer-grade
technology. Finally, a subset of this non-professional group
produces and consumes problematic content—the most
prominent of which is manipulated pornography.

The lines between these communities, however, are fuzzy

at best. And online, a wide variety of media labeled as
AI-generated is not actually made with AI but is presented
alongside other videos that are in YouTube playlists or
Reddit threads. In our interviews with producers across the
range of communities of practice, we identified a driving,

69 From January 2018 to June 2019, we collected and analyzed over 70 examples of
deepfake videos and images from YouTube, Instagram, Twitter, TikTok, PornHub, Arxiv,
and Mr. Deepfakes.
70 We completed eleven semi-structured interviews between September and December
2018.
71 This term is drawn from sociological and anthropological literature on knowledge pro-
duction refer to groups of individuals organize both on and off-line to experiment, learn,
and professionalize around practice-based activities. Jean Lave and Etienne Wenger,
Situated Learning: Legitimate Peripheral Participation (Cambridge University Press,
1991); Line Dubé, Anne Bourhis, and Réal Jacob, “The Impact of Structuring Character-
istics on the Launching of Virtual Communities of Practice,” Journal of Organizational
Change Management, April 1, 2005, https://doi.org/10.1108/09534810510589570.
DATA & SOCIETY - 34 -

optimistic belief that technological advancement is neces-

sarily good, which minimizes the potential harm brought
72

by new technologies. What remains important to articulate

is where to draw a line between experimental, expressive
play and new forms of image-based abuse, whether sexual,
political, or otherwise.

Virtual Performances

Initially, the public reception of AI-generated manipulations

was shaped by a small number of prominent examples in
major Hollywood films. Before deepfakes, Hollywood effects
artists were using computer-generated imagery that faked
performances. For instance, Tom Hanks’ titular character
Forrest Gump interacting with long-dead individuals by
inserting Gump into already existing footage using digital
compositing.

Recent Hollywood movies are able to create even more

ambitious virtual performances, not just combining two
sources of footage, but generating entirely new representa-
tions of face, voice, and body. Most notable among these are
recent films in the Star Wars franchise. The effects artists
at Industrial Light and Magic created a younger version
of actor Carrie Fischer for 2017’s Rogue One and a post-
humous version of her for 2018’s The Last Jedi. Indeed,
even in amateur circles, Star Wars content is particularly
popular. A popular Reddit user with the name “derpfakes,”
also created their own version of Fischer’s scenes in Rogue

72 This observation from our interviews is backed up in interviews from news coverage,
for example, in Megan Farokhmanesh, “Is It Legal to Swap Someone’s Face into
Porn without Consent?” The Verge, January 30, 2018, https://www.theverge.
com/2018/1/30/16945494/deepfakes-porn-face-swap-legal.
DEEPFAKES AND CHEAP FAKES - 35 -

One – highlighting just how close free and open source tools
can get to the “expensive and labor intensive” ones used by
Hollywood. 73

Face Swapping

Much of the activity of deepfake-focused communities has

involved specific technologies for face swapping. Original
software like FakeApp, FaceSwap, and DeepFace Lab
allow users to map one actor’s face onto another actor’s
performance. Several software users have used this tech-
nique to reproduce or comment on various Hollywood
versions, such as an October 2018 video that mapped a
young Harrison Ford’s face onto Alden Ehrenreich’s body
in Solo: A Star Wars Story.

The most publicly visible examples of open source face

swapping, however, have been a series of videos made by
projecting female actors’ faces onto the bodies of perform-
ers in pornography, using software like FakeApp and Adobe
After Effects. In November of 2017, for instance, one user
uploaded such a video using Gal Gadot’s face to Reddit.
In April 2018, pornographic deepfakes were banned from
Reddit, but the videos remain available on sites like Mr.
Deepfakes and Pornhub. In an interview with Mother-
board, the anonymous creator of the Gal Gadot video said,
“Every technology can be used with bad motivations, and it’s
impossible to stop that…. The main difference is how easy

73 Insight gleaned from exchange with derpfakes on November 14, 2018, and derpfake
video comments on YouTube and Reddit.
DATA & SOCIETY - 36 -

[it is] to do that by everyone. I don’t think it’s a bad thing

for more average people [to] engage in machine learning
research.” Another Redditor posted, “[T]he work that we
74

create here in this community is not with malicious intent.

Quite the opposite. We are painting with revolutionary, ex-
perimental technology, one that could quite possibly shape
the future of media and creative design.” In this example, 75

this new “revolutionary technology” is firmly situated in a

misogynist environment in which its practitioners mobilize
the bodily identity of women, without their consent, as an
unproblematic creative practice. 76

Indeed, it is not a bad thing for people to engage in machine

learning research. It is not inherently wrong to make,
circulate, and consume pornography. However, problems
arise when experimenting with machine learning is used in
the service of non-consensual objectification and harassment.
Regardless of the intention of those who create videos, the
effects wrought by technological play matters. AI-generated
pornographic videos highlight a set of interconnected

74 Samantha Cole, “AI-Assisted Fake Porn Is Here and We’re All Fucked,” Motherboard,
December 11, 2017, https://motherboard.vice.com/en_us/article/gydydm/gal-gadot-
fake-ai-porn.
75 Quote from Reddit user Quiet Horse, now deleted from Reddit, in Megan Farokhmanesh,
“Is It Legal to Swap Someone’s Face into Porn without Consent?,” The Verge, January
30, 2018, https://www.theverge.com/2018/1/30/16945494/deepfakes-porn-face-
swap-legal.
76 McGlynn, Rackley, and Houghton, “Beyond ‘Revenge Porn’”; Franks, “Unwilling Avatars”;
Citron, “Addressing Cyber Harassment”; Citron and Franks, “Criminalizing Revenge
Porn.” While this is generally true in online porn communities, pornographic practice
is by no means monolithic. The production of pornography by and for women, LGBTIQ
individuals, and people of color highlights that these groups have agency in producing
and enjoying their own counter-hegemonic pornographic content. The higher degree
of visibility and access to sexual culture for these groups might be described as
liberatory, or more radically democratic. In this vein, Feona Attwood suggests that
altporn in the form of SuicideGirls and Nerve.com represent “participatory cultures
which serve corporate and community needs” which can be understood as radically
democratic. (Feona Attwood, “No Money Shot? Commerce, Pornography and New
Sex Taste Cultures,” Sexualities 10, no. 4 (October 1, 2007): 441–56, https://doi.
org/10.1177/1363460707080982.)
DEEPFAKES AND CHEAP FAKES - 37 -

“Regardless of the intention

of those who create videos,
the effects wrought by
technological play matters.”

problems regarding the “democratization” of a technical

skill set. That the new technology’s first notable amateur
output depicts one woman engaged in a sex act without her
consent, while also erasing the original female actor in the
original video highlights how the expressive use of technol-
ogy can be wielded as harassment or harm individuals who
do not command public attention, and cannot command
the resources necessary to refute falsehoods.

Lip-synching and Voice Synthesis

Another strand of deepfake production focuses not on

dynamic virtual performances of face swapping but on
the presentation of new speech content for recognizable
figures. In November 2017, Stanford researchers published
a paper and model for Face 2 Face, a RNN-based video
77

production model that allows third parties the ability to “put

words in the mouths” of public figures like Barak Obama –
in real time. The software captures the third party’s facial
expressions as they talk into a webcam and then melds those
movements with the person’s face in the original video.

77 Thies et al, “Face2Face.”

DATA & SOCIETY - 38 -

In February 2018, Jordan Peele teamed with BuzzFeed to

produce a “deepfake” video in which Obama appears to
provide a public service announcement about the dangers
of the new technology. Like the derpfakes example men-
78

tioned above, the video was produced using Adobe After

Effects, FakeApp, and a team of creative professionals to
swap in Peele’s voice as he spoke in a voice that many know
as his Obama impression. It took hours for the FakeApp
79

application, trained by a human, to render a video in which

Obama’s mouth moves along with the Peele’s voice.

Versions of these techniques are part of the proprietary soft-

ware that was used to create the Zuckerberg fake from our
introduction. Their bespoke video-faking tool used audio
80

and video files from Zuckerberg’s Senate testimony from

April 2018 to create a new virtual performance. This pro-
prietary model was in some senses more sophisticated than
the GAN or RNN-generated video production generated by
computer science labs, as it incorporated more audio and
video training data. The artists who built it maintain that
the VDR model but was built on the work featured at 2017
SIGGRAPH conference coming from a computer science
81

lab at the University of Washington on a project led by Su-

pasorn Suwajanakorn and the Face2Face tool. 82

78 Craig Silverman, “How To Spot a DeepFake Like The Barack Obama-Jordan Peele Video,”
accessed April 18, 2018, https://www.buzzfeed.com/craigsilverman/obama-jordan-
peele-deepfake-video-debunk-buzzfeed.
79 James Vincent, “Watch Jordan Peele Use AI to Make Barack Obama Deliver a
PSA about Fake News,” The Verge, April 17, 2018, https://www.theverge.com/
tldr/2018/4/17/17247334/ai-fake-news-video-barack-obama-jordan-peele-buzzfeed.
80 Cade Metz, “A Fake Zuckerberg Video Challenges Facebook’s Rules,” The New York
Times, June 11, 2019, sec. Technology, https://www.nytimes.com/2019/06/11/tech-
nology/fake-zuckerberg-video-facebook.html.
81 The Association for Computing Machinery is a professional organization that supports
SIGGRAPH, a special interest group on computer graphics and interactivity.
82 “Canny AI: Imagine World Leaders Singing" Fxguide, accessed June 24, 2019, https://
www.fxguide.com/featured/canny-ai-imagine-world-leaders-singing/.
DATA & SOCIETY - 39 -

“But as these techniques become more

accessible, the risk of harm to private
figures will increase, especially
those who are politically, socially, or
economically vulnerable.”
DEEPFAKES AND CHEAP FAKES - 40 -

CONCLUSION
The problems wrought by audiovisual manipulation are
many and difficult to remedy, but we can work to reduce risk
and mitigate harm. This report has shown that individual
women, journalists, and others who are antagonistic to those
who hold economic and political power are going to be the
first to confront the politics of evidence in a “post-truth”
world. In these scenarios, questions of evidence are key:
Who should we trust, and on what basis?

The high-profile deepfakes cases discussed above are just

a fraction of the examples that exist. And more are being
created every day. So far, news coverage and public discussion
has paid the most attention to those cases that involve the
imitation – whether virtual performance, face swap, or voice
synthesis – of celebrities, politicians, and public figures. But
as these techniques become more accessible, the risk of
harm to private figures will increase, especially those who
are politically, socially, or economically vulnerable.

At present, anyone with a social media profile is fair game to

be faked. Social media relies on forms of soft consent, buried
in the terms of service that users habitually click through.
That soft consent is now being used to justify the collec-
tion of data far beyond web surfing or ad clicking habits.
DATA & SOCIETY - 41 -

It involves data that captures our faces and bodies —a 83

condition that demands greater transparency and true

consent. Social media functions as vast troves of images of
84

faces and bodies, and amateur programmers have already

built bespoke scrapers to extract these images as training
data. 85

There already have been numerous cases of AV manipulation

techniques used against private individuals: to settle personal
vendettas, exact blackmail, and trick people into participat-
86 87

ing in personalized financial scams. People are already faking

88

their classmates and co-workers in amateur AI-generated

83 Hamza Shaban, “A Google App That Matches Your Face to Artwork Is Wildly Popular.
It’s Also Raising Privacy Concerns.,” Washington Post. January 17, 2018, https://
www.washingtonpost.com/news/the-switch/wp/2018/01/16/google-app-that-
matches-your-face-to-artwork-is-wildly-popular-its-also-raising-privacy-concerns/;
Cole and Maiberg, “People Are Using AI to Create Fake Porn of Their Friends and
Classmates”; Natasha Singer, “Facebook’s Push for Facial Recognition Prompts Privacy
Alarms,” The New York Times, July 11, 2018, sec. Technology, https://www.nytimes.
com/2018/07/09/technology/facebook-facial-recognition-privacy.html.
84 Pavel Korshunov and Sebastien Marcel, “DeepFakes: A New Threat to Face Recog-
nition? Assessment and Detection,” ArXiv:1812.08685 [Cs], December 20, 2018,
http://arxiv.org/abs/1812.08685; Owen Hughes, “Is FaceApp Safe? Don’t Be so Quick
to Share Your Face Online, Warn Security Experts,” International Business Times UK,
April 27, 2017, https://www.ibtimes.co.uk/faceapp-safe-dont-be-so-quick-share-
your-face-online-warn-security-experts-1618975; Robert Chesney and Danielle
Keats Citron, “Deep Fakes: A Looming Challenge for Privacy, Democracy, and National
Security,” SSRN Scholarly Paper (Rochester, NY: Social Science Research Network, July
14, 2018), https://papers.ssrn.com/abstract=3213954; M. C. Elish and danah boyd,
“Situating Methods in the Magic of Big Data and AI,” Communication Monographs 85,
no. 1 (January 2, 2018): 57–80, https://doi.org/10.1080/03637751.2017.1375130.
85 Instagram Scraper and the Chrome extension DownAlbum are among many open source
tools that allow anyone to easily download images from publicly available Facebook or
Instagram accounts to do many things, including to generate training data for fakes.
86 Cole and Maiberg, “People Are Using AI to Create Fake Porn of Their Friends and Class-
mates.”
87 “Bella Thorne Steals Hacker’s Thunder, Publishes Nude Photos Herself,” Naked Security
(blog), June 18, 2019, https://nakedsecurity.sophos.com/2019/06/18/bella-thorne-
steals-hackers-thunder-publishes-nude-photos-herself/; “Beware the Fake Facebook
Sirens That Flirt You into Sextortion,” Naked Security (blog), March 23, 2018, https://
nakedsecurity.sophos.com/2018/03/23/beware-the-fake-facebook-sirens-that-flirt-
you-into-sextortion/.
88 BBC News, ”Fake Voices ‘Help Cyber-Crooks Steal Cash,’” BBC, July 8, 2019, https://
www.bbc.com/news/technology-48908736; Stacey Colino, “Don’t Fall Victim to the
Grandparents Scam,” AARP, April 18, 2018, http://www.aarp.org/money/scams-fraud/
info-2018/grandparent-scam-scenarios.html.
DEEPFAKES AND CHEAP FAKES - 42 -

pornography in online porn sites. Faked videos featuring

89

private individuals carry great consequence for the ways we

conceive of the evidentiary relationship between bodies and
audiovisual content, expertise, and structural inequalities.

The question of how to effectively understand and address

these images is non-trivial. Existing legal and platform policy
cannot effectively address the harm that has and will arise
from AV manipulation techniques. Even in the US there are
many legal paths one could take to stop fake pornographic
or otherwise harmful videos from being disseminated, but 90

there is no law to undo the unique damage these images

unleash. Moreover, the misuse of one’s likeness is expensive
91

to address through the legal system. In the US, laws like

Section 230 of Title 47 of the 1996 Communications
Decency Act are intended to remedy instances where courts
fall short by allowing platforms to respond to users’ requests
to remove harmful content. But even Section 230 has
proven incapable of fully addressing the range of problems
that have been thrown up by massive platform use. 92

The stakes are high for accurately understanding AV

manipulation. The fear of a hyperreality spawned by deep-
fakes can be used to foreclose other possible interventions.

89 Drew Hartwell, “Fake-Porn Videos Are Being Weaponized to Harass and Humiliate
Women: ‘Everybody Is a Potential Target,’” Washington Post, accessed April 5, 2019,
https://www.washingtonpost.com/technology/2018/12/30/fake-porn-videos-are-be-
ing-weaponized-harass-humiliate-women-everybody-is-potential-target/; Samantha
Cole and Emanuel Maiberg, “People Are Using AI to Create Fake Porn of Their Friends
and Classmates,” Motherboard (blog), January 26, 2018, https://motherboard.vice.
com/en_us/article/ev5eba/ai-fake-porn-of-friends-deepfakes.
90 “47 U.S. Code § 230 - Protection for Private Blocking and Screening of Offensive Mate-
rial,” Legal Information Institute, accessed March 1, 2019, https://www.law.cornell.edu/
uscode/text/47/230.
91 Franks, “The Desert of the Unreal”; Citron and Franks, “Criminalizing Revenge Porn.”
92 47 U.S. Code § 230 - Protection for Private Blocking and Screening of Offensive
Material,”; “Section 230 Protections,” Electronic Frontier Foundation, August 25, 2011,
https://www.eff.org/issues/bloggers/legal/liability/230.; Danielle Citron and Benjamin
Wittes, “The Internet Will Not Break: Denying Bad Samaritans § 230 Immunity,” Ford-
ham Law Review 86, no. 2 (November 1, 2017): 401; Citron and Franks, “Criminalizing
Revenge Porn.”
DATA & SOCIETY - 43 -

Sociologist and science studies scholar Donald Mackenzie

has written extensively about the consequences of creating
technology without critically assessing its potential for harm.
His research led him to ask a peculiar question: “Can we
uninvent the bomb?” No matter how much we may want
93

to uninvent the bomb, we cannot. Once new technology is

generated, it must be reckoned with as a social and cultural
phenomenon.

Nonetheless, many are focused on uninventing the “bomb”

they see in AI-generated video production. In the wake
of alarmist news coverage, multiple popular press articles
have proposed programs of improved literacy, arguing that 94

people must take steps to equip themselves to detect fakes.

Technologists are rushing to undo the coming “end of the
world” through a range of new technical solutions. Two of 95

the leading technological approaches are: automated fake

detection and distributed verification technologies to verify
all online and even offline interaction. Facebook has been
one of the loudest voices advocating automated systems to
detect AI-generated manipulations. The start-up TruePic
96 97

uses blockchain technology to verify press-captured images

and videos. The US Defense Advanced Research Projects
Agency (DARPA) funded MediFor to run diagnostics of

93 Donald Mackenzie, “Uninventing the Bomb?” Medicine and War 12, no. 3 (October 22,
2007): 202–11, https://doi.org/10.1080/13623699608409285.
94 Natasha Bernal, “Journalists at Wall Street Journal to Be Taught to Identify
‘Deepfakes,’” The Telegraph, November 15, 2018, https://www.telegraph.co.uk/
technology/2018/11/15/journalists-wall-street-journal-taught-identify-deepfakes/;
Suzanne Sunne, “What to Watch for in the Coming Wave of “Deep Fake" Videos,” Global
Investigative Journalism Network, May 28, 2018, https://gijn.org/2018/05/28/what-
to-watch-for-in-the-coming-wave-of-deep-fake-videos/; Siwei Lyu, “The Best Defense
against Deepfake AI Might Be ... Blinking,” Fast Company, August 31, 2018, https://
www.fastcompany.com/90230076/the-best-defense-against-deepfakes-ai-might-be-
blinking.
95 Martínez, “The Blockchain Solution to Our Deepfake Problems”; Lyu, “The Best Defense
against Deepfake AI Might Be... Blinking”; Greene, “Researchers Developed an AI to
Detect DeepFakes”; “Exposing Fake Videos.”
96 Facebook, “Expanding Fact-Checking to Photos and Videos | Facebook Newsroom,”
Facebook, September 13, 2018, https://newsroom.fb.com/news/2018/09/expand-
ing-fact-checking/.
97 TruePic, “Photos and Videos You Can Trust.”
DEEPFAKES AND CHEAP FAKES - 44 -

videos to determine differences in pixels within videos that

signal fakery. Other such verification proposals involve
98

doing away with net neutrality in the interest of marking

the provenance of all packets carrying video data so they are
easily surveilled to determine real-life identities of creators. 99

At present, there are four legislative bills in nascent stages

of consideration to force individuals to label content that
they have manipulated, and to allow the federal govern-
ment to impose massive fines against individuals who have
manipulated content deemed harmful. 100

Those forwarding these totalizing technical security steps

hold up the alleged uniqueness of deepfakes as justification.
Locating deepfakes in the longer history of the politics of
evidence, however, allows us to see that: a) decisions over
what counts as “evidence” have historically been a crucial

98 Matt Turek, “Media Forensics,” Defense Advanced Research Projects MediFor, 2018,
https://www.darpa.mil/program/media-forensics.
99 H. R. Hasan and K. Salah, “Combating Deepfake Videos Using Blockchain and Smart
Contracts,” IEEE Access 7 (2019): 41596–606, https://doi.org/10.1109/AC-
CESS.2019.2905689.
100 Yvette Clarke, “H.R.3230 Defending Each and Every Person from False Appearances by
Keeping Exploitation Subject to Accountability Act of 2019,” webpage, June 24, 2019,
https://www.congress.gov/bill/116th-congress/house-bill/3230; Ben Sasse, “S.3805
Malicious Deep Fake Prohibition Act of 2018,” webpage, December 21, 2018, https://
www.congress.gov/bill/115th-congress/senate-bill/3805/text; Tim Grayson “AB-1280
Crimes: Deceptive Recordings.,” webpage, April 22, 2019, https://leginfo.legislature.
ca.gov/faces/billTextClient.xhtml?bill_id=201920200AB1280. Bryan Hughes “SB 751,”
webpage, February 11. 2019. https://leginfo.legislature.ca.gov/faces/billTextClient.
xhtml?bill_id=201920200AB1280; In addition, Virginia has expanded existing revenge
porn law to cover fake nude images. Marcus Simon, “HB 2678 Amendment” webpage,
February 11, 2019. https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?-
bill_id=201920200AB1280
DATA & SOCIETY - 45 -

tool in defending the privilege of the already powerful; b) 101

the violence of AV manipulation can only be addressed by

a combination of technical and social solutions; c) public
agency over these technologies cannot be realized without
addressing structural inequality; and d) the violence of
AV manipulation will not be curtailed unless groups most
vulnerable to that violence are able to influence public
media systems.

Scholars, journalists, and activists are already considering

how we might reconfigure technical systems, their
ownership, and regulatory bodies to consider the politics of
evidence. Beyond the “social” policy solutions mentioned
above that penalize individuals for harmful behavior, more
encompassing solutions might be to enact federal measures
on corporations to encourage them to more meaningfully
address the fallout from their massive gains in the past 15
years. Information scholar Sarah T. Roberts has suggested
that contrary to the practice as it exists at present, the work
of content moderators should be valued and supported by
platform companies as one of many necessary steps to build
community that fosters pro-social values that go beyond

101 Literature in both science and technology studies (STS) and media studies discuss
how evidence in the form of formalized and informal data and discourses around these
types of evidence are shaped by social, political and cultural structures that reflect
and reify the status quo. At the same time, evidence can be mobilized by less powerful
entities to make justice-based arguments about the distribution of harms and power in
society. But even then, community-based counter-data is always judged according to
rules set by dominant discourses around truth that exist to protect the status quo of
privilege. The authors have published collaborative work on the politics of evidence as
they relate to community-based counter-data practices. Britt S. Paris et al., “Pursuing
a Toxic Agenda,” September 2017, https://100days.envirodatagov.org/pursuing-tox-
ic-agenda/; Morgan Currie, Joan Donovan, and Britt S. Paris, “Preserving for a More
Just Future: Tactics of Activist Data Archiving,” in Data Science Landscape: Towards
Research Standards and Protocols, ed. Usha Mujoo Munshi and Neeta Verma, Studies
in Big Data 38 (Singapore: Springer, 2018), 67–78; Britt S. Paris and Jennifer Pierre,
“Naming Experience: Registering Resistance and Mobilizing Change with Qualitative
Tools,” InterActions: Journal of Education and Information Studies 13, no. 1 (2017),
https://escholarship.org/uc/item/02d9w4qd; Britt S. Paris and Jennifer Pierre, “Bad
Data — Cultural Anthropology,” Cultural Anthropology Field Insights, April 28, 2017,
https://culanth.org/fieldsights/1107-bad-data.\\uc0\\u8221{} in {\\i{}Data Science
Landscape: Towards Research Standards and Protocols}, ed. Usha Mujoo Munshi and
Neeta Verma, Studies in Big Data 38 (Singapore: Springer, 2018
DEEPFAKES AND CHEAP FAKES - 46 -

profit motives. Feminist legal scholar Danielle Citron has

102

suggested tech companies could foster digital citizenship

by building in thoughtful mechanisms to discourage
user-generated content that is false or misleading, that
harasses, or negatively targets specific groups by developing
more rigorously pro-social terms of service (TOS) for
taking down and labeling nefarious content, and following
these TOS in a way that is transparent and open to public
oversight. Finally, Citron and legal scholar Mary Ann
103

Franks have suggested the necessity of reconsidering how

to hold platform intermediaries legally accountable for
spreading tortious content in ways that would promote
more effective cross-platform injunctions of harmful
content. These solutions taken together begin to show
104

a long and difficult way forward, but one that is,

nonetheless, possible.

Regardless of the difficulty, it is crucial to consider these larger

problems that contribute to complications of AI-generated
media as they enter an already troubled media landscape.
While we cannot start again, Mackenzie’s statement of the
atomic bomb is apt when thinking about what to do about
audiovisual manipulation: “We do not know, and should not
pretend to know, at this point in time, the solution to all

102 Sarah T. Roberts, Behind the Screen (New Haven, CT: Yale University Press, 2019),
https://yalebooks.yale.edu/book/9780300235883/behind-screen.
103 Danielle Keats Citron, Hate Crimes in Cyberspace, Reprint edition (Cambridge, MA:
Harvard University Press, 2016); Danielle Keats Citron and Helen Norton, “Intermedi-
aries and Hate Speech: Fostering Digital Citizenship for Our Information Age,” Boston
University Law Review 91 (2011): 1435.Reprint edition (Place of publication not
identified: Harvard University Press, 2016
104 Danielle Citron and Benjamin Wittes, “The Internet Will Not Break: Denying Bad Samar-
itans § 230 Immunity,” Fordham Law Review 86, no. 2 (November 1, 2017): 401; Mary
Anne Franks and Danielle Citron, “Criminalizing Revenge Porn,” Wake Forest Law Review
49 (2014): 345–92; Mary Anne Franks, “Drafting an Effective ‘Revenge Porn’ Law: A
Guide for Legislators” (Cyber Civil Rights Initiative, 2016), https://www.cybercivilrights.
org/guide-to-legislation/; Christopher Zara, “The Most Important Law in Tech Has a
Problem | Backchannel,” Wired, January 3, 2017, https://www.wired.com/2017/01/the-
most-important-law-in-tech-has-a-problem/.
DATA & SOCIETY - 47 -

the problems it may throw up.” What we do know is that

105

limiting the harm of AV manipulation will require an under-

standing of the history of evidence, and the social processes
that produce truth, in order to avoid new consolidations of
power for those who can claim an exclusive expertise.

105 Donald Mackenzie, “Uninventing the Bomb?” Medicine and War 12, no. 3 (October 22,
2007): 202–11, https://doi.org/10.1080/13623699608409285, 120

ACKNOWLEDGMENTS
We are grateful to everyone at Data & Society Research Institute for their generous support,
guidance, and helpful feedback through the duration of this project. In particular, we’d like to
thank Patrick Davison for his thoughtful editing and revisions. We are forever thankful for the
collaboration and ongoing discussions with members of Data & Society’s Media Manipulation
Initiative, past and present. Thanks to interviewees for graciously taking time to provide
background information on this work. Special thanks to external reviewers Fenwick McKelvey,
Tiziana Terranova, Aayush Bansal, and Samuel Gursky for their generous feedback on the draft.
A big thank you to our friends, family, and partners for supporting us in so many ways through this
project and during everything else.

DATA & SOCIETY

Data & Society is an independent nonprofit research institute that advances new frames for
understanding the implications of data-centric and automated technology. We conduct research
and build the field of actors to ensure that knowledge guides debate, decision-making, and
technical choices.

www.datasociety.net
@datasociety
Illustration by Jim Cooke
www.datasociety.net
@datasociety
 BEYOND ILLUSIONS

UNMASKING THE THREAT

OF SYNTHETIC MEDIA
FOR LAW ENFORCEMENT

JUNE 2024
 BEYOND ILLUSIONS
UNMASKING THE THREAT OF SYNTHETIC MEDIA
FOR LAW ENFORCEMENT
Table of Contents
Foreword ................................................................................................................................................ 4
Acknowledgement ................................................................................................................................. 5
Executive Summary ................................................................................................................................ 6
1. Introduction ................................................................................................................................ 7
2. Types of Synthetic Media and the Technology Behind ............................................................. 7
2.1 Deepfakes ................................................................................................................................. 7
2.2 Synthetic Audio ........................................................................................................................ 9
2.3 Generated Text ....................................................................................................................... 10
2.4 Synthetic IDs ........................................................................................................................... 12
3. Enabling and Enhancing Law Enforcement Capabilities ............................................................. 13
3.1 Montage Generation .............................................................................................................. 13
3.2 Undercover Operations and Covert Surveillance .................................................................. 13
3.3 Training and Public Engagement ........................................................................................... 14
4. Potential Challenges for Law Enforcement.............................................................................. 14
4.1 Evidence Authentication ........................................................................................................ 14
Challenge to Legal Proceedings ................................................................................................... 14
4.2 Victim Identification ............................................................................................................... 15
Impersonation .............................................................................................................................. 15
4.3 Misinformation and Propaganda ........................................................................................... 16
4.4 Privacy Concerns .................................................................................................................... 17
4.5 Forensics in the Synthetic Media Era..................................................................................... 17
5. Investigative Techniques and Forensic Analysis ...................................................................... 18
5.1 Collection of Evidence ................................................................................................................ 18
Source Verification ....................................................................................................................... 18
Metadata Analysis ........................................................................................................................ 18
Reverse Image Searching ............................................................................................................. 18
Linguistic Analysis......................................................................................................................... 18
5.2 Emerging Technology Solutions for Synthetic Media Detection .......................................... 19
Deep Learning Models ................................................................................................................. 19
Explainable AI ............................................................................................................................... 19
File Structure Analysis .................................................................................................................. 20
Biological Signals .......................................................................................................................... 20
Statistical Analysis at a Pixel Level .............................................................................................. 21
Geometrical and Behavioral Analysis .......................................................................................... 21
AI-generated Speech Analysis ...................................................................................................... 21

2|Page
6. Regulatory and Policy Considerations ..................................................................................... 22
6.1 Intellectual Property .............................................................................................................. 22
6.2 Explainable AI ......................................................................................................................... 23
7. Recent Reports on Synthetic Media......................................................................................... 24
7.1 Deepfakes ............................................................................................................................... 24
7.2 Synthetic Audio ...................................................................................................................... 24
7.3 Generated Text ....................................................................................................................... 25
7.4 Synthetic IDs ........................................................................................................................... 25
8. INTERPOL’s Role in Synthetic Media Forensics .......................................................................... 25
9. Conclusion and Recommendations ............................................................................................ 26

3|Page
Foreword
In the ever-advancing landscape of technology, particularly with the rise of Artificial Intelligence (AI),
synthetically generated media has emerged as a very impactful development for law enforcement.
Although not a new technology, the recent rise of generative AI platforms and deepfake-as-a-service
offerings has led to a surge in synthetic media variations, now easily produced and accessible online
worldwide.

While synthetically generated media offers numerous positive applications across society, it also
presents significant challenges. The accessibility and affordability of various AI platforms have enabled
criminals to exploit this technology, leveraging synthetic media to propagate and perpetuate criminal
activities. This poses considerable challenges for the global law enforcement community. Recognizing
the diverse forms of synthetic media and its associated challenges, INTERPOL is committed to
exploring this dynamic landscape to support our member countries in addressing both current and
future threats posed by synthetic media.

Developing robust forensic and investigative capabilities is crucial for detecting and differentiating
between genuine and manipulated media. This requires a multi-stakeholder approach involving
member countries, industry stakeholders, and academic institutions to pool our knowledge, and share
expertise and advancing technologies. Only through our collective efforts can we mitigate the adverse
effects of synthetic media and ensure the sustained integrity and reliability of global law enforcement
operations.

To support this endeavor, INTERPOL has developed this background paper to offer a comprehensive
analysis of this technology, delving into the potential opportunities offered by it and the challenges it
presents for law enforcement.

I trust that this background paper will serve as a cornerstone for INTERPOL in further exploring the
synthetic media landscape and assisting member countries in advancing and innovating policing.

Jürgen Stock

INTERPOL Secretary General

4|Page
Acknowledgement
The successful completion of this effort was made possible through extensive collaboration. There
were many collaborators involved in the creation of this background paper on synthetic media and its
implication for law enforcement. First and foremost, INTERPOL would like to thank the participants of
the INTERPOL Metaverse Expert Group that inspired the formation of the synthetic media expert
group to discuss current and upcoming threats and support the creation of this document.

INTERPOL would like to extend a special thanks to Mark Evenblij, Brandon Epstein, Paul Warren-Tape,
Matthew Adams, Martino Jerian and Manon den Dunnen for actively contributing to the structure and
content of this document and helped shape this paper. I would like to also thank Ananya Das,
Christopher Church, Fabio Bruno, Julie Tomaszewski, Janani Nair, Priscilla Cabuyao, Toshinobu
Yasuhira, Wookyung Jung, Lindeberg Leite, Mike Price, Parya Lotfi, Mark Nutall, Scott Landman, Jan
Collie, Giorgio Patrini and Julia Absalyamova who peer reviewed this document, which greatly assisted
in adding invaluable contributions and additional insights for this background paper and were
instrumental in filling in the knowledge gaps I extend sincere gratitude to all the experts for their
invaluable insights and contributions.

INTERPOL stands ready to continuously explore the opportunities and challenges that synthetic media
presents to law enforcement and support member countries in understanding, investigating and
applying digital forensics to this emerging threat.

Madan Oberoi
INTERPOL Executive Director of Technology and Innovation

5|Page
Executive Summary
With the rapid advancement of AI technology, synthetic media is becoming an influential mode of
content. Its ease of availability has enabled its exploitation by criminals for malicious
activities. Currently, synthetic media files possess the ability to deceive human perception, making it
challenging to determine their authenticity. As AI continues to evolve, these artificially generated
media will become increasingly sophisticated, posing significant challenges for law enforcement
agencies in conducting forensic analyses of such content.

This background paper on synthetic media and its implications for law enforcement, offers a thorough
introductory understanding to this emerging technology. Based on the research and analysis of the
current synthetic media landscape, the main findings of this paper that are needed to be put forward
for law enforcement officers include:

1. Synthetic media is a new and emerging field and tackling the threats posed by it requires
dynamic research and analysis of the landscape to understand the technology and find
detection solutions.

2. To combat this threat effectively, it is crucial for law enforcement agencies to gain a
comprehensive understanding of the multifaceted nature of synthetic media, including its
creation, distribution, and potential impact.

3. Law enforcement requires a holistic solution to investigate digital media files efficiently and
effectively in this synthetic content era.

4. The rise and increasing ease of AI generated content calls for a collaborative approach
between global law enforcement, private sector, and academia to stay at the forefront of
synthetic media forensics and investigations through ongoing vigilance, continuous
technological advancements, and international collaboration.

In order to assist law enforcement in member countries and enhance collaboration between law
enforcement, industry and academia, INTERPOL has drafted this background paper to explore and
understand the synthetic media landscape and tackle the threats posed by it. Based on the findings,
INTERPOL will continue to work closely with various stakeholders in this endeavor.

6|Page
 1. Introduction

Synthetic media refers to the type of media content, such as images, videos, audio, or text that has
been completely or partially generated or manipulated using artificial intelligence (AI) algorithms.
These technologies allow for the creation of highly realistic and convincing media that can be difficult
to distinguish from human-generated content.

Synthetically generated media has a multitude of positive applications across society, law
enforcement, and the private sector. It serves as valuable training material, enables the
anonymization of individuals, and consistently delivers high-quality content. While synthetic media
content is not a new technology, the recent proliferation of user-friendly generative AI platforms and
deepfake-as-a-service offerings has resulted in a surge of various synthetic media variations and use.
These can now be produced with minimal effort and are accessible to a broad internet user base
worldwide. This sudden rise of synthetic content on the internet has emerged as a significant concern
requiring global law enforcement attention.

Addressing the threats of synthetic media requires a multi-faceted approach. For instance, it may be
difficult to detect and assess the veracity and authenticity of legal documents or frameworks drafted
by AI platforms. Law enforcement agencies need to adapt their investigative approaches to detect
and verify the authenticity of media content, as well as collaborate with experts in the field of AI and
digital forensics to combat the misuse of synthetic media effectively.

2. Types of Synthetic Media and the Technology Behind

Various categories of synthetic media have emerged as a transformative force in the digital realm,
altering how we generate and engage with content. These diverse forms of artificially created or
modified media span a broad spectrum of innovative uses, each with its own unique potential and
impact. Whether it is in the domain of images, sound, videos, or text, these creations rely on advanced
technologies for their generation. With the capacity to imitate and, in some cases, surpass human-
produced content, synthetic media represents a dynamic frontier with a potential for both positive
and concerning implications. Experts believe that 90% of the content available online will be
synthetically sourced by 20251. Some of the key types of synthetic media discussed in this paper are
outlined below.

2.1 Deepfakes

Representing a specific type of synthetic media, deepfakes are created using sophisticated AI
algorithms to depict individuals in highly realistic, and often deceptive, situations that never occurred.
These advanced AI models are capable of creating highly convincing visual and audio content, often
featuring individuals in fabricated scenarios or expressing statements they have never stated.2

1
Giardina, C. (2023, January 8). CES: Could 90 Percent of Content Be AI-Driven by 2025? The Hollywood Reporter.
https://www.hollywoodreporter.com/movies/movie-news/ces-ai-sag-aftra-1235290431/
2
Goodfellow, I. J., et al. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems
(https://arxiv.org/pdf/1406.2661.pdf)

7|Page
The emergence of deepfakes has generated profound implications across various domains, raising
both interest and concern in equal measure. With reference to its usage, deepfakes have unlocked
new horizons in the realm of special effects, enriching the entertainment industry with lifelike CGI3
and offering unparalleled creative potential. Conversely, the nefarious use of deepfakes for
misinformation, identity theft, and manipulation has raised significant ethical and security concerns,
underscoring the urgent need for safeguards and countermeasures.

The principal algorithms used in the generation of deepfakes are:

Algorithm Description and Concept Application in Deepfakes

GANs are composed of a Generator and a

GANs are mainly used to
Generative Discriminator4. GANs engage in a creative
generate digital clones of a
Adversarial duel where the generator generates new
certain individual by
Networks media based on the training data and the
mimicking facial expressions in
(GANs) discriminator acts a gatekeeper, making sure
videos. (Figure1 (1))
the generated media looks real or fake.

Diffusion models provide a guided approach

to generating diverse and high-quality Diffusion Models are used to
deepfakes by gradually transitioning between create Picture to Picture
Diffusion distributions of input data points. This transformations that allow for
Models process preserves sample diversity and smooth transition between
facilitates the computation of conditional image distributions.(Figure1
probabilities, contributing to the creation of (2))
realistic deepfakes.5

Autoencoders are a type of neural network

that are trained to efficiently encode input
data, capturing essential features while Autoencoders excel at face
discarding irrelevant information.6 This swaps using its ability of
encoded representation is then used to retaining image quality while
Autoencoders
reconstruct the original input, enabling seamlessly replacing facial
autoencoders to generate deepfakes by features in videos deepfakes.
seamlessly replacing facial features in videos (Figure1 (3))
while retaining image quality and appear
authentic.
Table 1: Technology used for the generation of Deepfakes

3
CGI – Computer Generated Imagery. Computer Generated Imagery are a type of visual effects in which scenes, effects
and images are synthetically generated using a computer software.
4
Luca Guarnera et al. (2023) Level Up the Deepfake Detection: A Method to Effectively Discriminate Images Generated by
GAN Architectures and Diffusion Models (https://arxiv.org/pdf/2303.00608.pdf)
5
Aditya Ramesh et al. (2022) Hierarchical Text-Conditional Image Generation with CLIP Latents
(https://arxiv.org/pdf/2204.06125.pdf)
6
https://www.ibm.com/topics/autoencoder

8|Page
 7

Figure 1: The Different Deepfakes produced with different models.8

2.2 Synthetic Audio

Voice synthesis is a distinct niche within the synthetic media landscape that has emerged with the rise
of generative AI. It involves the creation of lifelike speech patterns and audio content, often mimicking
the vocal characteristics of specific individuals, or generating a totally new tone or style of speech. In
its constructive application, voice synthesis has revolutionized the accessibility of technology, aiding
individuals with speech impairments, and enhancing the capabilities of virtual assistants with more
human-like interactions.

On the other hand, voice synthesis could be misused for manipulation of audio recordings,
impersonations, and other deceptive practices. These nefarious activities forecast the critical need for
security measures and ethical guidelines to safeguard against misuse. The ability to generate
convincing synthetic voices has deepened concerns about identity theft, fraud, and the credibility of
audio evidence, necessitating vigilance and countermeasures to uphold the integrity of auditory
content in a digitally evolving world.

8 Image Contributed by DuckDuckGoose (AI Detection Services | Verifying ID & Combatting Fraud (duckduckgoose.ai))

9|Page
Some of the main AI algorithms utilized for creating synthetic audio are:

Algorithm Description and Concept Application in Synthetic Audio

RNNs detect patterns in sequential
data to synthesize audio files.9 By RNNs are used for voice synthesis,
Recurrent training on voice recordings, RNNs RNNs generate synthetic voices with
Neural learn the nuances of human speech, human-like quality, aiding in
Networks capturing phonetic, prosodic, and applications such as virtual assistants
(RNNs) expressive elements to produce and personalized speech synthesis for
accurate and personalized synthetic individuals with speech impairments.
voices.

WaveGAN utilizes principles from WaveGAN transforms image-

GANs to generate raw audio slices, processing models into audio synthesis
producing remarkably realistic audio tools, generating diverse and high-
WaveGAN and
samples.10 VAEs offer potential for quality audio content. VAEs enable
Variational
speech generation, musical unconditional generation of speech
Autoencoders
composition, and sound design, and music, pushing the boundaries of
(VAEs)
bridging the gap between audio synthesis with enhanced
computational efficiency and high- computational efficiency and
quality audio synthesis.11 exceptional quality.

Table 2: Algorithms used to synthesize audio

2.3 Generated Text

Text generation represents a distinct area of synthetic media, enabling the automated creation of
written content with remarkable authenticity. It is the task of generating text with the goal of
appearing indistinguishable to human-written text, more formally known as Natural Language
Generation (NLG). Text generation plays a pivotal role in the dynamic landscape of synthetic media,
contributing to the creation of diverse and engaging content. The current most advanced methods for
text generation, including platforms like ChatGPT or BART AI, enable the generation of human-like and
contextually relevant text across various domains. In the realm of synthetic media, these models
empower e.g. content creators to produce realistic and coherent narratives for virtual characters,
dialogues for interactive storytelling, and even generate new article or blog posts with minimal human
intervention.

9
Robin M. Schmidt (2019). Recurrent Neural Networks (RNNs): A gentle Introduction and Overview
(1912.05911.pdf (arxiv.org))
10
Chris Donahue et al. (2018) Adversarial Audio GANs (https://arxiv.org/pdf/1802.04208.pdf)
11
Antoine Caillon et al. (2021) RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
(https://arxiv.org/pdf/2111.05011.pdf)

10 | P a g e
The principal algorithms used in the generation of text are:

Algorithm Description and Concept Application in Generated Text

RNNs are used for generating text
with rich vocabulary, intricate
RNNs generate text with their ability to grammatical structures, and diverse
process and retain past information12, proper names. Handles tasks like
overcoming challenges like the vanishing balancing parentheses and quotes
Recurrent
gradient problem13. Modern RNNs excel across long distances,
Neural Network
in character-level language modelling, demonstrating proficiency in
(RNNs)
offering potential for revolutionizing text understanding and producing
compression and enhancing accessibility complex language structures.
for individuals with physical disabilities. Multiplicative RNNs (mRNNs)14
provide an edge in processing and
generating intricate text.
Transformer models within the context
of data-to-text generation, transform
structured data into readable narratives. Transformer models are utilized for
Traditionally, this involved two steps: creating summaries based on the
deciding what to say and how to say it, underlying structure of input data.
Transformer but with sequence-to-sequence (S2S) Performs data-to-text generation
Models learning, models now perform both tasks efficiently, addressing
tasks simultaneously. Challenges include challenges in producing responses
working with structured data and from structured data and
summaries to generate responses, which summaries.
are more complex than other text
generation tasks.15
GPT model is used for generating
GPT models is very good for generating
text that sounds natural and
human-like and contextually coherent
contextually relevant, owing to
text by predicting the next word based
Generative Pre- their understanding of relationships
on preceding context. Trained on
Trained and patterns within language.
extensive text and datasets, they
Transformer Versatile enough for fine-tuning to
demonstrate proficiency in various
(GPT) specific tasks or domains,
natural language processing (NLP) tasks,
enhancing performance in
including text completion, question
generating text tailored to
answering, and language translation.16
particular contexts.

Table 3: Algorithms used for text generation

12
Ilya Sutskever et al. (2011) Generating Text with Recurrent Neural Networks
(https://www.cs.toronto.edu/~jmartens/docs/RNN_Language.pdf)
13
Vanishing Gradient Problem - Vanishing gradient problem is a phenomenon that occurs during the training
of deep neural networks, where the gradients that are used to update the network become extremely small or
“vanish” as they are backpropagated from the output layers to the earlier layers. (engati.com)
14
mRNN - A Multiplicative RNN (mRNN) is a type of recurrent neural network with multiplicative connections.
(paperwithcode.com)
15
Li Gong et al. (2019) Enhanced Transformer Model for Data-to-Text Generation
(https://aclanthology.org/D19-5615.pdf)
16
Open AI (2023) GPT-4 Technical Report (https://cdn.openai.com/papers/gpt-4.pdf)

11 | P a g e
2.4 Synthetic IDs

Synthetic IDs are artificial identity documents to produce images resembling genuine and authentic
identification documents. These IDs are designed to replicate the appearance of legitimate
identification papers, including driver's licenses, passports, or other official forms of identification.

The primary aim of generating synthetic IDs is to support the volume of data available for training
deep neural networks, specifically for the purpose of detecting and preventing fraudulent identity
documents. This approach assists in enhancing fraud detection mechanisms while simultaneously
addressing concerns related to privacy and the sensitive nature of personal identity documents.17

Synthetic IDs can be generated using the following principal algorithms:

Algorithm Description and Concept Application in Synthetic IDs

Computer vision algorithms and
GANs work together to generate Computer Vision and GANs are
synthetic ID card images for employed to produce diverse
enhancing fraud detection systems. classes of ID cards, simulate
GAN models simulate genuine ID genuine examples, introduce
Computer Vision and card examples and introduce texture noise, and create
GANs texture noise, while computer composite scenarios, thereby
vision techniques enable automatic expanding datasets for fraud-
splicing to create composite detection network training and
scenarios, thereby expanding improving the accuracy of
datasets for training fraud- identifying fraudulent activities.
detection networks.

Diffusion Probabilistic Models,

especially Large diffusion models Diffusion Probabilistic Models are
are employed as generative models, utilized to generate and augment
to generate and augment document document data, particularly
data, address challenges in identity documents, by employing
accurately representing complex a customized pipeline that
Diffusion Probabilistic
identity documents. A customized integrates optical character
Models
pipeline combines optical character recognition, layout
recognition, LayoutLM18 understanding, face detection,
understanding, face detection, and and text layout generation,
text layout generation to create thereby improving the fidelity of
fake identity documents with synthetic identity documents.
improved fidelity.

Table 4: Algorithms for Synthetic ID Generation

17
Daniel Benalcazar et al. (2022) Synthetic ID Card Image Generation for Improving Presentation Attack Detection
(https://arxiv.org/pdf/2211.00098.pdf)
18
Yiheng Xu et. al. (2019). LayoutLM: Pre-training of Text and Layout for Document Image Understanding
(https://arxiv.org/abs/1912.13318 )

12 | P a g e
 (A) (B)

Figure 2: Synthetic IDs: (A) Driver’s License (B) A passport19. These synthetic IDs look very close to the
authentic IDs and often can be mistaken as original or even pass identity verification tests

3. Enabling and Enhancing Law Enforcement Capabilities

Synthetic media offers a range of benefits for law enforcement, revolutionizing various aspects of their
operations and strategies.

3.1 Montage Generation

Synthetic media may also be used to translate eyewitness descriptions into visual representations of
potential suspects: through analysis and interpretation of provided details, such as facial features or
distinct physical traits. This aids law enforcement investigations by providing leads and helping to
identify and locate potential suspects involved in criminal activities. By generating accurate and
detailed suspect images, synthetic media significantly supports law enforcement efforts, streamlining
investigations for more effective outcomes.20

3.2 Undercover Operations and Covert Surveillance

Synthetic media significantly helps law enforcement capabilities through its roles in undercover
operations and covert surveillance. Primarily, in undercover operations, this technology serves as a
cornerstone for crafting and adapting personas of undercover officers. By leveraging synthetic media,
law enforcement agents can swiftly modify appearances, incorporating fictional injuries, scars,
tattoos, or other alterations necessary to sustain cover identities. This adaptability ensures the
maintenance of plausible and dynamic personas critical for successful undercover assignments.

Moreover, in the realm of covert surveillance, synthetic media plays a key role in supporting law
enforcement by assisting in the creation of realistic backstories for undercover officers. This
technology contributes to the development of believable and authentic personas for undercover

19
Image Contributed by IDVerse. (Home - IDVerse)
20
https://www.researchgate.net/publication/342978037_Suspect_Face_Generation

13 | P a g e
operatives, enhancing the officers’ capabilities to operate incognito within diverse environments. By
providing the means to establish credible backstories, synthetic media augments the effectiveness
and adaptability of covert surveillance strategies employed by law enforcement agencies.21

3.3 Training and Public Engagement

Synthetic media plays a significant role in augmenting law enforcement capabilities, particularly in
training, public engagement, and officer anonymity. Within the domain of training, law enforcement
agencies leverage synthetic media to create highly realistic and immersive scenarios. By simulating
authentic training environments, officers can enhance their preparedness to handle diverse and
complex situations they might encounter in the field. This enables them to refine their skills, decision-
making, and response strategies in a controlled yet realistic setting.

Furthermore, synthetic media finds utility in event reconstructions, aiding law enforcement in
recreating crime scenes or incidents for investigative purposes or public appeals. By recreating
scenarios using synthetic media, law enforcement can potentially gather valuable evidence, seek
public assistance, and increase community engagement in solving cases.

4. Potential Challenges for Law Enforcement

The threat that synthetic media poses to law enforcement is primarily related to its potential for
misuse and deception. Here are a few threats it can pose:

4.1 Evidence Authentication

The rise of synthetic media brings forth significant challenges for evidence authentication in
investigations. Determining the authenticity of media content becomes a complex endeavor due to
the sophisticated manipulation capabilities. Traditional methods of visual and audio analysis may no
longer be sufficient to identify manipulated content, thereby requiring specialized expertise and the
utilization of advanced technologies for forensic analysis. This places additional burdens on
investigative bodies and legal authorities to keep pace with the evolving landscape of synthetic media
and its potential implications for the justice system.

Challenge to Legal Proceedings

Synthetic media can be used to create fake evidence, such as forged videos or manipulated images,
which can be presented as genuine in legal proceedings. This could lead to wrongful accusations, false
alibis, or the undermining of an investigation. In an era where digital evidence carries substantial
weight, the potential for malicious actors to exploit synthetic media for their advantage is a growing
concern. Instances of fake alibis, fabricated crime scenes, or manipulated testimonies can emerge,
casting doubt on the integrity of the legal system and the pursuit of justice. Consequently, the rise of

21
Kelly W Sundberg and Christina M Witt (2019); Undercover operations: Evolution and modern challenges.
Undercover operations: Evolution and modern challenges | Journal of the Australian Institute of Professional
Intelligence Officers (informit.org)

14 | P a g e
synthetic media requires a closer examination of how legal systems adapt to address these emerging
threats, ensuring that the pursuit of truth and fairness remains paramount.

The proliferation of synthetic media poses a significant challenge when it comes to proving the
authenticity of evidence in legal proceedings. In an environment where fabricated media can be
virtually indistinguishable from genuine content, the burden of verifying the veracity of audio, video,
or images becomes increasingly complex. This challenge is compounded by the emergence of the
"Liars Dividend," a phenomenon where defendants claim that genuinely authentic media has been
artificially generated, creating a climate of doubt and uncertainty. As a result, judicial systems face the
pressing need to adapt to this evolving landscape, developing robust methods and standards for
evidence authentication to uphold the integrity of legal proceedings.

4.2 Victim Identification

The emergence of synthetic media has given rise to a distressing concern regarding non-consensual
explicit content. Malicious actors can exploit this technology to create compromising content
featuring individuals without their knowledge or consent. In such scenarios, identifying and providing
support to the victims becomes notably intricate, as the line between reality and synthetic fabrication
blurs. Addressing these issues requires heightened vigilance, legal frameworks, and support systems
for potential victims to ensure their well-being and protection in an increasingly digital and
interconnected world.

Impersonation
Synthetic media enables highly convincing impersonations of individuals, presenting a concerning
challenge. For instance, an individual's voice can be synthesized with remarkable accuracy, making it
exceptionally challenging to distinguish between a genuine and manipulated phone call or audio
recording. This capability can be exploited for a range of malicious purposes, from carrying out
extortion schemes to disseminating false information, thereby amplifying concerns about identity
verification and trust in various aspects of communication and digital interactions. In an age where
authenticity is increasingly fragile, addressing the potential for impersonation is a crucial aspect of
managing the risks associated with synthetic media.

• AI Voice Scams
AI voice scams ingeniously leverage technology to replicate the voice of a victim's family member,
friend, or even impersonate customer service representatives, all with the intent to deceive.
Perpetrators utilize these fabricated voices to manipulate victims into revealing sensitive personal
information or coercing them into sending money. The scam operates by exploiting trust and
familiarity, leading unsuspecting individuals to believe they are interacting with someone they know
or a legitimate service provider, amplifying the success of the ruse.

• Bypassing liveness checks

A liveness check tests whether the person in front of the camera is the actual user or if the user is
attempting to spoof an identity that is not theirs. Active liveness is the most common and consists in
asking the user to perform certain actions like blinking, smiling, or following movement instructions
with their head. Despite its widespread adoption, active liveness checks are weak against attacks by

15 | P a g e
Deepfakes. The reason is that real-time Deepfakes can reproduce faithfully facial landmark
movements of the attackers. Hence, even if facial traits e.g. eye shape and color, the height of the
cheekbones, the shape of mouth do change, the attacker can move their head left and right and smile,
easily following the active liveness instructions.

Know Your Customer (KYC) is the practice of verifying an individual’s identity in compliance with laws
and regulations, primarily for anti-money laundering purposes. In response to the pronounced
inability to detect and prevent online fraud, the KYC process has been introduced to reduce instances
of illegal transactions. Due to its sensitivity for businesses, KYC is regulated by respective national and
international government agencies. KYC procedures are necessary when opening an account with a
bank or when conducting financial transactions of any sort. The possibility of bypassing KYCs with aid
of synthetic media may pose a huge threat to the online liveness check procedures, as it enables
chances of impersonation and thus compromises the reliability of the identity verification process.

In a recent incident, it was reported that a multinational corporation fell victim to a sophisticated
deepfake scam, resulting in a huge financial loss of $25.6 million. With the use of advanced synthetic
media technology, perpetrators digitally replicated high-ranking company officials, including the Chief
Financial Officer, within a simulated video conference setting. The deepfake representations
convincingly mimicked the appearance, voice, and mannerisms of genuine personnel, fostering an
illusion of authenticity. Under the guise of legitimacy, the victim was deceived into executing a series
of 15 transfers totaling millions of dollars to multiple bank accounts. Despite initial skepticism, the
victim's compliance was facilitated by the seamless interaction and seemingly genuine appearance of
the deepfake personas.22

4.3 Misinformation and Propaganda

Synthetic media's capacity to generate convincing news articles, videos, or social media posts raises
substantial concerns related to the dissemination of false information and the manipulation of public
opinion. This creates formidable challenges for law enforcement agencies, which heavily depend on
accurate and reliable information to conduct effective investigations and ensure public safety. The
proliferation of synthetic media exacerbates the complexity of discerning fact from fiction,
underscoring the pressing need for robust mechanisms to verify the authenticity of digital content and
safeguard against the potential consequences of misinformation and propaganda in an information-
driven society.

Simultaneously, the rise in use of text generation tools also raises serious concerns. These ranges from
questions of plagiarism and intellectual property rights, to ethical concerns, as it becomes crucial to
ensure responsible use to prevent misinformation, deepfakes, or manipulation of public opinion. Large
Language Models (LLM) are vastly used to generate content across social media. Unsupervised use of
this content may lead to the spread of misinformation and propaganda. Social media is a breeding
ground for such false information, attracting such malicious content to reach a larger audience.23

22
https://economictimes.indiatimes.com/industry/tech/hong-kong-mnc-suffers-25-6-million-loss-in-deepfake-
scam/articleshow/107465111.cms?from=mdr
23
https://www.cpahq.org/media/ivih25ue/handbook-on-disinformation-ai-and-synthetic-media.pdf

16 | P a g e
4.4 Privacy Concerns

Synthetic media's capability to create explicit or compromising content, featuring individuals who may
have never engaged in such behavior, escalates profound privacy concerns. This technological
potential can be exploited for malicious purposes, including harassment, blackmail, or causing
significant harm to an individual's reputation. The ease with which synthetic content can be produced
and disseminated amplifies the need for robust privacy protection measures and legal safeguards to
mitigate the potential harm that could be inflicted upon individuals.

For instance, in circumstances involving deepfake-fueled extortion, perpetrators engage victims in

interactions until they obtain explicit images or videos. The extortion scheme relies on these
manipulated, synthetic media as the leverage for blackmail, bypassing the need for establishing
rapport or befriending the victim. Once in possession of these compromising deepfake images or
videos, the culprits coerce victims into complying with their demands, threatening to expose or
distribute the falsified content if their demands are not met.

These synthetic IDs by impersonation of legitimate personal data mixed with synthetic information,
allow criminals to create false identities that could be exploited by fraudsters to secure credit lines,
apply for government benefits, intercept tax returns, and participate in various financial activities,
resulting in significant monetary losses for financial institutions and government agencies.24

4.5 Forensics in the Synthetic Media Era

A key element of the challenge facing forensic experts lies in the need to understand the AI media
generation process and integrate this advanced knowledge into the forensic identification steps.
While automated tools offer valuable assistance, they are not entirely adequate. To conduct effective
forensic analysis amid the ever-evolving landscape of AI-generated media, it is essential to combine
the use of these tools with rigorous analysis provided by experienced forensic professionals. This
balanced approach ensures an unbiased and interpretable examination.

24
https://legal.thomsonreuters.com/en/insights/articles/synthetic-identity-fraud

17 | P a g e
5. Investigative Techniques and Forensic Analysis

5.1 Collection of Evidence

Investigative techniques and forensic analysis are vital in combating the growing unlawful synthetic
media content. For a seized evidence file, the following investigative techniques can be applied to
check the authenticity of an evidence file.

Source Verification
This technique involves verifying the origin and authenticity of a particular piece of synthetic media.
Investigators assess the credibility of the source, the techniques used to create the content, and any
potential motives for dissemination. This multifaceted approach requires technical expertise,
adaptability to evolving methods, and advanced technologies to detect manipulated content and
mitigate harm.

Metadata Analysis
Metadata analysis by supplying vital contextual information aids in the realm of synthetic
media investigations. This includes details about the source, creation date, and editing history,
providing investigators with the means to trace the origins and authenticity of potentially deceptive
or harmful content. However, it is important to note that in certain scenarios, particularly when
the media content has been transmitted over various applications, the metadata may be
compromised or incomplete. This inherent limitation highlights the need for a multi-faceted
approach, combining metadata analysis with other investigative techniques to ensure a
comprehensive examination of synthetic media.

Reverse Image Searching

Reverse image searching is a process of taking an image and searching for similar or exact matches on
the internet. It plays an important role in the arsenal of investigative tools when determining the
authenticity or tracing the origins of media content, especially when dealing with morphed or
manipulated features. When the original, unaltered version of the content can be found online,
reverse image searching can quickly unveil the deception, assisting in the identification of manipulated
or synthetic media.

Linguistic Analysis
Delving into the linguistic aspects and writing style embedded within synthetic media is a strategic
approach for unraveling its true origins. The unique language patterns, cultural allusions, and even
grammatical anomalies can serve as valuable clues in discerning the authenticity of the content.
Experts employ linguistic analysis to ascertain whether the media in question is a product of
manipulation or an authentic piece. This examination of language serves as a linguistic fingerprint that
can help investigators determine whether the content aligns with the expected linguistic traits of the
purported source, thereby aiding in the authentication of multimedia materials.

18 | P a g e
5.2 Emerging Technology Solutions for Synthetic Media Detection

Synthetic media refers to any kind of media content that has been artificially created or manipulated
using advanced technologies. Some of the detection solutions currently used are illustrated below.

Deep Learning Models

Deep learning models in detecting synthetic IDs and synthetic media leverage various techniques and
algorithms and follow the concept of “AI is needed to detect AI”. One significant approach involves
training models to differentiate between real and AI-generated images or videos by identifying
inconsistencies or artifacts left by generative models. These include abnormal textures, synthetic
movements, or subtle flaws often overlooked by the human eye but detectable by a well-trained AI
system. Moreover, unsupervised learning techniques, like anomaly detection algorithms, prove useful
in flagging potential forgeries as generative AI fraud introduces atypical patterns into the data.

In addition to identifying anomalies, digital forensic techniques are applied to analyze the metadata
of images and videos, examining format, compression, and noise distribution. Generative AI typically
leaves distinct digital fingerprints that forensic tools can detect. To stay ahead, incorporating
adversarial training exposes detection models to the latest AI-generated fraud, enabling them to learn
and adapt to new and evolving generative techniques.

Further combining multiple AI models to analyze diverse aspects of the data, such as visual and
temporal features enhance detection rates. For instance, a deepfake video might display realistic
visuals but may lack synchronization between the facial and verbal expressions, detectable by an AI
trained to recognize such discrepancies. Moreover, AI systems cross-reference content with known
databases of authentic images, videos, or documents, revealing discrepancies that could signify
fraudulent activity. This proves particularly effective in document verification, where details are cross-
checked against official records, enhancing the precision of synthetic media and synthetic ID detection
methods.

Explainable AI
The differentiation between authentic and synthetic media is becoming critically important. Deepfake
detection leveraging AI presents a promising solution to manage the increasing volume of synthetic
content. However, the inherent "black box25" nature of these AI systems hampers the transparency of
their decision-making processes, raising the question, "Why is this image classified as fake?" As the
average individual's ability to distinguish between sophisticated synthetic images and reality
diminishes, the necessity for Explainable AI (XAI) in this domain grows. Some of the explainable
deepfake detection approaches include the following.

25Black Box - A black box refers to a system whose behavior has to be observed entirely by inputs and outputs. Even if the
internal structure of the application under examination can be understood, the tester chooses to ignore it. Black box
testing assesses a system solely from the outside, without the operator or tester knowing what is happening within the
system to generate responses to test actions. (techtarget.com)

19 | P a g e
 Method Description

CAM offers a generalized approach to enhance interpretability of AI decisions.

Class Activation
Directing focus to specific areas within deepfakes, CAM offers a valuable entry
Mapping (CAM)
point analysis and understanding to the AI reasoning.

Deepfake Deepfake detection typology helps identify the deepfake models which can
Detection provide insights into origins or creator identity, akin to tracing cybercriminal
Typology trends.

The digital fingerprinting technique traces deepfakes to their originating GAN

Digital
through unique digital fingerprints. While such fingerprints can be masked or
Fingerprinting
removed, their presence unequivocally indicates content manipulation.

Biometric analysis with the utilization of physiological signals like heartbeat

patterns and eye reflections aid in affirming human authenticity. Despite potential
Biometric Analysis for spoofing, when applied correctly, this method can be pivotal in flagging
deepfakes—such as in cases where high-resolution images lack expected biometric
data.

File Structure Analysis

File Structure Analysis is a non-content-based authentication approach that leverages the way a file is
constructed at a binary level. Because it does not rely on displayed imagery, the quality of the video
does not affect the analysis; meaning high quality or low-quality synthetic media is equally detectable.
File Structure Analysis also does not rely on metadata values which can easily change due to
intentional manipulation or when transmitted between devices. As a deterministic, transparent and
repeatable approach, the results of File Structure Analysis are explainable as to how they are
generated and can be evaluated by the end user. File Structure Analysis can also be used to prove
that a file is camera original and attributed to a specific brand and model of device, allowing for not
only the detection of synthetic media but the authentication of “real” imagery. While File Structure
Analysis will accurately and reliably identify camera original versus synthetic files at scale in an
automated process, it will not identify the specific content that could have changed.

Biological Signals
Biological signals, such as photoplethysmography (PPG) cells26, can assist in discerning between
authentic and synthetic media. They are used in a smart system that spots different kinds of generated
models used in synthetic videos. These signals have patterns that help show where videos might have
been altered by highlighting out the changes from the real biological data. These signals are harnessed
to power a classification network, enabling the detection of generative models employed in videos.
The intricate spatiotemporal patterns within these biological signals serve as a crucial representation
of residuals. These patterns effectively unveil manipulation artifacts by segregating them from the
authentic biological signals. Given that biological signals remain absent in deepfakes, their absence
yields distinct signatures discernible in the generative noise. Consequently, these biological signals
serve as an indispensable projection, representing the residuals within a known dimension. This

26
Photoplethysmography (PPG) cells - Photoplethysmography (PPG) measures the amount of light absorbed or reflected
by human tissues. (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8950880/)

20 | P a g e
projection allows for an exploration of unique signatures per model, enabling the identification and
differentiation of synthetic media from authentic content.27

Statistical Analysis at a Pixel Level

Statistical analysis at a pixel level involves scrutinizing decoded video pixels or coefficients to identify
irregularities that might indicate the presence of synthetic media. This process focuses into the details
of video data, examining statistical patterns and anomalies that could be attributed to the generation
of synthetic content. For instance, in this method, the artifacts generated by convolutional neural
networks is observed during their processing and often visible in the Fourier transform28 of pixels.

By analyzing the statistical characteristics of decoded video pixels or their transformations, this
technique aims to uncover telltale signs of synthetic generation. Anomalies in statistical patterns,
unusual fluctuations, or irregularities in pixel transformations may signify the manipulation or artificial
generation of the media. These irregularities can range from subtle variations to distinct patterns that
deviate from what would be expected in naturally captured or authentic content.

Geometrical and Behavioral Analysis

Geometrical and behavioral analysis plays a crucial role in detecting synthetic media by careful
inspection of various inconsistencies that might be imperceptible to the human eye. These analyses
look into elements like shadows, reflections, perspective, proportions, and other geometrical
attributes to unveil discrepancies that could indicate synthetic generation.

For instance, when examining shadows and reflections within synthetic media, an in-depth analysis
focuses on their consistency and adherence to natural laws. Synthetic content may inadvertently
introduce inconsistencies in how shadows fall or how reflections interact with the environment,
deviating from the expected behavior seen in authentic visual content. By systematically analyzing
these elements, anomalies such as incorrect lighting angles or inconsistent shadow lengths can be
identified, indicating potential manipulation.

Similarly, behavioral analysis involves studying the behavior of objects or entities depicted in the
media. Synthetic media might exhibit discrepancies in behavioral patterns that depart from expected
real-world behaviors. This examination involves assessing how objects move, interact, or behave
within the digital environment. Any unnatural or irregular movements, unnatural physics, or peculiar
interactions within the synthetic content can signal potential artificial manipulation.

AI-generated Speech Analysis

AI speech detection involves a blend of signal processing techniques and machine learning algorithms
to analyze and interpret audio data. Initially, the input speech undergoes preprocessing to extract
pertinent features, which can include spectrograms including Mel-frequency cepstral coefficients
(MFCCs), Linear Frequency Cepstral Coefficients (LFCCs). These extracted features encapsulate both

27
Umur Aybars Ciftci et al. (2020) How Do the Hearts of Deep Fakes Beat? Deep Fake Source Detection via Interpreting
Residuals with Biological Signals (https://arxiv.org/pdf/2008.11363.pdf)
28
Fourier Transform - The Fourier Transform is an important image processing tool which is used to decompose an image
into its sine and cosine components. The output of the transformation represents the image in the Fourier or frequency
domain, while the input image is the spatial domain equivalent. (homepages.inf.ac.uk)

21 | P a g e
frequency and time characteristics of the audio, enabling a detailed representation of speech patterns.
These features serve as inputs to deep learning models, such as convolutional neural networks (CNNs)
or recurrent neural networks (RNNs), adept at learning intricate patterns and variations in speech. By
leveraging these diverse feature sets, these models can power speech detection systems.

6. Regulatory and Policy Considerations

Due to the potential implications and risks associated with synthetic media, several regulations and
guidelines have been proposed or implemented to address concerns such as misinformation, privacy,
copyright infringement, and ethical use.

One example is the deepfake technology, which refers to the use of AI to create realistic synthetic
media, often used to manipulate videos or images of individuals. In many countries, laws regarding
privacy and defamation can be applied to deepfakes, holding creators accountable for violating privacy
rights, spreading false information, or defaming individuals.

6.1 Intellectual Property

The evolution of synthetic media poses significant challenges in copyright ownership. With AI-
generated content often blurring the lines of original authorship, determining rightful ownership
becomes intricate. Identifying the creator or owner of manipulated content in synthetic media raises
questions about copyright claims and protections. The complexities involved in establishing ownership
rights in AI-generated works call for a reassessment and potential adaptation of copyright laws to
address the novel aspects of content creation in the digital age.

The concept of fair use undergoes scrutiny in the context of synthetic media. Fair use exceptions are
subject to debate when assessing the transformative nature of AI-generated content. Differentiating
between legitimate transformative works and potential copyright infringement becomes increasingly
challenging. The interpretation of fair use laws needs refinement to accommodate the transformative
aspects of AI-generated content, ensuring the balance between creative freedom and copyright
protection.

The responsible use of AI in law enforcement prioritized the alignment of policing principles, ethical
standards, and human rights compliance. With this intent, INTERPOL, in partnership with the United
Nations Interregional Crime and Justice Research Institute (UNICRI), published the Toolkit for
Responsible AI Innovation in Law Enforcement29 in June 2023. This toolkit, comprises of seven
resources aimed at guiding law enforcement agencies in designing, developing, procuring, and
deploying AI technologies in a responsible manner.

Policymakers face the task of adapting existing intellectual property regulations to effectively
encompass the nuances presented by synthetic media. Establishing guidelines and standards that
govern AI-generated content is imperative to ensure compliance with intellectual property laws.
Crafting regulatory frameworks that address the intricate issues of copyright, licensing, trademark

29
https://www.interpol.int/en/How-we-work/Innovation/Artificial-Intelligence-Toolkit

22 | P a g e
protection, and fair use within the context of synthetic media creation is essential to safeguard the
interests of creators and copyright holders while fostering innovation.

Memo
In many countries, the exploration and utilization of generative AI, such as ChatGPT and Gemini
(previously Bard AI), have triggered intricate intellectual property debates under existing legislation.
There are countries whose domestic laws permit “information analysis”, allowing the use of
copyrighted materials for the training of generative AI models in the training stage.

6.2 Explainable AI

"Explainable AI" is emerging as a crucial factor in AI applications, especially in digital forensics, for
detecting synthetic or deepfake media and processing extensive data. To be considered "Explainable
AI," it is important for these tools to adhere to the principles outlined by NIST30 Circular 8312. This
entails providing clear, accurate, and understandable explanations for AI-generated outputs, including
confidence levels, beyond mere yes/no responses.

Memo
In some countries, within legal contexts, authenticating digital evidence traditionally involves
witness testimony. However, AI tools cannot testify, leading to requirements for expert
witnesses familiar with AI technology to authenticate evidence in court. Yet, challenges persist in
explaining AI processes adequately for authentication purposes. Moreover, some standards
assessing the admissibility of novel scientific testimony demand that AI techniques have been
tested, published, demonstrated error rates, follow standards, and enjoy wide scientific acceptance.
Transparency in AI algorithms becomes pivotal as a lack of explainability may hinder the
admissibility of AI-generated results.31

30
NIST - National Institute of Standards and Technology
31
https://medexforensics.com/2023/10/30/evaluating-the-use-of-ai-in-digital-evidence-and-courtroom-admissibility/

23 | P a g e
7. Recent Reports on Synthetic Media
With the rise of generative AI, the ease of producing synthetic media has significantly increased which
pose a serious concern in terms of their widespread creation and dissemination.

7.1 Deepfakes

• Two men have been founded guilty for using AI to generate exploitative images of children,
marking a significant instance in such cases. The court’s ruling established a significant
precedent by recognizing that sexually abusive content encompasses AI-generated imagery
with a "high level" of realism that can resemble real children and minors.32

• An investigation has been launched into the use of AI to manipulate images of young girls,
altering their clothing and circulating the edited photos.. It was revealed that multiple girls
were affected, and some of the preparators have been identified by the police. This stressed
the severity of the situation, calling for those responsible to cooperate and expressing fears
that the images might be disseminated on websites. This incident highlights the rising problem
of digital violence and the pressing need to address it.33

• In Quebec, Canada, a man was found guilty for producing synthetic videos of child sexual
abuse using deepfake technology. The judge highlighted the detrimental impact of these
synthetic images, not only fueling the market for child sexual abuse material but also
complicating police investigations and endangering the safety of children online whose
identities could be exploited.34 A Purple Notice35 was issued to alert INTERPOL member
countries on the use of deepfake technology to defraud and extort victims through
impersonation scams, online sexual blackmail and investment fraud.

• In 2023, The National Centre for Missing & Exploited Children (NCMEC) received
approximately 5000 reports of AI-generated child exploitation media files. NCMEC’s
CyberTipline 2023 Report37 highlights the concern of such AI-generated Child Sexual Abuse
Material or AIG-CSAM created by bad actors based on a real child or computer-generated
children. The report also states that more than 70% of these AIG-CSAM files were from
conventional online platforms38, thus highlighting the inadequate content tracking protocols
on these platforms. However, in April 2023, industry leaders like OpenAI, Amazon,
Anthropic, Civitai, Google, Meta, Metphysic, Microsoft, Mistral AI, and Stability AI have
joined an initiative led by Thorn to adopt ‘Safety by Design Principles’ to prevent the creation
and proliferation of AIG-CSAM39.

32
https://edition.cnn.com/2023/09/27/asia/south-korea-child-abuse-ai-sentenced-intl-hnk/index.html
33
https://edition.cnn.com/2023/09/20/europe/spain-deepfake-images-investigation-scli-intl/index.html
34
https://www.cbc.ca/news/canada/montreal/ai-child-abuse-images-1.6823808
35
INTERPOL Purple Notice: To seek or provide information on modus operandi, objects, devices and concealment methods used
by criminals.
36The National Center for Missing & Exploited Children is a private, non-profit corporation based in the United States
37
https://www.missingkids.org/cybertiplinedata
38
https://www.missingkids.org/blog/2024/generative-ai-csam-is-csam
39
https://openai.com/blog/child-safety-adopting-sbd-principles
24 | P a g e
7.2 Synthetic Audio

• A woman faced a harrowing ordeal when she received a call from someone who sounded
exactly like her grandson, claiming to be in jail and in need of bail money. Believing it to be
her grandson, she withdrew the ransom money from the bank and were about to send it when
a bank manager intervened. They soon realized the caller had impersonated their grandson,
and they had been scammed This incident reflects a growing trend of impersonation scams
aided by advancements in voice-generating AI technology. In recent years, such scams have
been on the rise, often targeting vulnerable individuals and resulting in substantial financial
losses. AI-driven voice-generating software can replicate a person's voice with a short audio
sample, enabling scammers to convincingly mimic trusted voices40

7.3 Generated Text

• A new phone application alert highlights the potential misuse of AI in the form of ChatGPT.
While ChatGPT has legitimate applications, it can also be exploited by scammers to generate
human-like summaries and stories, making it easier to craft well-written scam emails.
Criminals can abuse this technology to increase the volume of phishing attacks, potentially
leading to users clicking malicious links or divulging personal information. This approach
diverges from traditional phishing indicators, which often involve poorly written texts and
emails. With the rise of AI technologies aiding content creation, it's increasingly difficult to
distinguish phishing attempts from legitimate communication.41

7.4 Synthetic IDs

• A woman, known to the immigration authorities as an illegal immigrant utilized two fake
passports and IDs to reside and work in a foreign country for seven years. Employing these
synthetic IDs, she secured a tenancy in a house, opened bank accounts and obtained
employment.42

8. INTERPOL’s Role in Synthetic Media Forensics

INTERPOL will also continuously work with other stakeholders including experts from the private
sector and academia for:

• Analysis of emerging synthetic media trends, tools, and techniques to provide critical insights
into the evolution of threats and opportunities, enabling proactive and targeted responses.
• Forensic authentication to identify methodologies and standards for authenticating media
content. These measures will assist investigative efforts, helping to distinguish real content
from manipulated material.

40 https://www.washingtonpost.com/technology/2023/03/05/ai-voice-scam/
41
https://abc7chicago.com/what-is-chatgpt-google-chatbot-ai-online-scams/12952645/
42
https://www.dailymail.co.uk/news/article-12990911/namibian-mother-known-home-office-illegal-
immigrant-fake-passport-lived-four-bed-house.html
25 | P a g e
 • Multi-stakeholder threat assessment of the criminal landscape to provide an understanding
of its potential misuse as well as already evidenced misuse of this emerging technology by
criminal actors.

• Capacity Building and Training for law enforcement officers on the ever-evolving synthetic
media threat landscape and innovative solutions that will help identify, detect and analyze
synthetic media files to streamline synthetic media investigations. Since the field of synthetic
media forensics is closely related to AI due to the significant role AI plays in both the creation
and detection of synthetic media, INTERPOL aims to encourage the responsible use of AI
technologies across its member countries.

Reflecting the needs and ambitions of member countries in this area, a dedicated INTERPOL
Responsible AI Lab (I-RAIL) has been established. I-RAIL aims to be the focal point for matters related
to the responsible use of AI by law enforcement. In addition to facilitating general AI awareness,
I-RAIL provides practical support to law enforcement in member countries with regards to
theresponsible use of AI, including knowledge development and exchange, agency assessment
support and tailored capacity building and training.

9. Conclusion and Recommendations

Synthetic media poses an increasingly formidable challenge to law enforcement agencies

worldwide. Its sophisticated manipulation techniques, if left unchecked, can substantially
undermine the integrity of evidence and disrupt investigations. To combat this threat
effectively, it is crucial for law enforcement agencies to gain a comprehensive understanding of
the multifaceted nature of synthetic media, including its creation, distribution, and potential impact
on investigations.

Enhancing forensic and investigative capabilities is paramount to developing robust

tools and methodologies capable of detecting and differentiating between genuine and
manipulated media. Collaborative efforts between countries, industry stakeholders,
and academic institutions are essential in pooling resources, sharing expertise, and
developing cutting-edge technologies to counter synthetic media misuse effectively.
Through ongoing vigilance, continuous technological advancements, and international
collaboration, the adverse effects of synthetic media can be mitigated, ensuring the
sustained integrity and reliability of law enforcement operations globally.

26 | P a g e
INTERPOL INTERPOL_HQ @INTERPOL_HQ INTERPOL HQ INTERPOL
Shallow Fakes

Albertina Antognini & Andrew Keane Woods*

ABSTRACT

Scholars and policymakers are rightly concerned with online

deception, especially intentional efforts to spread fake news. But the
problem of deception on social media is both subtler and more endemic
than a series of malicious actors disseminating deepfakes. While platforms
are indeed awash in fakery, many of these fakes are shallow—superficial
tweaks to one’s self-presentation—and they are of the platforms’ own
making.
Every day on social media, users place filters on their selfies, post
photos out of context, and otherwise present a fake version of their lives.
This is no accident. The ability to curate a better-than-real image is the
sine qua non of social media platforms, whose business model relies on
blurring the distinction between true and false, authentic and inauthentic,
and, ultimately, content and advertising.
We argue that this widespread superficial fakery leads to a host of
underappreciated costs. At a bare minimum, the sheer scale of deception
warrants greater scrutiny, which would require more information sharing
from the platforms. What little we do know is troubling. The platforms’
internal research shows that users, especially younger users, feel enormous
pressure to adhere to a specific ideal of beauty. The pressure to conform is
intense and manifests itself in traditionally gendered and racialized ways,
with harms often falling on already-marginalized groups. Then there are
epistemic and democratic concerns. The erosion of public trust and
political polarization are often pinned on digital echo chambers, foreign
influence campaigns, or both. But what share of the blame belongs to the

* James E. Rogers Professor of Law and Milton O. Riepe Professor of Law,

respectively, University of Arizona James E. Rogers College of Law. The authors thank
workshop participants at the Technologies of Deception conference organized by Yale Law
School’s Information Society Project, Stanford Law School’s Grey Fellow’s Forum, the
Family Law Scholars and Teachers Conference, and the University of Arizona. We are
especially grateful for feedback from Michael Boucai, Andy Coan, Derek Bambauer, Jane
Bambauer, Ellie Bublick, Joanna Grossman, Aníbal Rosario-Lebrón, George Fisher, Jill
Hasday, Lynne Henderson, Xiaoqian Hu, Mugambi Jouet, Shalev Roisman, Charisa Kiyô
Smith, Tyler Valeska, Deepa Varadarajan, and Tammi Walker.

69

Electronic copy available at: https://ssrn.com/abstract=4606149

70 PENN STATE LAW REVIEW [Vol. 128:1

fact that so much of everyday life takes place in a space that is marked by
constant, casual deception?
This Article defines shallow fakes and explains their centrality to the
social media ecosystem. It then turns normative, assessing the costs of
shallow fakes, which often slip through the hard and soft law that govern
other kinds of public information sharing, like advertising and journalism.
We end with prescriptions, chief among them a need for more
transparency around how the platforms operate.

Table of Contents

I. INTRODUCTION ................................................................................................ 71
II. WHAT ARE SHALLOW FAKES? ....................................................................... 78
A. The Core Elements............................................................................. 79
1. Superficial ................................................................................ 79
2. Commonplace .......................................................................... 80
3. Online....................................................................................... 81
4. The Self .................................................................................... 81
B. Examples of Shallow Fakes ............................................................... 82
1. The Filter.................................................................................. 82
2. The Crop .................................................................................. 83
3. The Mislabel ............................................................................ 83
4. The Product Endorsement ........................................................ 84
C. Distinguishing Deepfakes .................................................................. 85
III. PLATFORMS FOR SHALLOW FAKERY ............................................................. 86
A. The Arms Race ................................................................................... 87
B. Deception in the Algorithm................................................................ 89
C. Platform Policies on Deception......................................................... 90
IV. THE PROBLEM WITH SHALLOW FAKES ......................................................... 94
A. Gendered Harms................................................................................ 96
1. Body Dysmorphia .................................................................... 97
2. Depression and Anxiety ........................................................... 98
3. Pressure to Sexualize ............................................................... 98
4. “Real-Life Filtered Look” ...................................................... 100
5. Reinforcing Traditional Gender Roles ................................... 103
B. Racialized Harms ............................................................................ 107
1. Blackfishing and Other Forms of Appropriation ................... 107
2. Whitewashing and Exclusion ................................................. 109
C. Democratic Harms .......................................................................... 111
1. The Erosion of Expertise........................................................ 111
2. The Erosion of Public Discourse............................................ 114
V. PLATFORM REGULATION.............................................................................. 116
A. Transparency Reforms ..................................................................... 117
B. Deceptive and Unfair Trade Practices by the Platforms ................. 119
C. Other Initiatives............................................................................... 122
VI. CONCLUSION .............................................................................................. 124

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 71

I. INTRODUCTION
Social media is awash in fakery.1 Scholars and policymakers have
become especially worried about malicious disinformation tools like
“deepfakes”—hyper-realistic fake videos made with artificial
intelligence.2 But the problem of deception online is both subtler and more
endemic than a series of bad actors engaged in information warfare. Most
of today’s online fakes are actually quite shallow—superficial tweaks to
one’s self-presentation.3 Every day on social media, users place filters on
their selfies, post photos out of context, and otherwise present a digitally-
enhanced version of their lives. Unlike deepfakes, these superficial tweaks
to one’s self-presentation—which we term “shallow fakes”—are enabled
and encouraged by the platforms.
The ability to curate a better-than-real image is the sine qua non of
social media platforms. Instagram, for example, owes its start to the filter:

1. See, e.g., Suroush Vousoughi et al., The Spread of True and False News Online,
359 SCIENCE 1146, 1148–49 (2018) (examining 12 years of Twitter data and showing that
“[f]alsehood reached more people” than the truth and that users were 70% more likely to
share fake news than real news); CAILIN O’CONNOR & JAMES OWEN WEATHERALL, THE
MISINFORMATION AGE: HOW FALSE BELIEFS SPREAD 147–67 (2019) (explaining how social
networks are particularly fertile breeding grounds for misinformation); RICHARD L. HASEN,
CHEAP SPEECH: HOW DISINFORMATION POISONS OUR POLITICS—AND HOW TO CURE IT
(2022) (explaining the particular impact social media networks and fake news can have on
democratic elections and proposing legal reforms); Peter Sucio, Social Media Is Full of
Fakes—As In Fake Followers New Study Finds, FORBES (Nov. 17, 2021, 1:09 PM),
https://bit.ly/3ooJbUB (describing how over a third of top influencer’s followers are fake
accounts); Hunt Allcott & Matthew Gentzkow, Social Media and Fake News in the 2016
Election, 31 J. ECON. PERSPECTIVES 211, 219–23 (2017) (discussing data that shows the
pervasiveness of fake news on social media in the leadup to the 2016 presidential election).
2. See Bobby Chesney & Danielle Citron, Deep Fakes: A Looming Challenge for
Privacy, Democracy, and National Security, 107 CALIF. L. REV. 1753, 1757 (2019)
(describing the problem of “[t]echnologies for altering images, video, or audio . . . in ways
that are highly-realistic and difficult to detect”). There are many different definitions of
deepfakes, but they all emphasize the use of sophisticated technology, specifically artificial
intelligence (AI) and deep learning, to manipulate content and deceive consumers. Indeed,
the word “deepfake” is a “portmanteau of ‘deep learning’”—a reference to a kind of AI
algorithm—and “fakes.” James Vincent, Why we need a better definition of ‘deepfake’,
THE VERGE (May 22, 2018, 2:53 PM), https://bit.ly/3WltoCB. Deepfakes originated in
pornography, when, in 2017, one anonymous Reddit user uploaded pornographic videos
featuring celebrities; most of the celebrities were female. See Russell Spivak,
“Deepfakes”: The Newest Way to Commit One of the Oldest Crimes, 3 GEO. L. TECH. REV.
339, 345–46 (2019). For one example among many of how advocates are responding to the
deepfake problem, see Prepare, Don’t Panic: Synthetic Media and Deepfakes, WITNESS
MEDIA LAB, https://bit.ly/43sz1Sj (last visited June 24, 2023).
3. This term is distinct from “Shallowfakes,” which is defined as “videos that have
been manipulated with basic editing tools or intentionally placed out of context.” HENRY
AJDER ET AL., THE STATE OF DEEPFAKES: LANDSCAPE, THREATS, AND IMPACT 11 (2019),
https://bit.ly/3BH6Se4. Both terms address low-technology edits, but that is where their
similarities end. We are not concerned with whether the user intends to deceive or whether
it is malicious. Our use of “shallow” is meant to both imply the surface level changes we
focus on along with their perceived unimportance.

Electronic copy available at: https://ssrn.com/abstract=4606149

72 PENN STATE LAW REVIEW [Vol. 128:1

at a time when photo-sharing tools had become commonplace, the

application stood out for its image-enhancing filters, which gave photos
an attractive, sepia-tinted look.4 The popularity of Instagram’s filters set
off an arms race for digital beautification.5 Snapchat introduced video
filters, which aggressively alter the appearance of one’s face.6 Even the
most playful Snapchat filters, like those that add Harry Potter glasses or a
crown of flowers, change users’ skin color, jawline, eye shape, and more.7
TikTok also offers its own filters. The race to beautify images is so intense
that TikTok users report finding beautification filters applied to their
images even when they have selected “no filter.”8
This dynamic is largely a result of platforms’ competition to attract
users. The business model of today’s biggest social media platforms9
depends on bringing users to their platforms to monetize their attention
through advertisements.10 Users are exposed to multiple layers of

4. See SARAH FRIER, NO FILTER: THE INSIDE STORY OF INSTAGRAM xxi (2020)
(“[B]ecause of filters that initially improved our subpar mobile photography, Instagram
started out as a place for enhanced images of people’s lives. Users began to accept, by
default, that everything they were seeing had been edited to look better.”); see also Amelia
Tait, How Instagram changed our world, THE GUARDIAN (May 3, 2020, 6:00 AM),
https://bit.ly/3Olir20 (“When Instagram launched, it offered filters that people could use to
make their photos—and by extension, their lives—look more appealing.”).
5. In some ways, Snapchat’s early success can be attributed to the playful filters that
allowed users, but especially young users, a way to be silly and rebel from the pressure to
conform to the kind of look one found on Instagram; eventually, though, that would change.
See FRIER, supra note 4, at 114, 179–207.
6. See id. at 113.
7. See Andrea Navarro, Snapchat’s “Pretty” Filters Allegedly Make You Whiter,
TEEN VOGUE (May 16, 2016), https://bit.ly/41R0Fqu (describing how filters like “flower
crown” do more than they first appear, including whitening and smoothing skin, thinning
the jawline, and more).
8. See Abby Ohlheiser, TikTok changed the shape of some people’s faces without
asking, MIT TECH. REV. (June 10, 2021), https://bit.ly/43aQSwn (describing users
discovering that TikTok applied beauty filters to users who had all filters turned off).
Additionally, Tristan Harris, the Executive Director for Human Technology, states:
Unless the government acts, the competition between technology businesses’
never-ending interest in capturing human attention, will irreversibly dismantle
the information environment, accelerate polarization leading towards civil war,
degrade the mental health of a generation of children and teenagers, and break
down the basis for trust itself, leading to market collapse and near permanent
civil disorder.
Optimizing for Engagement: Understanding the Use of Persuasive Technology on Internet
Platforms: Hearing Before the Subcomm. on Commc’n, Tech., Innovation, and the
Internet, of the S. Comm. on Commerce, 116 CONG. 50, 58 (2019) (Statement of Tristan
Harris, Exec. Dir. Ctr. for Humane Tech.).
9. We are addressing here the advertising-driven social media platforms—typified by
Facebook, Instagram, YouTube, TikTok, and Snapchat. There are smaller platforms, like
many dating apps, that are not advertising-driven.
10. See TIM WU, ATTENTION MERCHANTS: THE EPIC SCRAMBLE TO GET INSIDE OUR
HEADS 5 (2016) (“Over the last century, . . . we have come to accept a very different way

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 73

advertising at once, with varying degrees of transparency. Companies pay

platforms like Instagram to present products in their users’ feeds.11 These
ads can be targeted, based on user preferences, and are typically marked
as advertisements.12 There are also influencers—social media users who
make a living depicting a particular lifestyle—who sell products and
services they are paid by companies to promote.13 These promotions are
not always identifiable as advertisements. One study found that though
nearly a quarter of all Instagram posts were advertisements, most of those
had no label signifying as much.14 Another study found that 93% of
sponsored content by top influencers did not comply with Federal Trade
Commission (FTC) guidelines for endorsements.15 This is not accidental;
softening the line between authentic and inauthentic is the reason native
advertising works.16 Even the initial invitation from social media
platforms to users traffics in this muddling of boundaries by getting users
to spend time in an advertising setting that does not feel like one.17
The muddling goes deeper than ambiguous advertisements. The
platforms have created a market where all users, influencers or not,

of being, whereby nearly every bit of our lives is commercially exploited to the extent it
can be.”).
11. See Amanda Reaume, How Does Instagram Make Money for Facebook (Meta
Platforms), SEEKING ALPHA (Dec. 1, 2021, 9:00 AM), https://bit.ly/3pVgKhE.
12. See id. (describing the different ways that ads can be presented to users).
13. As one recent description explains:
Social media influencers are people with extensive social media followings who
share content on Instagram, TikTok, Twitter, Facebook, and other social media
applications . . . . Influencers receive money from brands to promote various
products to their followers. An influencer’s ability to earn money from
promotions correlates with their number of followers.
Stasia Skalbania, Comment, Advising 101 For the Growing Field of Social Media
Influencers, 97 WASH. L. REV. 667, 669–70 (2022) (identifying four categories of followers
based on the number of followers, which begin at “nano” and end in “mega” influencers).
14. See Influencer Ad Disclosure on Social Media: A Report Into Influencers’ Rate
of Compliance of Ad Disclosure on Instagram, ADVERTISING STANDARDS AUTHORITY
REPORT 4 (Mar. 18, 2021), https://bit.ly/43eeb8A.
15. See 93% of Top Celebrity Social Media Endorsements Violate FTC Guidelines,
MEDIAKIX (May 31, 2017), https://bit.ly/3Q9o4kF.
16. See Lili Levi, A “Faustian Pact”? Native Advertising and the Future of the Press,
57 ARIZ. L. REV. 647, 665 (2015) (arguing that the “entire raison d’être [of native
advertising] is precisely to disable consumers from being able to distinguish between
editorial content and commercial propaganda—to trick consumers and end-run ad
avoidance”); see also FRIER, supra note 4, at 138 (describing the power of the unlabeled
paid post on Instagram: “[s]ince consumers are much more likely to be swayed to buy
something if friends or family recommend it, as opposed to advertisements or product
reviews, these ambiguous paid posts were effective”).
17. Take Facebook’s stated goal: “to give people the power to build community and
bring the world closer together.” Andy Wu, The Facebook Trap, Technology and
Analytics, HARVARD BUSINESS REVIEW (Oct. 19, 2021), https://bit.ly/3oi3QK1. Of course,
that is not the only goal. Connecting users and increasing their engagement with each other
“directly drive advertising revenue, the predominant mode by which Facebook captures
value, i.e., monetizes the user base that otherwise uses Facebook for free.” Id.

Electronic copy available at: https://ssrn.com/abstract=4606149

74 PENN STATE LAW REVIEW [Vol. 128:1

compete for attention and for “likes” by posting filtered, edited, or

otherwise enhanced images of themselves.18 A recent survey found that
90% of women regularly apply filters to their selfie photos.19 One of the
most popular apps ever developed—a top-downloaded app for five years
in a row, now among the handful of billion-dollar unicorn apps—is
FaceTune, which allows users to digitally alter their photos.20 As a result
of this casual deception, users spend hours every day in a world where
most of the images they see are manipulated. From the platform’s
perspective, this is a feature, not a bug.21
To date, this widespread, superficial fakery has largely gone
unnoticed. Perhaps that is because it seems at best empowering and at
worst harmless. If users want to tweak their appearance online, that is their
prerogative. One could argue that filters are an autonomy-enhancing form
of digital identity.22 Under this account, filters allow for a positive form of
play, self-expression, or self-discovery.23 Moreover, today’s social media
users are savvy—they know that what happens online is not real, so no one
is being deceived or harmed. Even if this fakery did lead to some
discernable harm, this kind of subtle deception is certainly not new:
cropping, photoshopping, and airbrushing are old techniques.24 Under any
of these accounts, shallow fakes do not merit serious consideration.

18. See FRIER, supra note 4, at xxi (“Users began to accept, by default, that everything
they were seeing had been edited to look better. Reality didn’t matter as much as aspiration
and creativity.”).
19. See Sarah Fielding, 90% of Women Report Using a Filter on Their Photos,
VERYWELL MIND (Mar. 15, 2021), https://bit.ly/3MFN7Zx.
20. See Connie Loizos, The maker of popular selfie app FaceTune just landed $135
million at unicorn valuation, TECHCRUNCH (July 31, 2019, 7:00 AM),
https://bit.ly/3Co4CII.
21. See Alexandra J. Roberts, False Influencing, 109 GEO. L.J. 81, 84 (2020)
(describing how “[a]uthenticity lies at the core of the [influencer] advertising model” which
“creates an exceptionally fertile breeding ground for deception and consumer harm”); see
also Georgia Wells et al., Facebook Knows Instagram is Toxic for Teen Girls, Company
Documents Show, WALL ST. J. (Sept. 14, 2021, 7:59 AM), https://on.wsj.com/3MHDfQ7
(“The features that Instagram identifies as most harmful to teens [things like trying to live
a perfect life online, having a perfect body, only sharing one’s best moments] appear to be
at the platform’s core.”).
22. See Sofia P. Caldeira et al., Exploring the Politics of Gender Representation on
Instagram: Self-representations of Femininity, 5 DIGEST. J. DIVERSITY & GENDER STUD.
23, 25 (2018) (“[T]here is still a sense of optimism surrounding the political potential of
self-representation on apps such as Instagram.”).
23. See, e.g., JULIE E. COHEN, CONFIGURING THE NETWORKED SELF: LAW, CODE, AND
THE PLAY OF EVERYDAY PRACTICE 53–58 (2012) (arguing for “a renewed appreciation for
the play of everyday practice” and the centrality of the idea of “play” to mature conceptions
of the networked self and information law).
24. See generally HANY FARID, FAKE PHOTOS (2019) (describing a long history of
photographic manipulation, including by leaders like Stalin and Mussolini, who regularly
doctored photographs to achieve political ends).

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 75

We disagree. At a bare minimum, the scale of visual deception

described here—and its relationship to disinformation and the overall
health of the digital ecosystem—warrants greater attention. Much of what
we know about the platforms comes from whistleblowers, like former
Facebook employee Frances Haugen, whose leaks to the Wall Street
Journal led to international outcry and a round of Congressional
hearings.25 Knowing more about the impact of social media—and the
casual deception it engenders—would require drastically more
transparency from the platforms.
What little we do know is troubling. The extent and type of shallow
fakes documented here suggest that users are under enormous pressure to
conform to standards dictated by the platforms. If users were fully in
control, we would expect filters to amplify the diversity of the online
visual field, making people as distinctly unique as they want to be. But
that is not what we find. Instead, millions of teens use the same filter that
whitens their skin to the same hue and complain that they feel drained by
the constant need to comply with a particular beauty standard.26 This
beauty standard is not simply an organic reflection of community values;
rather, it is the result of their interactions with the platforms’ algorithms.27
That this dynamic exists and that it is harmful is well known to these
platforms. Facebook’s own internal research suggests that time spent on
Facebook and Instagram has a profound negative effect on the mental
health of its users, leading to anxiety and depression.28 Independent
research corroborates these findings, noting that a significant number of
suicidal teens said their darkest thoughts were prompted by the platform
and that the firm’s products made “body image issues worse for one in

25. See the facebook files, WALL ST. J. (2021), https://bit.ly/3PiJeg1 (last visited June
24, 2023) (collecting news reports from the fall of 2021 that rely on leaked internal
Facebook documents to describe that “platforms are riddled with flaws that cause harm,
often in ways only the company fully understands”); see Reed Albergotti, Frances Haugen
took thousands of Facebook documents: This is how she did it, WASH. POST (Oct. 26, 2021,
12:00 PM), https://bit.ly/3rIqGvS (“For nearly a month, Haugen has made headlines for
her decision to blow the whistle on Facebook, testifying in front of Congress, appearing on
“60 Minutes” and on the cover of Time Magazine. Her revelations have created a firestorm.
And Facebook is reportedly considering a name change.”).
26. See Rosalind Gill, Changing the perfect picture: Smartphones, social media and
appearance pressures, CITY UNIVERSITY OF LONDON 5 (2020), https://bit.ly/3q01Mr4.
27. See infra Sections III.A and III.B.
28. See Wells et al., supra note 21. There were counter-studies—the internal research
was not all negative—but there was enough information that was worrying. See Instagram
Press Release, What Our Research Really Says About Teen Well-Being and Instagram
(Sept. 26, 2021), https://bit.ly/41Tk9ef.

Electronic copy available at: https://ssrn.com/abstract=4606149

76 PENN STATE LAW REVIEW [Vol. 128:1

three teen girls.”29 The pressure to conform to traditional norms can also
exclude nonbinary users.30
The social media ecosystem is not only deeply gendered, it is also
racialized. Reports of blackfishing are common, in which white users go
to extreme lengths, including darkening their skin, to appear Black.31
There are also simultaneous reports of whitewashing, in which filters
whiten the skin of non-white users and perpetuate various forms of digital
exclusion.32 With race, as with gender, digital tools appear to cheapen and
flatten user diversity.
Then there are epistemic and democratic concerns. What happens to
truth in a world where so much of everyday life is marked by constant
deception? And what are the implications for democracy and public
discourse? As people spend more time in online spaces where deception
is the norm, what happens to democratic deliberation? Blame for the
erosion of public trust and political polarization is often pinned on digital
echo chambers, foreign influence campaigns, or both.33 But we propose
that some share of the blame belongs to the fact that so much of everyday
life takes place in a space that is marked by sustained, but subtle,
deception.34

29. See Instagram Press Release, supra note 28.

30. See infra Section IV.A.5; see also Hattie Garlick, Why gender stereotypes are
perpetuated on Instagram, FINANCIAL TIMES (Mar. 13, 2020), https://bit.ly/44RA9PA
(“[I]n the strange world of social media, new pressures are perpetuating antiquated gender
stereotypes.”). But see Jenna Wortham, On Instagram, Seeing Between The (Gender)
Lines, N.Y. TIMES (Nov. 15, 2018), https://nyti.ms/3r2f1HV (arguing that social media
“has turned out to be the perfect tool for nonbinary people to find and model their unique
places on the gender spectrum”). Wortham identifies the tangible benefits of community
and belonging that social media provides nonbinary users, while also noting that such
platforms can still be sites of social control and hostility. Id.
31. See Wesley E. Stevens, Blackfishing on Instagram: Influencing and the
Commodification of Black Urban Aesthetics, SOCIAL MEDIA + SOC’Y 1 (Aug. 13, 2021),
https://bit.ly/3of5YCe (defining blackfishing as “a practice in which cultural and economic
agents appropriate Black culture and urban aesthetics in an effort to capitalize on Black
markets”); see also infra Section IV.B.1.
32. See Rachel Jacoby Zoldan, FaceApp Creator Apologizes for Whitewashing
“Hot” Filter, TEEN VOGUE (Apr. 25, 2017), https://bit.ly/439H1H7 (noting the outcry and
describing how the app lightens skin tones); Neha Prakash, Snapchat faces an outcry
against ‘whitewashing filters’, MASHABLE (May 16, 2016), https://bit.ly/3WniDQb; see
also infra Section V.B.2.
33. See, e.g., ELI PARISER, THE FILTER BUBBLE: HOW THE NEW PERSONALIZED WEB
IS CHANGING WHAT WE READ AND HOW WE THINK (2011) (describing how filters allow
internet users to self-select into information that confirms pre-existing biases, enables
radicalization, and generally erodes civil discourse); PHILIP N. HOWARD ET AL., UNIV. OF
OXFORD, THE IRA, SOCIAL MEDIA AND POLITICAL POLARIZATION IN THE UNITED STATES,
2012-2018 (2018), https://bit.ly/43dh4Xc.
34. The best example we have come across that links obsession over improving one’s
self to misinformation about society is a piece of excellent journalism. See Molly Young,
How Amanda Chantal Bacon Perfected the Celebrity Wellness Business, N.Y. TIMES
MAGAZINE (May 25, 2017), https://nyti.ms/3Wnnq47 (“We tend to think of ‘wellness’ as

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 77

This Article is the first to fully consider the effects of these shallow
fakes. There is little scholarship on this type of deception. That, we
suspect, has to do with the fact that the problem is “shallow”—dealing
with surface-level aesthetics and thus seen as superficial and frivolous—
not to mention that much of the harm is felt by already-marginalized
communities. The closest analog is a small literature on influencer
marketing.35 This gap stands in stark contrast to deepfakes, where there
has been a considerable scholarly and policy response.36 In 2020, Congress
passed two bills—the U.S. National Defense Authorization Act
(“NDAA”) and the Identifying Outputs of Generative Adversarial
Networks (“IOGAN”) Act—with provisions aimed at addressing the
deepfake problem.37 Social media platforms have also responded. In the
last three years, Twitter,38 Facebook, TikTok, Snapchat, and YouTube
have updated their terms of service to explicitly address deepfakes.39
Deepfakes are undoubtedly a serious problem. But focusing solely on
deepfakes—which are now banned under most platforms’ inauthentic
content policies—provides the false sense that what remains is authentic.40
We aim to remedy the gap in scholarship and policy, which is
especially notable given that there are several ways the law could

the province of swoony liberal elites, but it does, in fact, blossom at both cultural poles.”).
A similar point has also been made in discussing the specific practice of “stealth
marketing,” in which Ellen Goodman argues that it “harms . . . by degrading public
discourse and undermining the public’s trust in mediated communication.” Ellen P.
Goodman, Stealth Marketing and Editorial Integrity, 85 TEX. L. REV. 83, 87 (2006).
35. See, e.g., Roberts, supra note 21; Skalbania, supra note 13, at 669–70; Annamarie
White Carty, Cancelled: Morality Clauses in Influencer Era, 26 LEWIS & CLARK L. REV.
565 (2022); Megan K. Bannigan & Beth Shane, Towards Truth in Influencing: Risks and
Rewards of Disclosing Influencer Marketing in the Fashion Industry, 64 N.Y.L. SCH. L.
REV. 247 (2019/2020).
36. See Chesney & Citron, supra note 2.
37. See William M. (Mac) Thornberry National Defense Authorization Act for Fiscal
Year 2021, Pub. L. No. 116-283, 133 Stat. 3388 (2021); Identifying Outputs of Generative
Adversarial Networks (IOGAN) Act, Pub. L. No. 116-258, 134. Stat. 1150 (2020) (to be
codified in scattered sections of the U.S. Code). The 2021 NDAA was passed over the
President’s veto, while the President signed the IOGAN Act. See Scott Briscoe, U.S. Laws
Address Deepfakes, TODAY IN SECURITY (Jan. 12, 2021), https://bit.ly/3MnzjTj.
38. Although Twitter now goes by the name X, we will continue to refer to the firm
as Twitter given that this is the commonly used name for the firm’s service.
39. Most of these are outright bans on any manipulated content that could lead to
harm. For an example, see Vanessa Pappas, Combating misinformation and election
interference on TikTok, TIKTOK NEWSROOM (Aug. 5, 2020), https://bit.ly/3OskL7y (“Our
Community Guidelines prohibit misinformation that could cause harm to our community
or the larger public, including content that misleads people about elections or other civic
processes, content distributed by disinformation campaigns, and health misinformation.”)
But rather than taking down “synthetic and misleading media,” Twitter often will label
Tweets “to help people understand their authenticity and to provide additional context.”
Synthetic and manipulated media policy, TWITTER, https://bit.ly/42OrNrm (last visited
June 25, 2023).
40. See infra Section IV.C.

Electronic copy available at: https://ssrn.com/abstract=4606149

78 PENN STATE LAW REVIEW [Vol. 128:1

intervene.41 The most obvious and most pressing need is for more
transparency from the platforms.42 There is a growing demand for national
transparency legislation, and we explain how such legislation would
alleviate some of the concerns raised here. Because the platforms are
advertising networks, we also explain how existing rules promulgated by
FTC that prohibit deception in the marketplace can apply more broadly to
social media fakery.43 Finally, there are other measures, like industry
norms and multistakeholder initiatives, that could help. Indeed,
multistakeholder initiatives have had success with revising social media
policies in related areas, especially with regard to violent and extremist
content.44
The Article proceeds in four parts. Parts II and III are descriptive,
outlining the many ways in which platforms provide the tools for users to
engage in shallow fakery. Part II provides a taxonomy of different types
of shallow fakes, and Part III explains how platforms promote and
encourage them, regardless of user preference. In Part IV, the Article turns
normative, assessing the costs of shallow fakes, in addition to possible
benefits. Part V looks ahead to implications for regulators, scholars, and,
ultimately, users.

II. WHAT ARE SHALLOW FAKES?

We define shallow fakes as superficial, commonplace deceptions
about one’s self-presentation online. Note that this definition does not turn
on intent; we are interested both in actors who intend to deceive, for
whatever reason, and those who do not. This definition covers a wide
range of image-enhancement techniques, including those that were
available before the rise of today’s digital networks, such as airbrushing.
Importantly, we begin from the premise, as Erving Goffman has observed,
that “life itself is a dramatically enacted thing.”45 All of us perform a kind
of “presentation of the self” every day, often by attempting to enhance our
image in subtle, and not-so-subtle, ways.

41. See infra Part IV.

42. See infra Section V.A.
43. See infra Section V.B.
44. See infra Section V.C.
45. ERVING GOFFMAN, THE PRESENTATION OF THE SELF IN EVERYDAY LIFE 72 (1959).
Indeed, we accept that performance is part of every social interaction and that deception
routinely occurs as a descriptive matter. See id. at 249 (“All the world is not, of course, a
stage, but the crucial ways in which it isn’t are not easy to specify.”). In this Article, we
are concerned with the deceptions in the presentation of one’s self online that are offered
by the various platforms.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 79

The aim in Part II is principally descriptive.46 This Part begins by

setting out core elements of shallow fakes; it then provides some
prototypical examples. Much of the data we rely on here is necessarily
incomplete because it is based on user surveys, leaked internal documents,
and the platforms’ own policies; these are tiny snapshots into how
platforms function but, by definition, are not comprehensive. We end Part
II by distinguishing shallow fakes from deepfakes, which have received
the lion’s share of attention thus far.

A. The Core Elements

Shallow fakes consist of four core elements. First, shallow fakes are
superficial tweaks to one’s image that are made without any specific intent
and might therefore be understood as harmless. Second, they are
commonplace. Third, they take place online, which makes them different
from what we are accustomed to offline.47 Fourth, and finally, they affect
one’s self-presentation; they are not principally about other people or
about facts at large.

1. Superficial
Shallow fakes are superficial edits to observable characteristics. They
are meant to improve the user’s physical appearance. For this reason, they
are typically seen as innocuous. The platforms themselves describe this
kind of enhancement—the use of filters, lighting, and crops—as harmless.
Instagram’s policy for deceptive material, for example, only applies to
edits that are “beyond adjustments for clarity or quality,” which leaves
considerable room for image enhancement.48 Clarifying the platform’s
stance on deepfakes, one platform spokesperson explained that content is
regularly manipulated “often for benign reasons” and that this content is
considered authentic.49 The seemingly benign nature of this widespread
media manipulation is why Instagram can describe itself as “an authentic
and safe place for inspiration and expression.”50 The paradox of shallow

46. Significantly, we are not critiquing the users who deploy the techniques we
describe. In Part III, we argue that the reason there is so much fakery on platforms is the
result of deliberate choices by the social media platforms.
47. In addition to the “front regions” and “back regions” that Erving Goffman
identified as crucial to social performance, he also identifies “the outside.” GOFFMAN,
supra note 45, at 134–35. Online, there is no “back region” or “outside”—the audience is
only privy to the front regions, which raises a set of issues specific to social media.
48. Monika Bickert, Enforcing Against Manipulated Media, META (Jan. 6, 2020),
https://bit.ly/3qZ6Kob.
49. Id.
50. Community Guidelines, INSTAGRAM, https://bit.ly/3piTOsJ. The description of
Community Guidelines continues: “Remember to post authentic content, and don’t post
anything you’ve copied or collected from the Internet that you don’t have the right to post.”

Electronic copy available at: https://ssrn.com/abstract=4606149

80 PENN STATE LAW REVIEW [Vol. 128:1

fakes is that they are subtle and superficial, which makes them less
suspicious and, in turn, gives them enormous reach.51

2. Commonplace
Making changes to one’s appearance is the norm on today’s social
media platforms. A survey conducted in the United Kingdom found that
90% of women aged 18–30 reported using a filter before posting online
photos.52 The purpose of these filters is to physically alter one’s
appearance—“to even out skin tone, reshape [the] jaw or nose, shave off
weight, brighten or bronze skin, and whiten teeth.”53 Unsurprisingly, the
same survey found that women feel “bombarded” and “overwhelmed” by
the pressure to look as good as the other filtered images they see online.54
One person explained, “[I]t is everywhere, all the time, and social pressure
to look a certain way is very real.”55
An indication of just how commonplace digital image distortion has
become is the success of photo editing apps such as FaceTune,56 an
extraordinarily popular app which claims to “effortlessly enhance every
selfie.”57 Its effects are palpable: “FaceTuning your jawline [has become]
the Instagram equivalent of checking your eyeliner in the bathroom of the
bar.”58 Within a year of its release, FaceTune was the number-one
downloaded app in the “photo and video” category on Apple’s platform in
120 countries.59 By 2018, it had spent four years as the most-downloaded
paid app worldwide and across all of Apple’s app categories.60 Lightricks,
the company that makes FaceTune, is valued at $1 billion dollars, having
recently raised $135 million in Series C funding.61 The app has been
downloaded 180 million times.62

51. See 1 J. THOMAS MCCARTHY, MCCARTHY ON TRADEMARKS AND UNFAIR

COMPETITION § 2:22 (5th ed. 2020) (“[I]f the deception is truly effective, the consumer may
not even be aware of it.”).
52. See Gill, supra note 26, at 36.
53. Id. at 35.
54. Id. at 28.
55. Id.
56. See Loizos, supra note 20 (noting that the app “empowers users to cover their
gray hairs, refine their jaw lines and reshape their noses”).
57. FACETUNE, https://bit.ly/42XKwA7 (last visited Sept. 12, 2023).
58. Jia Tolentino, The Age of Instagram Face, THE NEW YORKER (Dec. 12, 2019),
https://bit.ly/3CMehJn.
59. See Randy Nelson, These Apps and Games Have Spent the Most Time at No. 1
on the App Store, SENSOR TOWER (July 2018), https://bit.ly/3CIIER8.
60. See id.
61. See Loizos, supra note 20.
62. See id.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 81

3. Online
Another key component of our definition of shallow fakes is that they
occur online. Online deception is a different, and more difficult, problem
than offline deception for at least two reasons. First, online deception is
harder to identify. When one sees an airbrushed billboard of a celebrity or
a model, one understands at some level that the image is not a true
representation of a real person. Also, the billboard is clearly understood to
be an advertisement because all billboards are advertisements. Online, the
space between the “real” and the “fake” is much narrower. This is both
because of who is engaging in the deception and the context in which it
occurs. While comparing social media filters to airbrushing in magazines,
one survey respondent explained, “What’s different about social media is
these aren’t just celebrities and supermodels, these are people you know.
The feeling of ‘why isn’t that me’ becomes even stronger and more
significant.”63
Second, the context is less clearly defined, which means that casual,
online deception seeps into all aspects of one’s life. A magazine is a
highly-stylized product, and it is something you can pick up and put down.
Even if you access a magazine digitally, it has some distance from your
everyday life and you know and expect it to be an idealized version of real
life. Online airbrushing, however, is something everyone—including your
friends, classmates, and colleagues—does all the time. In the aggregate,
we start to live in a world where it can be hard to separate fact from fiction.

4. The Self
Shallow fakes address how individuals present themselves and their
lives online in a deceptive way. It is not deception about others or about
the world at large. This self-focused deception can occur in many forms,
but the basic idea is that rather than choosing to reflect a mere snapshot or
a moment in time, people, intentionally or not, curate the images they post
in a way that distorts their lives. As one of the interviewees in a study
addressing online deception noted about the kinds of things people post to
social media, “[T]his isn’t a realistic perception of everyday life. Things
go wrong but we only want other people to see the perfect bits. You can
so easily make people think you lead this picture perfect life when for most
people this is not the case.”64
This is not gross deception. It is designed to be slight—shallow fakes
are meant to be small enough to be believable. This is why, for example,
a British study of teenage girls’ use of social media found that teens feel
pressure to post images that are perfect—which requires using filters—but

63. Fielding, supra note 19 (quoting Tess Brigham).

64. Gill, supra note 26, at 23.

Electronic copy available at: https://ssrn.com/abstract=4606149

82 PENN STATE LAW REVIEW [Vol. 128:1

not so perfect that the images appear wholly unrealistic. As one subject
said, “I don’t want my pictures to look too fake . . . . I want it to look as
natural as possible, even though I’m wearing makeup. I want it to look like
I haven’t put a filter on.”65
This deception in the presentation of one’s self, or one’s life, is what
distinguishes shallow fakes from other forms of fakery, like fake news,
that have received so much attention. Indeed, one of the most widely
discussed frameworks for identifying and distinguishing different types of
fake news does not even mention the subtle deception in self-presentation
of the kind we describe here.66

B. Examples of Shallow Fakes

Shallow fakes are everywhere, but they are not all the same. In this
Section we provide a few paradigmatic examples to fill out our definition.
While we discuss each fake separately, they often appear in conjunction.
We do not mean to critique any particular user for relying on these
techniques, but we do argue that this fakery is harmful.67 In Part IV, we
discuss why.

1. The Filter
Perhaps the most widespread deception today is the use of the filter:
a tool that digitally enhances the appearance of a person or object. Filters
can be basic, like changing the light and color in a photo. But they can be
more sophisticated and now increasingly rely on artificial intelligence to
change the physical appearance of an image. For example, it is common
for people to use filters that change their jawline, eye shape, skin tone, and
more.68 These beautification filters are effective primarily because they are

65. Id. at 26.

66. See Claire Wardle, Fake news. It’s complicated, FIRST DRAFT NEWS (Feb. 16,
2017), https://bit.ly/3MSz0RH; see also FIGHTING FAKE NEWS WORKSHOP REPORT, YALE
INFORMATION SOCIETY PROJECT (2017), https://bit.ly/437iXFq.
67. We would, however, like to take this opportunity to criticize the humblebrag,
which is more of a misdirection than a “shallow fake,” but worth calling out, nonetheless.
The humblebrag is presented as a complaint about something—say, the weight of the gold
medal hanging around one’s neck—but the complaint is an excuse to boast about that very
thing. See generally HARRIS WITTELS, HUMBLEBRAG: THE ART OF FALSE MODESTY (2012).
Humblebrags are just one example of a larger pattern in which people explain their
motivations one way when they are, in fact, another. Law professors on Twitter are prime
offenders.
68. See Tolentino, supra note 58. Tolentino writes:
Snapchat . . . has maintained its user base in large part by providing photo filters,
some of which allow you to become intimately familiar with what your face
would look like if it were ten per cent more conventionally attractive—if it were
thinner, or had smoother skin, larger eyes, fuller lips.
Id.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 83

subtle. As one beauty blogger said, “If done properly, it should be hard to
tell you’ve used it.”69 They are both understated and ubiquitous.

2. The Crop
Another way that online media can deceive is by allowing a user to
present an image decontextualized from any surrounding facts. Cropped
out of the frame is more information, like a broader context or another
perspective. Cropping photos as a means of deception has a long history.70
Today, the cropped photo is standard fare on social media. Cropping can
be done to zoom in and make an image more visible, but it can also be
done to deceive, or to hide a broader setting. One classic example of the
way a crop can alter one’s surroundings is the sandbox masquerading as a
beach.71 In social media, it is also common to crop out ring lights, staging
equipment, and other tools used to create a highly constructed, yet
seemingly natural, photograph.72 This is the reason for the increasingly
popular “behind the scenes” shot, which exposes the “ridiculous reality”
behind many Instagram posts.73

3. The Mislabel
Content can be misleading not only because it is cropped, or taken
out of context, but also because it is mislabeled. For example, a filtered
image accompanied by the label “#nofilter” implies that the user applied
no filter when they might have. Researchers studying the use of the
“#nofilter” label found that about 12% of those labeled as such had a filter
applied to them.74

69. Olivia Solon, FaceTune is conquering Instagram—but does it take airbrushing

too far?, GUARDIAN (Mar. 9, 2018, 3:01 AM), https://bit.ly/3qcnO9I.
70. Stalin, for example, famously cropped and photoshopped his political enemies
out of photographs. DAVID KING, THE COMMISSAR VANISHES: THE FALSIFICATION OF
PHOTOGRAPHS AND ART IN STALIN’S RUSSIA 13 (1997) (“Many photographic deletions were
not the result of retouching at all but of straightforward cropping. Art departments have
always cropped photographs on aesthetic grounds, but in the Soviet Union cropping was
also used with political objectives in mind.”).
71. See Olivia Devereux-Evans, Influencer, 20, goes viral with TikTok videos
showing her glamorous ‘beach’ shots are actually made with buckets of sand and a kiddie
pool in the backyard, DAILYMAIL ONLINE (April 24, 2022, 5:43 AM),
https://bit.ly/3N0eKO1. Interestingly, the influencer’s post describing her fakery, and
being transparent about how she manufactured an image, is what went viral. See id.
72. See Taylor Lorenz, The Instagram Aesthetic Is Over, THE ATLANTIC (Apr. 23,
2019), https://bit.ly/42gAo54.
73. See Rachel Hosie, An Instagram influencer shares behind-the-scenes videos
showing the ridiculous reality behind her glamorous photos, INSIDER (Oct. 15, 2019, 7:44
AM), https://bit.ly/438ta4k.
74. See Renee Engeln, The #nofilter Lie, PSYCH. TODAY (July 23, 2019),
https://bit.ly/43tUhGJ. If the 12% number holds true across Instagram, then that would
mean that “there are roughly 30 million phony #nofilter pics on Instagram.” Id.

Electronic copy available at: https://ssrn.com/abstract=4606149

84 PENN STATE LAW REVIEW [Vol. 128:1

This labeling sleight of hand is consequential in the digital realm:

psychological research shows that people are more likely to believe a false
statement if it is accompanied by a photograph.75 Moreover, images are
much more likely to be shared by people and promoted by the algorithms
that sort social media content.76 This kind of mislabeling can thus reach
many users.

4. The Product Endorsement

One of the most common forms of deception online is a paid or
sponsored post that endorses a product and is not obviously labeled as an
advertisement. Advertising is ubiquitous on social media platforms—they
are, after all, advertising platforms.77 Yet one customary form of
advertising is not disclosed as such. Consider the following scenario: a
beer brand runs an advertisement on Instagram where sponsored posts
appear and promote the beer. This post is typically obvious as an
advertisement and adequately identified. But the same brand could also
privately pay a popular influencer to post photos of themselves enjoying
that beer on Instagram.78 These influencers’ posts do not always come with
a disclaimer identifying them as advertisements: one recent study by the
British Advertising Standards Authority found that nearly a quarter of all
Instagram posts were advertisements, and yet only one third of those ads
were labeled as such.79 The study looked at 24,000 Stories on Instagram

75. See Deryn Strange et al., Photographs Cause False Memories for the News, 136
ACTA PSYCHOLOGICA 90, 90–94 (Jan. 2011) (reporting the results of an experiment that
found people were more likely to remember false news reports and with more confidence
if those reports were accompanied by photographs).
76. Yiyi Li & Ying Xie, Is a Picture Worth a Thousand Words? An Empirical Study
of Image Content and Social Media Engagement, 57 J. MKTG. RSCH. 1, 1 (Nov. 18, 2019),
https://bit.ly/45C427W.
77. See Matthew Johnston, How Does Facebook (Meta) Make Money?,
INVESTOPEDIA (Jan. 10, 2023), https://bit.ly/45u6ld7 (“Meta Platforms (META), the
company that owns Facebook, primarily makes money by selling advertising space on its
various social media platforms. Major competitors include Apple (AAPL), Alphabet
(GOOGL) Google and YouTube, Tencent Music Entertainment Group (TME), Amazon
(AMZN), and X Corp (formerly Twitter).”).
78. It is difficult to find data on this topic. Model and influencer Emily Ratajkowski
explains the terms of one such exchange the following way:
A large hotel conglomerate had just opened a new luxury resort in the Maldives.
The hotel cost $400 million to build . . . . The hotel group needed to generate
awareness, and having me visit and tag their account and the location was
valuable to them. For this kind of advertisement, I was able to make a shit ton of
money just by vacationing here for five days and posting the occasional picture.
EMILY RATAJKOWSKI, MY BODY 87 (2021).
79. See Influencer Ad Disclosure on Social Media, supra note 14, at 4.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 85

across a three-week period and found that nearly 3,800 of those were paid
advertisements with no label.80

C. Distinguishing Deepfakes
Finally, distinguishing shallow fakes from the better-known—and
more regulated—concept of deepfakes is analytically useful. The policy
responses to deepfakes are also important to briefly consider in describing
the phenomenon, given how they exacerbate the lack of attention paid to
shallow fakes and normalize the presence of shallow fakes online.
The core definition of a deepfake is that it uses artificial intelligence
to create images that trick the eye—the kind of computer-generated
images that once were available only to a well-resourced movie studio.81
Shallow fakes, however, do not turn in any meaningful way on the use of
high technology.82 As our examples above show, consumers of online
content can be misled by low-tech manipulations, like showing only part
of an interaction, either from one angle or from one perspective.83
Also, deepfakes are described as something that is done to images of
another—not to oneself. The illustrations provided by Bobby Chesney and
Danielle Citron in their foundational article on deepfakes generally feature
a sophisticated actor using digital tools to manipulate an image of someone
else.84 Shallow fakery, on the other hand, is something we do to ourselves.
Where deepfake scholars are worried about “the creation of realistic
impersonations out of digital whole cloth,”85 shallow fakes are people
merely impersonating better versions of themselves. The problem then is
not that someone else might be falsely portrayed as endorsing a product,
service, idea, or politician;86 it is that people are portraying false
presentations of themselves every day.
The discussion on deepfakes further frames the problem of online
deception around bad actors with an intent to deceive. But much of online

80. See id. at 3; see also Keisha Phippen, Are You Influencing Responsibly?, NAT’L
L. REV. (Apr. 7, 2021), https://bit.ly/43v3oXT (reporting the results of a study by the
British Advertising Standards Authority).
81. See Chesney & Citron, supra note 2, at 1763 (“As the volume and sophistication
of publicly available deep-fake research and services increase, user-friendly tools will be
developed and propagated online, allowing diffusion to reach beyond experts.”).
82. Although they can involve more sophisticated technology. See discussion infra
Part III.
83. One well-documented problem is when someone posts police body camera
footage that is taken out of context. See Emmeline Taylor & Murray Lee, The Camera
Never Lies?: Police Body-Worn Cameras and Operational Discretion, in POLICE
VISIBILITY: PRIVACY, SURVEILLANCE, AND THE FALSE PROMISE OF BODY-WORN CAMERAS
80–95 (Bryce Clayton Newell ed., 2021) (describing how police-worn camera video clips
are often taken out of context).
84. See Chesney & Citron, supra note 2, at 1776.
85. Id. at 1758.
86. See id. at 1774.

Electronic copy available at: https://ssrn.com/abstract=4606149

86 PENN STATE LAW REVIEW [Vol. 128:1

deception is done with mixed motives, without malice, or maybe even

without any specific intent. Focusing on actors’ intent overly constrains
the conversation about online deception. It also places the emphasis on the
individual users, rather than on the policy promoted by the platforms. Yet
the users are often merely responding to the incentives created by the
platform market.87
Following calls from scholars and policymakers, the platforms have
taken deepfakes seriously. Nearly all platforms ban deepfakes, and all try
to root out what they describe as “inauthentic content.”88 This response to
eradicating deepfakes assumes there is a meaningful distinction between
“authentic” and “inauthentic” content. Indeed, one of the leading harms
associated with deepfakes is that “a skeptical public will be primed to
doubt the authenticity of real audio and video evidence.”89 Not only does
this assertion suppose that a discernible line exists between authentic and
inauthentic content, but it also characterizes as authentic a digital world
that is deeply inauthentic. Carving out a set of deepfakes and calling them
“inauthentic content” helps to legitimize the constant flood of shallow
fakes that fill our platforms daily.
Fleshing out the problem of shallow fakes, as a companion to
deepfakes, will hopefully redirect the conversation and the reform efforts
currently underway.

III. PLATFORMS FOR SHALLOW FAKERY

The rise of shallow fakes is not accidental—it is the result of
intentional design choices by social media platforms with the goal of
attracting as many users as possible, as young as possible, in order to keep
them online as long as possible. Platforms facilitate the use of shallow
fakes by creating an ecosystem where applying increasingly aggressive
beautification filters is the norm. The platforms also rely on algorithms
and other dark patterns that leave the user with little control over the
content they are exposed to online. Finally, platforms maintain a set of
policies that assume there is a definitive line between “inauthentic” and
“authentic” content on the platform, thereby making shallow fakes
difficult to identify and address as a distinct phenomenon.

87. See discussion infra Part III.

88. See, e.g., Inauthentic Behavior, FACEBOOK TRANSPARENCY CENTER
https://bit.ly/43XtUK8 (last visited June 25, 2023) (“In line with our commitment to
authenticity, we do not allow people to misrepresent themselves on Facebook, use fake
accounts, artificially boost the popularity of content or engage in behaviors designed to
enable other violations under our Community Standards.”).
89. Chesney & Citron, supra note 2, at 1785 (emphasis added).

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 87

A. The Arms Race

Shallow fakes start with one person tweaking a photo, followed by
another person doing so to compete with the “perfection” of the first, and
then another, until the Internet is awash in unreal and unrealistic images.
At least two different kinds of “arms races”90 lead to this cascade of fakery.
The first race is among individual users. The second is between the various
platforms.
In a world where people vie for attention and swipes, if one person
uses beauty filters to enhance their image, the incentive is for everyone
else to do the same.91 The effect goes well beyond influencers; indeed, all
“users know they will be judged” by the “followers, likes, and comments”
they seek for various reasons, including “personal validation, social
standing, and even financial reward.”92 It is against this backdrop that
filters help “to ease the pressure of gaining likes and followers.”93 And the
pressure does build. As one study participant put it, “[E]ven when you say,
oh, I’m fine, that doesn’t bother me, in the back of your mind, it is still
like, well, everyone else on Instagram looks this way.”94 Data on teen
users, in particular, show that teens take “dozens of different angles of the
same shot, finding the perfect one, then edit[] away their imperfections
before posting.”95 If their photo does not get enough likes, they delete it.96
This user activity does not occur in a vacuum.97 It is a direct result of
the second arms race, which is taking place among the platforms—to have
people use, and remain on, their sites. What that means for each platform
varies. The purpose of the Instagram filter, which “give[s] Instagrammers

90. Tristan Harris used the term “arms race” to describe the pressure the platforms
experience to “beautify” their users. See ELISE HU, FLAWLESS: LESSONS IN LOOKS AND
CULTURE FROM THE K-BEAUTY CAPITAL 132 (2023) (quoting Tristan Harris).
91. In chronicling the rise of Instagram, Sarah Frier notes how it affected everyone
on the platform, becoming “a tool for crafting and capitalizing on a public image, not just
for famous figures but for everybody.” FRIER, supra note 4, at 128.
92. Id. at 233.
93. Id. at 173.
94. Gill, supra note 26, at 30.
95. FRIER, supra note 4, at 114.
96. See id. (explaining that “they would often delete pictures if they didn’t get 11
likes,” which “was the number of likes that would turn a list of names below an Instagram
post into a number—a space-conserving design that had turned into a popularity tipping
point for young people”). In a hall-of-mirrors sort of way, teens would have separate
accounts called “finstas,” short for “fake Instagram” accounts where they could post
images that were more realistic and unedited. Id. at 182–83. These accounts were, for the
most part, private. Id. This is also the reason for the widespread meme known as “felt cute,
might delete.” See Feeling Cute, Might Delete Later, KNOW YOUR MEME,
https://bit.ly/3sRUHda (last visited Sept. 12, 2023).
97. See, e.g., FRIER, supra note 4, at 278–79 (“Instagram isn’t designed to be a neutral
technology, like electricity or computer code. It’s an intentionally crafted experience, with
an impact on its users that is not inevitable, but is the product of a series of choices by its
makers about how to shape behavior.”).

Electronic copy available at: https://ssrn.com/abstract=4606149

88 PENN STATE LAW REVIEW [Vol. 128:1

permission to present their reality as more interesting and beautiful than it

actually was” is also “exactly what would help make the product
popular.”98 Instagram rewards likes, comments, and followers.99 The
platform gives users nearly immediate feedback on their posts, and the
images that repeatedly lead to the “best” results according to Instagram’s
metrics are “airbrushed selfies, crazy action shots, and scantily clad
influencers.”100 YouTube, on the other hand, rewards creators for how
much time the user spends watching their content.101 YouTube measures
that time by considering the percentage of a video viewed and the average
duration that the user watched.102 To succeed according to YouTube’s
metrics then, creators have to alter their format and transition to longer
videos, like the “15-minute makeup tutorial videos.”103 They also must
tailor their material to keep viewers watching, which might mean
presenting off-the-wall conspiracy theories.104
All social media platforms share the need to grow their user-bases,
and this enters them into an arms race amongst themselves to do just that.
This has meant that each company attempts to out-offer the other; it has
become an inevitable feature of the marketplace for customers. In the
context of shallow fakes, if one company employs a beauty filter that
attracts customers, that will cause the other companies to develop
something similar.105 This competition explains why Snapchat, which
initially began as an antidote to those apps that “conform to unrealistic
notions of beauty or perfection” and aimed to create a space for users “to
be funny, honest, or whatever else you might feel like at the moment,”106
spent $150 million to buy a company, Looksery, that uses facial
recognition technology to “photoshop video chats and messages in real
time.”107 It explains why Facebook, after failing to acquire Snapchat,
bought the Masquerade app, which allows people to digitally enhance their

98. Id. at 23.

99. See id. at 234.
100. Id. at 238.
101. See id. at 233.
102. See id.
103. Id.
104. See id.
105. As Tristan Harris has said, each company competes to provide resources, like
beauty filters, that the other does not have. See #1736 - Tristan Harris & Daniel
Schmachtenberger, THE JOE ROGAN EXPERIENCE, SPOTIFY (Nov. 2021),
https://spoti.fi/3JwPBs2 (“[I]f one attention company doesn’t add the beautification filter,
the other one will.”).
106. Sophia Bernazzani, A Brief History of Snapchat, HUBSPOT (Oct. 29, 2019),
https://bit.ly/3MTu1QD (quoting Snapchat founder Evan Spiegel).
107. Alyson Shontell, Snapchat buys Looksery, a 2-year-old startup that lets you
Photoshop your face while you video chat, BUSINESS INSIDER (Sept. 15, 2015, 3:36 PM),
https://bit.ly/3MF5H3Y.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 89

selfies with filters like Snapchat’s.108 And it explains why the

beautification app FaceTune is allegedly worth more than a billion
dollars.109 This race-to-the-bottom dynamic also explains why the
platforms have been caught applying slight beautification filters by default
without warning users, even where the user has not selected a filter.110

B. Deception in the Algorithm

Social media platforms not only provide a medium where fakery by
users is the norm, but they also employ dark patterns, which are user
interfaces “whose designers knowingly confuse users, make it difficult for
users to express their actual preferences, or manipulate users into taking
certain actions.”111 The premise of the algorithms is to show a user content
they will like, presumably based on the user’s past actions on the
platform.112 Users are accustomed to a feed of information that regularly
suggests the next article, the next link, the next video. That feed, however,
does not just reflect user preferences. Because the feed is optimized to,
above all, keep a user engaged, it also manipulates those preferences,
nudging the viewer in one direction or another. Suppose a user searches
YouTube for “cooking videos.” The results might also recommend a video
of a lion wrestling with a crocodile, if it thinks that user will watch it. In
this way, the algorithm has deceived the user—it purported to show
relevant results and instead it showed results a user might click and watch
just because the video is enticing. A Pew Center study confirmed this
phenomenon, revealing that the YouTube recommendation system pushed
users into watching “progressively longer and more popular content,”
regardless of what had been previously viewed.113

108. See Paresh Dave, Facebook buys Masquerade, app company that competes with
Snapchat’s lenses, L.A. TIMES (Mar. 9, 2016, 10:24 AM), https://bit.ly/43mDaH4.
109. See Loizos, supra note 20.
110. See Ohlheiser, supra note 8.
111. See Jamie Luguri & Lior Jacob Strahlevitz, Shining a Light on Dark Patterns,
13 J. LEGAL. ANAL. 43 (2021) (reporting the results of studies that illustrate how dark
patterns work).
112. See Clodagh O’Brien, How Do Social Media Algorithms Work?, DIGIT. MKT.
INST. (Jan. 19, 2022), https://bit.ly/3BRGXAa (“Algorithms are used on social media to
sort content in a user’s feed. With so much content available, it’s a way for social networks
to prioritize content they think a user will like based on a number of factors.”).
113. Aaron Smith et al., Many Turn to YouTube for Children’s Content, News, How-
To Lessons, PEW RESEARCH CENTER (Nov. 7, 2018), https://bit.ly/3MBz2vU. One of the
findings shows that 28% of the “unique videos” recommended to users “were
recommended more than once over the study period, suggesting that the recommendation
algorithm points viewers to a consistent set of videos with some regularity. In fact, a small
number of these videos (134 in total) were recommended more than 100 times.” Id.
Similarly, “regardless of whether the initial video was chosen based on date posted, view
count, relevance or user rating,” the YouTube algorithm “consistently suggested more
popular videos.” Id.

Electronic copy available at: https://ssrn.com/abstract=4606149

90 PENN STATE LAW REVIEW [Vol. 128:1

This dynamic is precisely what makes TikTok’s algorithm famously

addictive. The algorithm shows users things that it knows will keep them
on the platform, based only in part on the users’ interests.114 One
interpretation of the algorithm is that it seems to know you better than you
know yourself.115 Perhaps. But revealing one’s true self is not the
algorithm’s purpose; its purpose is, above all else, to keep users on the site.
In so doing, the algorithm ends up showing users a specific slice of the
internet, one that is being promoted by companies rather than by the will
of the individual user.
Take the example of on-line pornography: PornHub, which provides
free online pornography, relies on algorithms of the same kind used by
other large companies like Amazon, Netflix, and Facebook.116 These
algorithms track data to learn about their users, including their search
histories, location, and times when they are online. Even by merely
aggregating this data, they direct the content that users see. Thus, someone
with otherwise unconventional sexual desires might find themselves
pushed by the data into “very stereotyped, often sexist, often racist ways,
and also just with a narrow-minded view of sexuality.”117 The result, as is
true on other online platforms, is that “[o]nline-porn users don’t
necessarily realize that their porn-use patterns are largely molded by a
corporation.”118 What is presented as the mere extension of one’s
preferences is, in fact, shaping those preferences, with the ultimate goal of
increasing time spent on the platforms. Today’s platform algorithms do
not just reflect users’ preferences—they also mold them.

C. Platform Policies on Deception

Platform policies have begun to address the problem of online
deception. But in doing so, they have unwittingly exacerbated the problem
of shallow fakes. As an initial matter, the policies are too broad to
implement effectively. The policies are also inconsistent. They prohibit

114. See Ben Smith, How TikTok Reads Your Mind, N.Y. TIMES (Dec. 5, 2021),
https://bit.ly/3qcm5RC.
115. See id. (“[T]he app is shockingly good at reading your preferences and steering
you to one of its many ‘sides,’ whether you’re interested in socialism or Excel tips or sex,
conservative politics or a specific celebrity. It’s astonishingly good at revealing people’s
desires even to themselves.”).
116. See Joe Pinsker, The Hidden Economics of Porn, THE ATLANTIC (Apr. 4, 2016),
https://bit.ly/3OzykSD.
117. Id. Shira Tarrant, author of THE PORNOGRAPHY INDUSTRY, elaborates: “If you
are interested in something like double oral, and you put that into a browser, you’re going
to get two women giving one guy a blowjob . . . you’re not likely to get two men or two
people giving a woman oral sex.” Id.
118. Id.; see also AMIA SRINIVASAN, THE RIGHT TO SEX 67–68 (2021) (describing
how “free online porn doesn’t just reflect preexisting sexual tastes” given the ways that
companies use algorithms).

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 91

deception in some contexts only some of the time; in other instances,

deception is prohibited in theory but allowed in practice. At worst, the
policies allow or even openly encourage deception. Further, the policies
are selectively and inconsistently enforced. Such inadequate attempts at
rooting out obvious instances of deception, combined with the focus on
deepfakes, entrenches shallow fakes by creating the impression that the
remaining content is authentic and therefore trustworthy.
The sheer breadth of these policies conveys conflicting messages
about when deception is prohibited. As a simple example, consider the
following statement from Snapchat’s Terms of Service: “We prohibit
pretending to be someone (or something) that you’re not, or attempting to
deceive people about who you are.”119 That sounds like both a reasonable
effort by the platform to root out deception and also a description of the
majority of Snapchat posts. Pretending to be someone or something else is
among the most common things that people do online.120 Instagram
provides yet another illustration. Its terms of use, which all users agree to
by signing onto the platform, include: “You can’t do anything unlawful,
misleading, or fraudulent or for an illegal or unauthorized purpose.”121
Reading this, one might think that posting a photograph of someone posing
on a towel in a sandbox, designed to make it look like a beach, would be
banned. But it is not. Although obviously “misleading,” it is allowed.
Perhaps this is because the platforms’ own terms prohibiting
deception in one context are incompatible with terms that refer to
“misleading” content in other contexts. When Meta (then Facebook)
announced in 2018 that it was ramping up its efforts to reduce the spread
of “false information,” it did not define the term. It then noted that
“[a]lthough false news does not violate our Community Standards, it often
violates our policies in other categories, such as spam, hate speech or fake
accounts, which we remove.”122 In a different set of policies, titled
“Reducing the Spread of False Information on Instagram,” the platform
declared that it may remove or reduce the availability of anything that is
“false information, altered content, or content with missing context.”123
Read together, a piece of false information could be “misleading” under

119. Community Guidelines, SNAP (Aug. 2023), https://bit.ly/43bq8MY.

120. See Priya Singh, This holiday, let’s stop this social media pretending, CNN
(Dec. 23, 2021, 9:39 PM), https://bit.ly/3IDVraU (“Everyone from regular people to
Olympic athletes and Fortune 500 CEOs feel an unending pressure to pretend everything’s
not just OK but is actually great. Certainly, it’s compounded by the pressure to measure
oneself against the distorted images of peers on social media.”).
121. Terms of Use, INSTAGRAM, https://bit.ly/3qaxQZ7 (last visited Sept. 12, 2023).
122. Tessa Lyons, Hard Questions: What’s Facebook’s Strategy for Stopping False
News?, META (May 23, 2018), https://bit.ly/3Mxaep0.
123. Reducing the Spread of False Information on Instagram, INSTAGRAM,
https://bit.ly/3CN1eri (last visited Sept. 12, 2023).

Electronic copy available at: https://ssrn.com/abstract=4606149

92 PENN STATE LAW REVIEW [Vol. 128:1

the community standards policy, but not the fake accounts or hate speech
policy, and vice versa. There is no consistent definition for what counts as
“misleading.”
Then there is the one-off narrowing of the type of deception that is
targeted. Some deception, for example, is banned only when it is
monetized, and only when it is monetized in certain ways. These rules
further confuse the kind of deception that is prohibited. For example,
Facebook openly acknowledges that “[a] lot of the misinformation that
spreads on Facebook is financially motivated.”124 To that end, the firm’s
Instagram Content Monetization Policies do not allow “[m]isinformation,”
which they define as “content that has been rated false by a third-party fact
checker.”125 But, that kind of information is implicitly allowed as long as
it is not “monetized.”126 Moreover, it is presumably the case that this
statement means that the user cannot monetize the misleading content, but
Facebook can because Facebook can still drive engagement (and therefore
advertising sales) to its platform by allowing, and even recommending,
deceptive content by algorithm.127
Unsurprisingly, then, enforcement is inconsistent and unpredictable.
The sweeping scope of these policies means that platforms have a great
deal of discretion in deciding how to implement them. Indeed, platforms
have a record of selectively intervening to manage content they deem
deceptive.128 The mechanisms the platforms rely on for identifying
misinformation are partly at fault. Most of the misinformation discovered
on Instagram depends on reports from users and verification from a “third-
party fact checker,”129 which limits enforcement of the misleading and
deceptive content rules to a tiny range of deception on the platform. The
majority of content will therefore not be flagged as deceptive—and,
ironically, the more deceptive it is, the less likely it is to be flagged, given
that the deception will be well-hidden. Of the content that is flagged,
action will be taken only if and when such content is reviewed by a third
party. Content moderation is thus limited to what can be verified as

124. Lyons, supra note 122.

125. Instagram Content Monetization Policies, INSTAGRAM, https://bit.ly/43XWOd8
(last visited Sept. 12, 2023).
126. Id.
127. See Ryan Mac & Cecilia Kang, Whistle-Blower Says Facebook ‘Chooses Profits
Over Safety’, N.Y. TIMES (Oct. 3, 2021), https://bit.ly/3prct5t (“In reality, Facebook knew
its algorithms and platforms promoted this type of harmful content, and it failed to deploy
internally recommended or lasting countermeasures.”) (quoting Frances Haugen).
128. See Jeff Horwitz, Facebook Says Its Rules Apply to All. Company Documents
Reveal a Secret Elite That’s Exempt, WALL ST. J. (Sept. 13, 2021, 10:21 AM),
https://bit.ly/3Pv7NX9 (describing how the firm’s enforcement of its own rules is uneven,
and nonexistent for a group of select users who had been whitelisted).
129. Instagram Content Monetization Policies, supra note 125.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 93

misleading. If something cannot be verified as misleading, presumably

cheaply and quickly, then it stays up.
Many platforms do not even have a policy on whether a post that is
sponsored should be identified as such. The policies’ silence allows
advertisements to take place without any mention that they are, in fact,
advertisements.130 There are countless examples of users recommending a
particular product, or posting a YouTube video about their experience with
a particular product, without any mention that they are receiving payments
or kickbacks to advertise that product.131 While presented as spontaneous,
independent, and even heartfelt recommendations, the individual was
asked to promote this product in exchange for a financial benefit.132 These
posts, even if truthful in their content, are advertisements.
Moreover, the platforms’ more recent attempts to root out deepfakes
further normalizes the presence of deceptive content. Consider Meta’s
statement declaring its newly stepped-up efforts to eliminate deepfakes.
The head of Meta’s public policy team draws a distinction between
manipulation that is benign and manipulation that intentionally misleads:
“Some of that content is manipulated, often for benign reasons, like
making a video sharper or audio clearer. But there are people who engage
in media manipulation in order to mislead.”133 Intent aside, however, it is
all manipulation. And, of course, much of what the platform calls
“benign”—like the use of filters and editing, tools the platform encourages
users to use—is explicitly designed to mislead, regardless of specific user
preference.134
The policy solutions being developed to address deepfakes depend
on preserving the distinction between types of deception. For example, the
technology firm Adobe is working on a project, the “Content Authenticity

130. This problem pervades the wellness industry, the success of which has been
buoyed by online platforms. See discussion infra Section V.C.1. Goop, a company founded
by Gwyneth Paltrow, broke off its relationship with Condé Nast given the latter’s
requirement to separate ad content from informative content. See Taffy Brodesser-Akner,
How Goop’s Haters Made Gwyneth Paltrow’s Company Worth $250 Million, N.Y. TIMES
MAGAZINE (July 25, 2018), https://bit.ly/46psZ70 (explaining that “they weren’t allowed
to use the magazine as part of their ‘contextual commerce’ strategy” even though Goop
“wanted to be able to sell Goop products (in addition to other products, just as they do on
their site” and treat the “magazine customer [as] also a regular customer” of Goop).
131. See Roberts, supra note 21, at 120.
132. Of course, an influencer might be invited to promote a product they already used
and loved; but failing to disclose the payment for promoting a product is still deceptive.
133. Bickert, supra note 48.
134. The academic conversation around deepfakes similarly asserts that shallow
fakes are harmless and widespread. For example, Chesney and Citron define the problem
posed by deepfakes by distinguishing them from the general deception that takes place
online all the time: “[i]nnocuous doctoring of images—such as tweaks to lighting or the
application of a filter to improve image quality—is ubiquitous.” Chesney & Citron, supra
note 2, at 1759.

Electronic copy available at: https://ssrn.com/abstract=4606149

94 PENN STATE LAW REVIEW [Vol. 128:1

Initiative,” to allow users to see at a technical level whether a piece of

content has been digitally manipulated.135 While this might seem like an
unqualified good, the tool risks giving viewers false confidence that the
remaining material they are viewing has not been manipulated in any way,
only because it has not been digitally manipulated. Yet more analog tools,
like mislabeling a video, cropping the frame, and taking a short video clip
out of context, can cause just as much harm as a digitally manipulated clip.
In this way, rooting out deepfakes reifies the idea of an “authentic” media
when, in fact, all media is subject to editing, interpretation, and
contestation.
Ultimately, these policies all take place in the context of the platforms
authorizing deception. At a basic level, these platforms allow, and
encourage, users to alter images they post. That initial invitation—offered
by the same companies that have promulgated policies against misleading
content—is baked into the fabric of the platform, untouched by any of the
policies that purport to root out deception.
The final picture that emerges is the following confused jumble of
rules: misleading content is banned and discouraged, but only sometimes
and only in some contexts; different rules apply if money is involved; and
this is all subject to the caveat that it becomes a problem only if someone
notices—and even if someone notices, the platform might not do anything
about the deception and, ultimately, allow it. The result of this morass of
policies is an information landscape that is deeply confusing and rife with
fakery. In effect, the policies aimed at eradicating deception are
problematic in a more fundamental way: they create the illusion that by
accounting for and addressing the deception that takes place online, what
remains is authentic and, for the most part, real. In this digital world, it can
be nearly impossible to tell truth from deception.

IV. THE PROBLEM WITH SHALLOW FAKES

We have thus far described the prevalence of shallow fakes and
identified how platforms create a market for engaging in online fakery.
Now we turn normative. Should we be worried about shallow fakes? What
is so problematic about self-enhancement or deception about the self?
Deception is fundamental to human interaction. We all lie a little.
Research shows that we lie every day in our social interactions, with some
estimates putting it at two lies per day.136 More fundamentally, it is

135. See Eric Abent, Adobe expands Content Authenticity Initiative tools to fight
misinformation, SLASHGEAR (Oct. 26, 2021, 8:12 AM), https://bit.ly/46eL7jE; see also You
decide what content to trust, VERIFY, https://bit.ly/3r0ouzm (last visited Sept. 12, 2023).
136. See Bella M. DePaulo et al., Lying in Everyday Life, 70 J. PERSONALITY AND
SOC. PSYCH. 979, 991 (1996) (“Participants in the community study, on the average, told a
lie every day; participants in the college student study told two.”).

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 95

understandable, even expected, that we would want to present the best

versions of ourselves to our peers.137 And, there might be real value in
protecting the decision to assume a virtual identity as an important aspect
of self-discovery or self-control.138
We do not dispute the existence, or at times even the necessity, of
certain kinds of deception. Instead, we argue that the ecosystem in which
people have incentives to fake many aspects of their digital selves has
downsides worth taking seriously. In this Part, we outline three categories
of harm. First, shallow fakes are deeply gendered, encouraging and
propagating traditional gender roles and normative body types and
exacerbating body issues, especially for young women and those who are
gender-nonconforming. Second, many of the filters that propagate today’s
digital deception enable both racial appropriation and exclusion. The fact
that platforms push users toward conformity is exactly the opposite of
what we would expect to see in a medium that is purportedly designed for
self-expression, like social media. Third, and finally, we explore some of
the broader societal harms from living in a digital ecosystem marked by
the kind of casual deception that takes place every day online.
One might think that if shallow fakes were as problematic as we
argue, they never would have become so commonplace. But this logic is
inverted. It is precisely because shallow fakes are so commonplace that
they are problematic. The harms we describe are harms produced by the
ubiquity of shallow fakes. We do not take issue with any single instance
of shallow fakery but instead call attention to the aggregate effect of living
in a world awash in shallow fakes.
Although we are interested in an aggregated problem, that does not
mean that the harms are only felt in the aggregate. To the contrary, living
in the world of shallow fakes can and does cause acute harms to
individuals. This very dynamic—that the harm is felt at the individual
level but is not inflicted at the individual level—is what makes shallow
fakery difficult to remedy. One cannot pinpoint a single act of deception
as the cause of someone’s body dysmorphia or feelings of exclusion or any
other consequent harm. Not only is this dynamic an obstacle to addressing

137. See GOFFMAN, supra note 45, at 35 (discussing “the tendency for performers to
offer their observers an impression that is idealized in several different ways”).
138. See, e.g., Naramol (Jaja) Pipoppinyo, Queer Identity Online: The Importance of
TikTok and Other Media Platforms, MEDIUM (Dec. 1, 2020), https://bit.ly/42XhjoW
(describing “Gay TikTok” and its repudiation of “the mainstream, allowing [users] to
embrace what it truly means to be queer”). But see Daniel Kershaw, LGBTQ and Online
Identities, MEDIUM (Dec. 19, 2013), https://bit.ly/3NMan9N (noting the ways that gender
play arises in online communities but also finding that “the community sometimes finds
this form of gender [play] deceptive” and having a “fluid identity” can become overly
idealistic).

Electronic copy available at: https://ssrn.com/abstract=4606149

96 PENN STATE LAW REVIEW [Vol. 128:1

the problem, but it is also a reason for why shallow fakes are so persistent
and problematic.

A. Gendered Harms
There is a distinctly gendered nature to the harms produced by
everyday deception online. When Frances Haugen, the Facebook
whistleblower, testified before Congress in October of 2021, she revealed
that the firm doggedly pursued teenage users, particularly girls, despite the
fact that the firm’s own research suggested a range of harms for those
users.139 As the Wall Street Journal explained in describing the firm’s
internal research, the firm repeatedly found “that Instagram is harmful for
a sizable percentage of [young users], most notably teenage girls.”140
But the range of gendered harms are broader and go much deeper.
Shallow fakes are harmful to all marginalized genders. The political
economy of seeking clicks, engagement, and an online audience reinforces
traditional gender norms and excludes those who do not fit into the narrow
binary presented. This is a complicated critique to undertake given that
female influencers have gained particular traction in these online spaces,
often by capitalizing on traditionally feminine roles, and have succeeded
in monetizing activities that are otherwise excluded from the market.141
The harms we identify are not the commercialization of previously
“intimate” spaces. Rather, the harms follow from how these spaces get

139. See Protecting Kids Online; Testimony From A Facebook Whistleblower:

Hearing Before the Subcomm. on Consumer Protection, Product Safety, and Data Security
of the S. Comm. on Commerce, Science, and Transportation, 117 Cong. 9 (2021)
(Statement of Frances Haugen) available at: https://bit.ly/3pelE9D [hereinafter Testimony
From A Facebook Whistleblower]. Some have argued that even the attention paid to
Frances Haugen as a whistleblower has a gendered and racist history. See Maria Farrell,
Two Facebook whistleblowers leaned in, but only one became a media star, THE
CONVERSATIONALIST, https://bit.ly/3r3Ycwb (last visited June 25, 2023) (arguing that only
one whistleblower “was given the role of princess, commanding attention and praise,” but
in so doing reinforcing the accepted hierarchy of the importance of politics as compared to
the relative unimportance of appearance, noting “Instagram’s possible effects on their
[Americans who work for the Wall Street Journal] adolescent daughters’ self-image”).
Before Frances Haugen there was Sophie Zhang, the daughter of Chinese immigrants. In
deciding to become a whistleblower, she made the decision to hide her trans identity. Her
8,000 word memo focused more on Facebook’s lack of attention to the political
manipulation taking place online. See Karen Hao, She risked everything to expose
Facebook. Now she’s telling her story., MIT TECH. REV. (July 29, 2021),
https://bit.ly/44f58EY.
140. See Wells et al., supra note 21.
141. See, e.g., Noah D. Zatz, Working at the Boundaries of Markets: Prison Labor
and the Economic Dimension of Employment Relationships, 61 VAND. L. REV. 857, 916
(2008) (“Feminist scholarship has pioneered broader accounts of where economic activity,
and particularly productive work, occurs. This research focuses principally on activities
associated with the family—sexuality, reproduction, housework, and caregiving.”).

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 97

constructed—in ways that reinforce stereotypically male and female roles

and exclude those who do not fit into traditional gender scripts.
We begin this section by identifying some acute and immediate
harms. We end by stepping back and taking stock of the different ways the
platforms entrench normative performances of gender.

1. Body Dysmorphia
People who spend time on social media sites regularly have higher
rates of body dysmorphia, which is defined as “a mental health condition
in which you can’t stop thinking about one or more perceived defects or
flaws in your appearance.”142 As one researcher explained, “Smartphones,
together with the cosmetics industry, are producing significant shifts in
young women’s visual literacies of the body—particularly the face—such
that they quite literally see themselves and others differently from previous
generations.”143 Facebook’s research notes that this is especially true for
the kinds of filtered images that are shared on Instagram, where it found
that “[s]haring or viewing filtered selfies in stories made people feel
worse.”144 These findings are not a niche concern, as Facebook reveals that
it makes “body image issues worse for one in three teen girls.”145
Contributing to the problem is simply the physical reality of taking a
selfie. That is, “the angle and close distance at which selfies are taken may
distort facial features and lead to dissatisfaction.”146 But it is made worse
by online tools that allow people to manipulate their image with endless
tweaks and filters. For example, one study found that adolescent girls who
spent more time manipulating their photos reported higher levels of body
dysmorphia than those who spent less time doing so.147 This is merely a
correlation; it is possible that the causal story is that girls with higher rates
of body dysmorphia are likely to spend more time filtering their photos.148

142. MAYO CLINIC, Body dysmorphic disorder, https://bit.ly/3N4s83T (last visited

Sept. 12, 2022).
143. Gill, supra note 26, at 35.
144. Wells et al., supra note 21.
145. Id.
146. Susruthi Rajanala et al., Selfies-Living In The Era of Filtered Photographs, 20
JAMA FACIAL PLASTIC SURGERY 443 (2018); see also Brittany Ward et al., Nasal
Distortion in Short-Distance Photographs: The Selfie Effect, JAMA FACIAL PLASTIC
SURGERY (Mar. 1, 2018), https://bit.ly/3rHHp2p.
147. See Sian A McLean et al., Photoshopping the Selfie: Self Photo Editing and
Photo Investment are Associated with Body Dissatisfaction in Adolescent Girls, 48 INT’L
J. EATING DISORDERS 1132 (Dec. 2015).
148. Girls generally tend to be more invested in their social media use than boys. See
Jacqueline Nesi et al., Emotional Responses to Social Media Experiences Among
Adolescents: Longitudinal Associations with Depressive Symptoms, J. CLIN. CHILD &
ADOLESCENT PSYCH. 12 (2021) (“Girls reported more frequent positive and negative
emotional responses to social media compared to boys.”). Of course, online spaces may
also provide a sense of community for these same teens, see Victoria Rideout et al., Digital

Electronic copy available at: https://ssrn.com/abstract=4606149

98 PENN STATE LAW REVIEW [Vol. 128:1

That would still be problematic, though, because the filtering tools seem
to create a feedback loop for a slice of the population that is already
susceptible to body dysmorphia.149
There is enough additional evidence to create a credible claim that
the very existence of the filtering applications contributes to the incidence
of dissatisfaction with one’s body. Another study found that 32% of
teenage girls said that when they felt bad about their bodies, Instagram
made them feel worse.150 In the words of one study participant, the feeling
of seeing highly edited images of other women on Instagram is “like stick
thin women with the most amazing butt and the most amazing long hair,
and I’m just like, this isn’t me, and why am I constantly seeing this? And
it does make you feel abnormal, sometimes, and you are normal.”151
Another said, “[W]hen it comes to my skin, I know in my head that is
normal. But when you see the content, it’s like, it does make you feel
almost abnormal because it’s showing you that it shouldn’t be that way.”152
The prevalence of shallow fakery redefines what is considered normal.

2. Depression and Anxiety

Young people, especially young girls, are under extraordinary
pressure to conform to what they see on social media. The result is an
epidemic of mental health problems. One Facebook study shows that
13.5% of teen girls reported suicidal thoughts increased since joining
Instagram.153 As one internal report conducted by Facebook noted,
“[t]eens blame Instagram for increases in the rate of anxiety and
depression.”154 This is, of course, related to body image and norm-
conforming pressures described above. As one teen who was interviewed
commented, “I see all these perfect bodies in bikinis and it makes me feel
really low.”155

3. Pressure to Sexualize
There is intense pressure, even for very young users, to present
themselves as sexual beings. A majority of women surveyed about social
media representations mentioned sexualization without being prompted.156

Health Practices, Social Media Use, and Mental Well-Being Among Teens and Young
Adults in the U.S., PROVIDENCE ST. JOSEPH HEALTH DIGIT. COMMONS 15 (2018), but this
does not minimize the harms caused by the shallow fakes found online.
149. See generally Gill, supra note 26.
150. See Wells et al., supra note 21.
151. Gill, supra note 26, at 19.
152. Id.
153. See Wells et al., supra note 21.
154. Id.
155. Gill, supra note 26, at 19.
156. See id., at 42.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 99

Online photographs are being edited not just to beautify but to sexualize.
It is, in the words of one study, “a visually centered social media that
involves the presence of sexualized imagery,” which, in turn, has a
negative impact on mental health.157 Seventy-five percent of all young
women in a recent survey said that they feel pressure to receive “likes” on
their social media posts.158 Another study found that “sexualized photos
garnered more likes on Instagram,” suggesting that teens feel pressure to
post sexualized versions of themselves.159 This leads to lower feelings of
self-esteem and body image.160 Such findings are widespread and are
consistent with longstanding concerns about sexualization in traditional
media and self-esteem problems, especially in younger people.161
To be clear, the harm we identify is the pressure to conform to the
images presented online, a pressure which functions as a one-way ratchet.
Some individuals might feel empowered to present sexualized images of
themselves and therefore not experience it as a “harm” at all.162 Our
concern is not with these users but with the baseline the platforms create.
All users are pushed to participate in this dynamic, which makes it less of
a choice and more like the price of admission.163 These platforms

157. Francesca Guizzo et al., Instagram Sexualization: When Post Make You Feel
Dissatisfied and Wanting to Change Your Body, 39 BODY IMAGE 62, 62 (Dec. 2021).
158. Gill, supra note 26, at 24.
159. Laura Ramsey & Amber L. Horan, Picture This: Women’s Self-Sexualization in
Photos on Social Media, 133 PERSONALITY AND INDIVIDUAL DIFFERENCES 85, 85 (2018);
see also Kun Yan et al., A Sexy Post a Day Brings the “Likes” Your Way: A Content
Analytic Investigation of Sexualization in Fraternity Instagram Posts, 26 SEXUALITY &
CULTURE 685, 685 (2022) (finding a positive association between the degree of
sexualization in a post and the traffic and likes it received).
160. See id.
161. See Marika Skowronski et al., Predicting Adolescents’ Self-Objectification from
Sexualized Video Game and Instagram Use: A Longitudinal Study, 84 SEX ROLES 584, 585
(2021) (reporting the results of a longitudinal study involving 660 German adolescents and
concluding that “sexualization in video games and on Instagram can play an important role
in increasing body image concerns among adolescents”); see also Thomas Plieger et al.,
The Association Between Sexism, Self-Sexualization, and the Evaluation of Sexy Photos on
Instagram, 12 FRONTIERS IN PSYCH. 1, 1 (Aug. 2021) (reporting results of a survey of 916
participants and finding that “there were substantial correlations between appropriateness
and attractiveness evaluations of the presented photos and the self-sexualizing posting
behavior and enjoyment of sexualization of female users”).
162. See, e.g., Emily Ratajkowski, Emily Ratajkowski Explores What It Means to Be
Hyper Feminine, HARPER’S BAZAAR (Aug. 8, 2019), https://bit.ly/3WEQMLw.
Ratajkowski explains that:
Despite the countless experiences I’ve had in which I was made to feel extremely
ashamed and, at times, even gross for playing with sexiness, it felt good to play
with my feminine side then, and it still does now. I like feeling sexy in the way
that makes me, personally, feel sexy. Period.
Id.
163. One response to this harm, as to all the harms we identify, is to remove oneself
from social media entirely. While that might be a solution for some individuals, that is not
directly responsive to the harms caused by social media platforms—those harms are what

Electronic copy available at: https://ssrn.com/abstract=4606149

100 PENN STATE LAW REVIEW [Vol. 128:1

perpetuate a distinctly modern feminist double-bind: women can become

powerful players in a digital ecosystem, but the way they can achieve
success is the timeworn one of relying on, and revealing, their body.164
Images that are sexualized are, of course, distinct from images that
involve sex, which we do not identify as a harm.165 Quite the opposite, in
fact—we hope people find whatever satisfaction it is they are seeking
online, sexual or otherwise.166

4. “Real-Life Filtered Look”

Cosmetic surgery rates have dramatically increased alongside the rise
of social media, and many plastic surgeons now specialize in “Instagram
Face.”167 Customers specifically seek to replicate how they appear on
social media; they want a “real-life filtered look.”168 A 2021 survey by the
American Academy of Facial Plastic and Reconstructive Surgery found
that 77% of facial surgeons had treated customers in the previous year
whose decision to undergo surgery was motivated by a desire to look better

our piece seeks to expose, with the aim of beginning a conversation on how to remedy
them.
164. See, e.g., RATAJKOWSKI, supra note 78, at 5. Ratajkowski writes:
In many ways, I have been undeniably rewarded by capitalizing on my sexuality.
I became internationally recognizable, amassed an audience of millions, and
have made more money through endorsements and fashion campaigns than my
parents (an English professor and a painting teacher) ever dreamed of earning in
their lifetimes. I built a platform by sharing images of myself and my body
online, making my body and subsequently my name recognizable, which, at least
in part, gave me the ability to publish this book. But in other, less overt ways,
I’ve felt objectified and limited by my position in the world as a so-called sex
symbol . . . Whatever influence and status I’ve gained were only granted to me
because I appealed to men.
Id.
165. Others, however, do. For a recent review of different feminist takes—negative
and positive—on porn and its meteoric rise online, see SRINIVASAN, supra note 118, at 33–
71.
166. See, e.g., Emily Witt, A Hookup App for the Emotionally Mature, THE NEW
YORKER (July 11, 2022), https://bit.ly/3C2Y8Pt (describing the author’s experience on
Feeld, a dating app that “is popular with nonbinary and trans people, married couples trying
to spice up their sex lives, hard-core B.D.S.M. enthusiasts, and ‘digisexuals,’ who prefer
their erotic contact with others mediated by a screen”).
167. See Tolentino, supra note 58. Tolentino describes the “Instagram Face”:
It’s a young face, of course, with poreless skin and plump, high cheekbones. It
has catlike eyes and long, cartoonish lashes; it has a small, neat nose and full,
lush lips . . . . The face is distinctly white but ambiguously ethnic—it suggests a
National Geographic composite illustrating what Americans will look like in
2050 . . . .
Id.
168. American Academy of Facial Plastic Surgery, AAFPRS Announces Annual
Survey Results: Demand for Facial Plastic Surgery Skyrockets As Pandemic Drags On,
PR NEWSWIRE (Feb. 10, 2022), https://bit.ly/4518R9L [hereinafter AAFPRS Survey].

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 101

in selfies.169 The tools offered by social media platforms present

unrealistic images of bodies, often unattainable without a filter or
surgery.170 Yet, as beauty filters make people feel bad about their bodies,
they also provide a template for how to fix them. These tools are having a
profound effect on the cosmetic surgery market.
The autonomy and agency that are often lauded in discussions of
plastic surgery171 are less compelling when the ideals of beauty are being
set by social media companies in uniform ways.172 The debate over
whether cosmetic surgery is ever the product of true agency or
unadulterated choice runs deep.173 Regardless of which side one falls on,
evidence shows that cosmetic surgery is generally not undertaken to be
beautiful, but to fit in—women engage in it as a way “to become ordinary,
normal.”174 What is considered the norm, then, matters. And norm-setting
is not value-neutral: “[n]ot every body will do; nor are all differences the
same in Western culture.”175 Today’s digital platforms are creating and

169. See id.

170. Low self-esteem is a major predictor in whether a woman will undergo plastic
surgery. See Bonell et al., The Cosmetic Surgery Paradox: Toward a Contemporary
Understanding of Cosmetic Surgery Popularisation and Attitudes, 38 BODY IMAGE 230,
233 (2021). Studies show, however, that “[d]espite the fact that low self-esteem directly
predicts women’s likelihood to undergo cosmetic surgery, research has indicated that
longitudinal self-esteem improvements post-surgery are either small or non-significant.”
Id. at 234 (citations omitted).
171. See Melissa Febos, The Feminist Case for Breast Reduction, N.Y. TIMES
MAGAZINE (May 10, 2022), https://bit.ly/3WDU5CJ (using breast reduction surgery as a
stand-in for cosmetic surgery writ large, proposing that electing to undergo surgery is a
feminist act, allowing her to “no longer [feel] so defined by [her] corporeal form”).
172. The “Snapchat selfie” is something that plastic surgeons now specialize in. See
Anna Davies, People are getting surgery to look like their Snapchat selfies, BBC THREE
(Apr. 19, 2018), https://bit.ly/43tkYLM.
173. See Kathy Davis, Revisiting Feminist Debates on Cosmetic Surgery: Some
Reflections on Suffering, Agency, and Embodied Difference, COSMETIC SURGERY: A
FEMINIST PRIMER 38–39 (Cressida Heyes & Meredith Jones eds., 2016) (describing
criticism of her work by feminist philosopher Susan Bordo, in particular the reliance on
concepts of “agency” and “choice” and “freedom” in describing the women who undertake
elective cosmetic surgery).
174. Id. at 36. The very existence of “[c]osmetic surgery is predicated upon
definitions of physical normality,” and these “categories of ‘normality’ and ‘abnormality’
are drawn upon in both medical discourse on cosmetic surgery . . . and in individuals’
accounts of their surgical experiences.” Id. at 43.
175. Id. at 45. Davis criticizes the discourse that presents “cosmetic surgery . . . as
neutral technology, ideally suited in altering the body in accordance with an individual’s
personal preferences” because, among other problems, “[i]t discounts the universality of
white, Western norms of appearance, which shape individuals’ perceptions of what they
consider to be desirable appearance as well as the kinds of interventions that are deemed
acceptable.” Id. at 44–45. We are mostly discussing these questions from an American
point of view, and thus Western culture matters. Beauty ideals, however, of course differ
across the world and it is often the case that “local cultural dynamics trump[] outside
influences when it comes to health and beauty norms.” HU, supra note 90, at 152. Elise Hu
has considered the popularity of plastic surgery in South Korea specifically and has

Electronic copy available at: https://ssrn.com/abstract=4606149

102 PENN STATE LAW REVIEW [Vol. 128:1

perpetuating these norms en masse, as evidenced by the ubiquity of the

requests for a particular kind of filtered look.
The norm-creation these platforms are engaged in extends to the
procedure of cosmetic surgery itself. TikTok is replete with videos
demonstrating before and after pictures of procedures like nose jobs.176
The black eyes, broken nose, and nose bleeds that follow these procedures
are included in 15-second clips that culminate with a celebratory “‘nose
job check’ sound.”177 Currently, the TikTok hashtag #nosejob has over 1.6
billion views.178
The problem we identify is not the desire for cosmetic surgery, nor
even the cosmetic surgery itself. We also roundly reject the notion that
these desires or discussions are “frivolous”179—indeed, our whole concern
is with a swath of behavior that could be, and is, easily dismissed as such.
Instead, our critique is of the conditions that have led to a pronounced
increase in these elective procedures along with the specific nature of the
requests,180 namely, to mimic a highly stylized image offered by the digital
platforms that is unattainable without filters or, in the physical world,

explained how definitions of beauty are informed internally, making it “colonialist to

emphasize race as a determining factor in the beauty decisions of Asians.” Id. at 152–53.
She does note that current ideals of being “cute” in South Korea are informed by “anime
characters or the enhancements from digital filters on . . . social apps.” Id. at 158.
176. On TikTok, “[p]lastic surgery videos are endemic . . . . The hashtag
#plasticsurgery has over 3.8 billion views” while “[t]he hashtag #nosejobcheck, which
mainly consists of videos showcasing before-and-after clips of nasal surgery, has
accumulated over one billion views on the platform.” Joshua Zitser, Insider created a tiktok
account and set the age at 14 to test how long before a plastic surgeon’s promotional video
appeared. It only took 8 minutes, INSIDER (Jan. 10, 2021, 10:41 AM),
https://bit.ly/3C5lLa2.
177. These videos are visible to young users, and TikTok’s community guidelines do
not prevent them from being shown as long as they are “not ‘graphic’ and don’t contain
‘gore.’” Id. Charli D’Amelio, one of TikTok’s most viewed users, documented her nose
job when she was 16. The caption reads: “two broken noses lots of nose bleeds and
breathing problems for 11 months! I can finally breathe like normal and get back to
dancing.” She then tags her surgeon, Dr. Kanodia, @drkanodia90210. Charli D’Amelio
(@charlidamelio), TIKTOK (Jul. 28, 2020), https://bit.ly/3N4cdT1.
178. See Zitser, supra note 176.
179. Davis, supra note 173, at 87 (describing Iris Young’s criticism that “much of
the cosmetic surgery women undergo must be ‘frivolous and unnecessary, like diamonds
or furs’”) (internal citation omitted).
180. The desire to change one’s body was exacerbated by the COVID-19 pandemic
because people were spending time online using applications that did not have filters,
which is believed to have caused a considerable rise in elective cosmetic surgeries.
AAFPRS Survey, supra note 168 (“Unlike selfies and video editing apps like TikTok and
Reels on Instagram, the video conferencing used for school, work and ZOOMing with
family and friends does not allow for filtering capabilities, making it a particularly easy
lens for self-scrutiny.”). As the President of the leading professional association for facial
plastic surgeons noted, “[r]eal time video cannot be FaceTuned or photoshopped to smooth
out a bump on the nose, crow’s feet or a sagging neck.” Id.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 103

cosmetic surgery.181 The harm we are surfacing is of social media platform

algorithms driving a very particular, very limited standard of beauty,
which is causing people to see themselves differently182 and to change their
bodies and their faces as a result.183

5. Reinforcing Traditional Gender Roles

Online fakery in general reinforces rather than disrupts traditional
gender norms. A survey of teenage girls revealed “a very high degree of
consensus about how women ‘should’ look: ‘no body hair, white teeth,
curvy but slim, good skin.’”184 The same report found that young women
feel pressure to have a certain look and that such pressures are especially
hard for anyone who is nonbinary. Nearly everyone is chasing a look that
is “too perfect, too sexualised, too white, too heteronormative, too middle
class and do[es] not represent the lives of disabled and/or nonbinary
people.”185 Indeed, not just the selves but the lives that users present online
are stereotypically heteronormative. One lesbian survey respondent noted
that “[w]omen are so often presented in relation to men (as a wife,
girlfriend, sex object or in a heterosexual family), [that] as a lesbian
woman I don’t relate to these images.”186
Much of the fakery that people engage in online is something people
do to themselves.187 But the filter compounds this process by changing
appearances in ways that reinforce pre-existing gender norms. For
example, the beautification filters that have become increasingly common

181. Lele Pons, who is ranked as the number-one, highest-paid non-celebrity on

Instagram, is very open about her plastic surgery and posts before and after pictures of
herself on her site. See Lele Pons (@lelepons), INSTAGRAM (March 5, 2022),
https://bit.ly/3INe38v.
182. See Tate Ryan-Mosley, Beauty filters are changing the way young girls see
themselves, MIT TECH. REV. (Apr. 2, 2021), https://bit.ly/45ARxcD.
183. See HU, supra note 90, at 135 (“The technological gaze keeps feeding us an ever-
evolving, ever-narrowing beauty ideal. With enough money, we use ever-improving
artificial implants, injections, or surgery to look however we want, fueling a filter-to-filler
pipeline.”).
184. Gill, supra note 26, at 26.
185. Id. at 8. Another person described the conformity as follows: “A very fit, slim
body has been normalised as what a woman should look like. Men also face the
normalisation of a fit physique.” Id. at 29; see also Por que queremos ser iguales?, LA
VANGUARDIA (June 29, 2015), https://bit.ly/3qZhkeS (discussing the criticisms of the Miss
Spain 2022 pageant for picking three finalists who all looked very similar and addressing
the paradox of living in a society that is at once demanding more equality and greater
diversity but also experiencing a push towards a singular definition of beauty represented
on social media networks).
186. Gill, supra note 26, at 45.
187. Even if they describe disliking it. As one influencer notes on her Instagram
stories: “I literally just always use a Paris filter. Like it’s just habit at this point.” In the
next slide, she writes “Here with Paris filter. These filters fuck you up.” See Sara Foster
(@sarafoster), INSTAGRAM (July 6, 2022).

Electronic copy available at: https://ssrn.com/abstract=4606149

104 PENN STATE LAW REVIEW [Vol. 128:1

on apps are designed to identify a female face and make it more slender,
while they make a face identified as male more broad.188 These same filters
try to make women’s bodies thinner and men’s more muscular.
There are counter examples, to be sure. A recent report described how
Instagram can be a “lifeline” for nonbinary people “struggling to find
others just like them.”189 The same report noted, however, that the space
is filled with risks for nonbinary people, who post pictures of themselves
only to be criticized and bullied for not conforming to gender
stereotypes.190 One five-year study of queer youth of color found that
social media platforms like Facebook were dangerously heteronormative
and created spaces of “default publicness,” resulting in offline harms like
being disowned from one’s family.191
The marketplace for clicks and sponsored content pushes individuals
into more traditional roles if they want to succeed. Content online exists
not only in a market for likes but in an actual market where platforms are
seeking payment, and individual users are seeking endorsements. Many
brands use algorithms to help identify which individuals they should
contact to sponsor their products online. Studies of these algorithms reveal
how certain terms and activities are categorized in ways that marginalize
already-marginalized users. For example, Peg, a UK-based tool that
enables brands to identify possible marketers, codes the use of the term
“queer” as profanity, making individuals who employ that term less
attractive to brand partnerships and thereby excluding them from the
market and from the ability to influence.192 Similarly, YouTube creators
who identify as LGBTQ+ have a difficult time monetizing their content
given that such content is often age-restricted and marked as “not being
‘advertiser and family friendly.’”193

188. See Sage Anderson, Snapchat’s ‘gender-swap’ filter exposes the internet’s
casual transphobia, MASHABLE (May 16, 2019), http://bitly.ws/F59W.
189. Wortham, supra note 30.
190. See id. (interviewing one user who “doesn’t identify as any gender” who spoke
to the possibilities that social media creates of rendering gender nonconforming lives
visible and of challenging “mainstream perceptions of gender,” while also noting the
violence such individuals face, explaining, “I still receive daily hate mail from people of
all genders telling me that my body hair is ugly & that I need to shave to be more ‘real’ &
‘beautiful’”) (internal quotation marks omitted).
191. See Alexander Cho, Default Publicness: Queer Youth of Color, Social Media,
and Being Outed By The Machine, 1 NEW MEDIA & SOC. 3183, 3184, 3187 (2017)
(contrasting Facebook to Tumblr, and showing that queer young people preferred Tumblr,
which has less of a default public character).
192. See Sophie Bishop, Influencer Management Tools: Algorithmic Cultures, Brand
Safety, and Bias, 7 SOCIAL MEDIA & SOCIETY 1, 8 (March 30, 2021), http://bitly.ws/F5ar
(explaining “[w]hile queer does have roots as a homophobic slur, it is a term widely used
in activism and LGBTQ+ communities, in addition to within deconstructivist theory to
recognize that sexualities are ‘unstable, fluid, and constructed”) (citation omitted).
193. Zoë Glatt & Sarah Banet-Weiser, Productive Ambivalence, Economies of
Visibility and the Political Potential of Feminist YouTubers, ASSOCIATION OF INTERNET

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 105

Even acts that appear to subvert heteronormative ideals end up

propping them up. Take the example of women who present themselves
as mothers online. “Mommy bloggers” have been largely replaced by
mommy Instagrammers, who, instead of sharing “the ups and downs of
their lives,” now post about “trips to Greece and carefully arranged living
rooms.”194 Instagram is, by its very nature, more image-based than
narrative-based and feeds into a particular ecosystem that relies on “likes
and hearts.”195 Minor insurrections against the demands of motherhood
appear as hashtag #momfails or as purported displays of a less-than-
perfect life. Most of these “failures,” however, involve leaning into the
role of perfect mother and continue to portray largely unattainable goals.
Research into situations in which moms cast themselves as a “hot mess”
or describe their homes as chaotic shows that they are neither: a “visual
inspection of these posts revealed clean and well organized homes[] and
women who were well kempt (e.g., styled and coloured hair, wearing
cosmetics, wearing clean, nice clothes, accessorized outfits with jewellery,
etc.).”196
These admissions of imperfection are often paired with product
placement. Consider, for instance, a picture of a young mother’s ten-day
postpartum belly, seemingly awash in one of Instagram’s filters,
accompanied by the caption “[s]o proud of my body and the life it has
created.”197 The image is at once intended to be relatable—the belly is still
visible after giving birth—but also aspirational in that there are no other
signs of the baby on the body, which is tan and blonde, positioned against
a bathroom that is spotless and framed as place for relaxation (candles and
a loofa are visible in the background).198 The caption further expresses
gentle self-acceptance: “giving myself all the love, care, hot showers, and
R&R it needs.”199 Nowhere does it state that the post is sponsored. Yet the

RESEARCHERS (AOIR) VIRTUAL CONFERENCE (2020); see also Ari Ezra Waldman,
Disorderly Content, 97 WASH. L. REV. 907, 910–11 (2022) (arguing that content
moderation maintains and reifies “social media as ‘straight spaces’ that are hostile to queer,
nonnormative expression”).
194. “The death of the mom blog has something to do with shifts in how people
consume and create on the Internet. Blogging on the whole has fizzled as audiences and
writers have moved to other platforms.” Sarah Pulliam Bailey, What ever happened to the
mommy blog?, CHICAGO TRIBUNE (Jan. 29, 2018), https://bit.ly/45duS5x. Those platforms
include Instagram, which “is built for beauty (its filters make your life look better), not for
rawness.” Id.
195. Id. (“The shift to shorter posts and an emphasis on likes and hearts has changed
the tone and content of what moms find online: more pictures, fewer words, less grit.”).
196. Kelly D. Harding et al., #sendwine: An Analysis of Motherhood, Alcohol Use
and #winemom Culture on Instagram, 15 SUBSTANCE ABUSE: RESEARCH AND TREATMENT
1, 4–5 (2021).
197. Pamela Tick (@pamelatick), INSTAGRAM (Dec. 17, 2021), http://bitly.ws/F5fN.
198. See id.
199. Id.

Electronic copy available at: https://ssrn.com/abstract=4606149

106 PENN STATE LAW REVIEW [Vol. 128:1

image includes a clickable link to Spanx, a company that “specializes in

foundation garments intended to make people appear thinner.”200 The
message of self-love is thus in service to a brand whose purpose is to
minimize the very bulge the mom is outwardly celebrating, making her fit
into the conventional—slim—body image that women, including those
who just gave birth, are pressured to achieve.201
This is no coincidence: promoting self-care goes hand-in-hand with
product placement.202 Motherhood is so well-represented on social media
platforms because women make decisions over household purchases and
companies are eager to reach them as audience members and
consumers.203 In exploring the use of alcohol on Instagram by mothers,
one study found that posts tend to position wine as a means of coping with
overwhelming “domestic and motherly responsibilities.”204 Rather than
question the conditions that lead the allocation of caretaking
responsibilities to fall entirely on “mom,” or address any potential
underlying health or mental issues, the content presents a product, wine,
as a way of “bringing serenity and peace for women.”205 Wine becomes a
“part of routine self-care, and . . . a necessary way to cope with being a
modern mother.”206
While monetizing work within the home is in some ways preferable
to the alternative of keeping “women’s work” gratuitous and outside of a
cognizable market exchange, the kind of work that is recognized is

200. Spanx, WIKIPEDIA, https://bit.ly/46jYBuC.

201. Breastfeeding challenges and feeding complications likewise appear in
conjunction with a recommendation for a specific powdered milk product. See Daphne
Thompson, Hannah Bronfman Shares Her Infant Feeding Journey & What Led Her to
Bobbie, MAMAZINE (Aug. 18, 2021), https://bit.ly/3rystmW (discussing Bronfman’s
“feeding journey” in conjunction with @bobbie, a company “founded by moms”). See also
Melissa Murray, Foreword: The Milkmaid’s Tale, 57 CAL. W.L. REV. 210, 227–28 (2021)
(describing the pressure imposed on mothers to conform and that “[i]n a culture in which
‘breast is best’—and indeed, is so self-evidently correct that alternatives are never even
broached or contemplated—it is easy (or at least easier) to condemn those who depart from
the orthodoxy as deviant” especially “when these individuals are Black women”).
202. When mothers engage in mini-rebellions and challenge the norms of how a
“good mother” acts, their revolutions stall in the register of self-care. In the context of wine
drinking on Instagram, the following findings have been made:
#winemom culture has enabled momtrepreneurs, women who are balancing the
roles of business owner and parent, to create an online persona that can be used
on social media sites to provide one-stop-shops that sell products while allowing
women to partake in an online community of like-minded individuals with the
shared experience of motherhood. The creation of these online communities has
allowed wine companies (amongst other companies) to market to women without
seeming too obvious or obnoxious.
Harding et al., supra note 196, at 2.
203. See id.
204. Id. at 6.
205. Id. at 8.
206. Id.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 107

necessarily marked as “woman’s work” in these online spaces. And the

individuals who are most successful in this specific market are middle-
class, white women who continue to be most legible as mothers.207 Not all
mothers can engage in this market on the same terms, or at all. These posts
participate in constructing an ideal mother and an ideal consumer who is
“white, middle class, [and] cisgender.”208

B. Racialized Harms
As the discussion of gender makes clear, users construct their images
online in ways that implicate race. In this section, we identify further racial
harms that include blackfishing, whitewashing, and other forms of
appropriation and exclusion. To a certain extent, of course, this reflects the
interests of users—but the platforms enable such practices by offering
specific filters that amplify and propagate each of the harms we identify.

1. Blackfishing and Other Forms of Appropriation

Blackfishing is a form of cultural appropriation in which someone
who is not Black acts or appears to be Black in order to acquire some sort
of capital—to appear more attractive or garner attention or acquire
financial gain.209 The concern underlying blackfishing is that someone can
tweak their image in ways that allow them to be Black, without actually
being Black, and in this manner can pick and choose the benefits of being
Black without experiencing any of the drawbacks. As one critic suggests,
“[i]nstead of appreciating Black culture from the sidelines, there’s this
need to own it, to participate in it without wanting the full experience of
Blackness and the systemic discrimination that comes with it.”210 While
concerns over cultural appropriation have existed for centuries,
blackfishing is a phenomenon distinctly tied to social media. Its prevalence
is directly related to the rise of filters and other software that allow people
to change their appearance by “using image editing tools to darken their
complexion or change their facial features to appear more Black.”211 It is

207. See id. at 7 (discussing how “wine mom culture” on social media leads to “a
continued reproduction of white, middle-upper class, neoliberal values”).
208. Id. The study explains: “It is notable that only one image corresponded to a
visible woman of colour, with all other images aligning with predominantly white women
who present themselves as being affluent.” Id.
209. See Stevens, supra note 31, at 1 (defining blackfishing as “a practice in which
cultural and economic agents appropriate Black culture and urban aesthetics in an effort to
capitalize on Black markets”). For a nuanced take on how race can be considered mutable,
see Deepa Das Acevedo, (Im)mutable Race?, 116 NW. U. L. REV. ONLINE 88 (2021).
210. Faith Karimi, What ‘Blackfishing’ means and why people do it, CNN (July 8,
2021, 8:37 AM), https://bit.ly/3MlOfRI.
211. Zawn Villines, What to know about blackfishing, MED. NEWS TODAY (Nov. 9,
2021), https://bit.ly/3OA5d1p.

Electronic copy available at: https://ssrn.com/abstract=4606149

108 PENN STATE LAW REVIEW [Vol. 128:1

also apparently successful: three of the top ten Instagram earners are
routinely accused of blackfishing.212
Examples abound. Emma Hallberg is a Swedish model and
influencer. She has over half a million followers on Instagram and she is
known for her makeup tutorials, which showcase her skin’s bronzed
glow.213 Most people assumed that Emma was Black, but, it turns out, that
she identifies as white. As she told BuzzFeed, “I do not see myself as
anything else than white,” and “I get a deep tan naturally from the sun.”214
Emma was accused, along with other white Instagram models, of
“adopting what some have called digital blackface, altering their
appearance with makeup and using Afrocentric hairstyles.”215 The purpose
in doing so is “to build their personal brand and secure lucrative brand
endorsements”—which, for many users, is the reason for being on social
media.216
Appropriation can take place not only in terms of physical appearance
but also in terms of activity. Addison Rae, a famous TikToker, has
performed and popularized a series of dances authored by Black creators,
without always giving them credit.217 The scale of blackfishing on TikTok
became significant enough that in 2021, Black TikTok creators staged a
strike.218
Blackfishing is, in many ways, only the latest chapter in a very “old
story about white people profiting off of black aesthetics to project a sense

212. The top-earners are Kim Kardashian, Ariana Grande, and Kylie Jenner. See
Donna Tang, How Much Do Instagram Influencers Make, CREDITDONKEY, (Apr. 12,
2022), https://bit.ly/3q4NVPW; Stevens, supra note 209, at 1 (listing Kim Kardashian and
Ariana Grande as being accused of blackfishing); Ryan Schocket, Kylie Jenner is Being
Accused of Blackfishing And the Twitter Reactions Say It All, BUZZFEED (Oct. 23, 2021),
https://bit.ly/45jYVsF.
213. See Tanya Chen, A White Teen Is Denying She Is “Posing” As A Black Woman
On Instagram After Followers Said They Felt Duped, BUZZFEED NEWS (Nov. 13, 2018,
5:05 PM), https://bit.ly/3Wt26dN.
214. Id.
215. Stevens, supra note 31, at 1.
216. Id.
217. Addison Rae received backlash for performing dances on Jimmy Fallon and not
giving credit to the choreographers. She does not claim otherwise and lists the creators on
her YouTube channel. Describing them, she has said, “They’re all so talented and I
definitely don’t do them justice.” Joe Price, Addison Rae Under Fire for Not Crediting
Black TikTok Creators While Performing Challenges on ‘Fallon’ (Update), COMPLEX
(Mar. 29, 2021), https://bit.ly/3pWoqjy.
218. See Sharon Pruitt-Young, Black TikTok Creators are on Strike to Protest a Lack
of Credit For Their Work, NAT. PUB. RADIO (July 1, 2021, 11:00 PM),
https://bit.ly/43cJQr8. TikTok claims it is taking steps to reduce the ability to engage in
cultural appropriation, but Black users say there has been little change. See Vanessa Pappas
& Kudzi Chikumbu, A message to our Black community, TIKTOK (June 1, 2020),
https://bit.ly/45teQoA; Kalhan Rosenblatt, Months after TikTok apologized to Black
creators, many say little has changed, NBC NEWS (Feb. 9, 2021, 5:11 AM),
https://bit.ly/3Ixcmf6.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 109

of edge without feeling any of the associated struggle.”219 But filters

contribute to, and accelerate, this process by making these alterations
easily available. The nature of being online also makes the changes
difficult to identify when the only image one has access to is of a filtered
individual. The market for likes, and the paid sponsorships that permeate
the platforms, facilitate the use of Black culture “to promote the products
they endorse in their posts and to increase their following.”220
The platforms themselves also directly traffic in clearly stereotypical
portrayals of different cultures and ethnicities. Instagram used to offer a
filter called “Choco skin” which allowed users to automatically darken
their skin and hair.221 Other filters were aimed at making users appear more
Asian, with names like “Asian Beauty” and “Geisha.”222 Snapchat has
repeatedly offered filters that allow users to transform their faces into
cartoon versions of Jamaican Rastafarians (the “Bob Marley” filter) and
had one that was “anime-inspired,” in which the user’s face would be
augmented with a rice hat, squinted eyes, and buckteeth.223 Snapchat has
since removed both filters.
Of course, invoking a culture that is not one’s own is not per se
problematic. And there are important expressive benefits to seeing
different ethnicities represented on these digital platforms. But there are
ways to pay homage to another culture without portraying it as one’s own
for financial gain or by reducing it to offensive stereotypes.224 A platform
offering users a Geisha filter that automatically allows them to digitally
dress up as a Geisha is not that.

2. Whitewashing and Exclusion

While digital image enhancing tools allow people to appear darker
than they are, filters also automatically whiten darker skin in photos. For
example, the popular filtering app FaceApp sparked an outcry for its “Hot”

219. Spencer Kornhaber, How Ariana Grande Fell Off the Cultural-Appropriation
Tightrope, ATLANTIC (Jan. 23, 2019), https://bit.ly/43kPG9R.
220. Stevens, supra note 31, at 6.
221. Charlie Duffield, Instagram ‘choco skin’ filter is form of brownface, says
teacher and activist, EVENING STANDARD (Sept. 1, 2020), https://bit.ly/43gV5iR.
222. Sarah Lee, Instagram Filters: ‘Our skin is for life, not for likes’, BBC NEWS
(Oct. 19, 2020), https://bit.ly/3oG4Yav.
223. See Robinson Meyer, The Repeated Racism of Snapchat, ATLANTIC (Aug. 13,
2016), https://bit.ly/3WMsT4P; see also Sam Levin, Snapchat faces backlash over filters
that promotes racist stereotypes of Asians, THE GUARDIAN (Aug. 10, 2016, 2:04 PM),
https://bit.ly/43jmdxp.
224. Rihanna’s decision to wear Chinese couture at the 2015 Met Gala is a good
example; she used her platform to respectfully honor Chinese designs, and she gave credit
to those designers. See Jenni Avins & Quartz, The Dos and Don’ts of Cultural
Appropriation, THE ATLANTIC (Oct. 20, 2015), https://bit.ly/45F0kdE. The setting was also
relevant: it was a costume ball at the Met Museum, and she chose a designer whose work
was on display at the Museum. See id.

Electronic copy available at: https://ssrn.com/abstract=4606149

110 PENN STATE LAW REVIEW [Vol. 128:1

filter which made darker skin tones appear lighter.225 The app’s filter did
not promise to make users whiter, but rather to make them “hotter,” which,
according to the platform, meant having whiter skin.226 The same is true
for Snapchat’s “flower crown” filter, which one would expect to merely
add a flower crown to images but which also whitens one’s skin tone
considerably.227 Similarly, Instagram’s “Attraction” filter—which has
been used in over 143,000 videos—pushes people towards European
standards of beauty.228
Whitewashing and blackfishing are not as paradoxical as might
initially appear. The problem with blackfishing is that it allows individuals
to selectively choose which aspects of Black culture to claim, in a world
that still holds mainstream white beauty as the norm.229 Indeed, the
baseline, “unfiltered” look is to whiten and lighten. The cumulative effect
of these filters is that they regularly exclude people of color.230 For
example, the “Glow” filter on TikTok, which is designed to make a face
look more beautiful, simply does not work on some people of color.231 As
one TikTok creator put it, “my first reaction was like, ‘Oh, great, another
one of those beauty filters that changes our features to make us cater to the
European so-called beauty standards.”232 The Glow filter has been used on
over 3 million TikTok videos.233 As one Myanmarese TikTok user
explained, “You have to be a white woman. You have to have darker skin
almost, but in the, bronze-y, ‘white woman with a tan’ way rather than
like, actually working for people with different skin tones.”234 Even though
the user base is diverse, the tools offered by the platforms are not.235

225. See Zoldan, supra note 32.

226. See id.
227. See Prakash, supra note 32.
228. See Jennimai Nguyen, TikTok beauty filters can be super realistic-unless you’re
a person of color, MASHABLE (Aug. 6, 2021), https://bit.ly/3ML5xrI.
229. See Stevens, supra note 31, at 12 (“The light-skinned, loosely textured hair and
plump lips featured in their content are indicative of normative White beauty standards as
a measure to which the black feminine body should aspire.”).
230. See Nguyen, supra note 228.
231. See id.
232. Id.
233. See id.
234. Id.
235. See id. These effects are embodied in the kinds of cosmetic procedures that are
pursued. As Kathy Davis has observed in her sustained study of cosmetic surgery:
A critique of cosmetic surgery and, more generally, a politics of the body, cannot
be reduced to either gender or race. An exclusive focus on gender would be
inadequate for understanding why the practice of cosmetic surgery has been a
primarily white, western enterprise. By the same token, an exclusive focus on
race or ethnicity could not account for the fact that most operations on ‘Jewish
noses’ or ‘Oriental eyelids’ are performed on women.
Kathy Davis, Surgical Passing: Or Why Michael Jackson’s Nose Makes “Us” Uneasy, 4
FEM. THEORY 73, 85 (2003).

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 111

C. Democratic Harms
We have not meant to imply in our assessments of the harms that
users are naïve and unsuspecting victims. In fact, because everything is
being faked all the time, many of us view and engage in the online sphere
with a certain cynicism. “That can’t be real,” or “they’re just trying to sell
something,” are typical everyday reactions to a medium in which everyone
is searching for clicks and scrolls. This reaction is heightened given that
all of these interactions take place against a background of
commercialization.236 Knowing that “goods”—a product, a lifestyle, a
better version of one’s self—are constantly being peddled on social media
means that people react skeptically to posts. The corollary worry about
widespread deception is that no one trusts anyone about anything.
This skepticism manifests itself in a number of important ways that
threaten a healthy, functioning democratic society. We identify two: the
erosion of expertise and the inability to engage in informed public
discourse. While others have raised these concerns in the context of fake
news, their relationship with shallow fakes has not been considered. We
do that here. In addressing the erosion of expertise, we consider the rise of
the wellness industry—which has truly taken off on social media
platforms—as a case study. In considering the impoverishment of our
ability to engage in productive dialogue, we examine the role that constant,
casual, and unchecked deception plays on the internet.

1. The Erosion of Expertise

One of the appeals of social media is its democratic nature: anyone
can have a voice. Users know that any individual with a smart device and
an internet connection can join in the discourse—there is no test to pass or
requirement to satisfy prior to endorsing a perspective or stating an
opinion.237 It is precisely because everyone can speak that it is harder for
authorities on a particular topic to stand out. The result has meant an
erosion of public trust in experts. The Pew Research Center has shown that

236. See WU, supra note 10, at 5.

237. An exception to the lack of correctives on any information presented, and a turn
towards expertise, happened during the COVID-19 pandemic. See Helping People Stay
Safe and Informed about COVID-19 Vaccines, INSTAGRAM BLOG (Mar. 16, 2021),
https://bit.ly/3qoZjWX (stating Instagram’s practices of removing posts that violate its
COVID-19 and vaccine policies and attaching COVID-19 and vaccine information labels
to posts containing potentially unfounded health claims that direct users to “more credible
information from health experts including the WHO and the CDC”).

Electronic copy available at: https://ssrn.com/abstract=4606149

112 PENN STATE LAW REVIEW [Vol. 128:1

Americans’ trust in experts is declining.238 Other research has shown that

this decline in trust is related specifically to social media use.239
The visual medium we have described is one where people present,
and are presented with, unreal and unrealistic images. Routine exposure to
such images leads many users to feel bad about themselves and to believe
they can make small changes (edits, touch-ups, glow-ups) to their bodies
or their lives to better themselves. In short, the world of shallow fakes
makes self-improvement virtually a necessity. It is no surprise that the
wellness industry has thrived on social media. Not only does the industry
provide the tools for betterment of the self, but the platforms provide fertile
ground for claims about various health and beauty products that go
unchecked.
The core feature of the wellness movement is the permission to focus
on, and obsess over, the presentation of the self. 240 Similar to how shallow
fakes function, the wellness industry simultaneously manufactures the
desire and supplies the tools for betterment. As writer Molly Young
detailed in a deeply studied profile of a “moon juice” purveyor:
What Goop (and acolytes like Moon Juice) sell is the notion that it’s
not only excusable but worthy for a person to spend hours a day
focused on her tiniest mood shifts, food choices, beauty rituals,
exercise habits, bathing routines and sleep schedule. What they sell is
self-absorption as the ultimate luxury product.241
This turn inward means that the external world of facts and
knowledge matters less.242 The wellness industry depends on this turn—
on the ability to disseminate opinions as knowledge and to sell goods
under the guise of presenting facts. Take Goop, the lifestyle brand started
by Gwyneth Paltrow: it began as a newsletter and has grown into a quarter-
billion dollar business.243 Goop is built on the concept of self-improvement

238. See Brian Kennedy et al., Americans’ Trust in Scientists, Other Groups
Declines, PEW RSCH. CTR. (Feb. 15, 2022), https://bit.ly/3C8W1tG.
239. See Dominik Andrzej Stecula et al., How trust in experts and media use affect
acceptance of common anti-vaccination claims, MISINFORMATION REV. (Jan. 14, 2020),
https://bit.ly/3MOTD05.
240. Shallow fakes are of a piece with the obsession over self-improvement and self-
care. See JIA TOLENTINO, TRICK MIRROR: REFLECTIONS ON SELF-DELUSION 80 (2019) (“Old
requirements, instead of being over-thrown, are rebranded. Beauty work is labeled as ‘self-
care’ to make it sound progressive.”).
241. Young, supra note 34.
242. See Peter Dahlgren, Commentary, Public Sphere Participation Online: The
Ambiguities of Affect, 12 INT’L J. OF COMMC’N 2052, 2065 (2018) (“Today, in the viral
world of online information . . . what we feel—is clearly on the rise. Truth becomes
reconfigured as an inner subjective reality with an affective leap and thus becomes the
foundation for validity claims about reality.”).
243. See Brodesser-Akner, supra note 130.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 113

through self-care.244 It sells products like the “$66 ‘Jade Egg,’” a stone
which is meant to be inserted into the vagina and which Goop claims
“could balance hormones, regulate menstrual cycles, prevent uterine
prolapse, and increase bladder control.” None of these claims are
supported by scientific evidence.245 As a result of these and other
assertions, ten California District Attorney’s offices filed suit for false and
misleading advertisement in violation of the state Business and
Professions Code, section 17500.246 Goop settled for $145,000.247
These deceptive marketing practices might be dismissed as the kind
that happen in all industries. But the wellness industry is particularly rife
with deceptive and unfair trade practices and is part of the rising tide of
online misinformation.248 As one report set forth, “[a] surge in
misinformation has grown with the internet, making wellness strategies
appear to have scientific foundations when instead they’re fueling baseless
and sometimes harmful theories.”249 To take another related example,
vitamin sales, which are heavily promoted on social media, have jumped
40% from 2019 to 2020, despite little medical evidence to show that they
increase health or wellness.250
Of course, traditional media also sells beauty, glamour, and the
products that help achieve both. But there are journalistic standards they
must adhere to that are wholly absent in the virtual arena. In 2017, Condé
Nast, a global mass media company that owns Vogue, a fashion magazine
marketed to women,251 decided to partner with Goop to deliver content to
Vogue’s readers.252 The deal soon fell apart. The problem was twofold.
First, Vogue publishes a magazine, not a catalog, which means that it must
enforce a separation between content and product placement—and
therefore a separation between reader and consumer—which Goop does
not do.253 Second, Vogue requires fact-checking and support for scientific
claims made.254 Goop, however, understands that support to be

244. See id.

245. See Bill Bostock, Gwyneth Paltrow’s Goop settles $145,000 lawsuit over
baseless vaginal eggs health claims, BUS. INSIDER (Sep. 5, 2018, 7:15 AM),
https://bit.ly/42isoR8 (noting that the settlement also related to an essential oil that Goop
claimed would cure depression).
246. See Complaint at 3, People v. Goop, Inc. (Cal. App. 4th Supp. Aug. 31, 2018).
No. 18CV001176.
247. See id.
248. See Marisa Fernandez, Beware the “science” behind some wellness industry’s
claims, AXIOS (Feb. 15, 2020), https://bit.ly/3OQzn0J.
249. Id.
250. See id.
251. See Vogue (magazine), WIKIPEDIA, https://bit.ly/43z2rxA.
252. See Brodesser-Akner, supra note 130.
253. See id.
254. See id. As the profile on Goop and Gwyneth Paltrow (“G.P.”) explains:

Electronic copy available at: https://ssrn.com/abstract=4606149

114 PENN STATE LAW REVIEW [Vol. 128:1

unnecessary, given that “they’re never asserting anything like a fact” but
rather “just asking unconventional sources some interesting questions.”255
Facts, which remain relevant to traditional media, are of diminishing
importance to social media.

2. The Erosion of Public Discourse

The disregard for facts in the social media space encourages a distrust
of the information presented in ways that affect our most important
democratic institutions.256 Concerns about the erosion of public discourse
are not new; they have long followed concerns over the rise of social media
generally.257 These concerns were especially acute in the wake of the riots
of January 6, 2021, which were propagated at least in part based on false
information about the results of the 2020 presidential election.258 The
accounts surrounding these events have focused on fake news and
misinformation. They involve the insidiousness of rumors and our biases
towards conspiracy theories. But what share of the blame of our new
digital ecosystem belongs to the fact that we live in a visual medium that
is permeated with fakery?
Here is one plausible account. As people live more of their lives in
spaces where they cannot know whether to trust the images they see, they
will come to distrust the information that is shared. Under this narrative,
the filter and the fake news story are part of the same information
landscape, where everything is believable enough but also, possibly,
faked. As the percentage of material that is faked goes up—and we know
that most people are manipulating their online presentations of self—we

G.P. would say, then what is science, and is it all-encompassing and altruistic
and without error and always acting in the interests of humanity? These questions
had been plaguing Goop for a while – not just what is a fact, or how important is
a fact, but also what exactly is Goop allowed to be suggesting?
Id.
255. Id.
256. In discussing the more specific practice of “stealth marketing,” Ellen Goodman
argues that it “harms . . . by degrading public discourse and undermining the public’s trust
in mediated communication.” Goodman, supra note 34, at 87.
257. See PARISER, supra note 33; see also Khiara M. Bridges, Language on the Move:
“Cancel Culture,” “Critical Race Theory,” and the Digital Public Sphere, 131 YALE L.J.
FORUM 767, 770–71 (2022). Bridges writes:
On social media, rational debate – the hallmark of the civic deliberations that
took place in the Habermasian public sphere – is not a dominant presence. When
one dares to open the Twitter app, one is more likely to encounter abusive speech,
ad hominem attacks, and wildly fact-free and logic-free statements than rational
argumentation.
Id.
258. See Bill McCarthy, Misinformation and the jan. 6 insurrection: when ‘patriot
warriors’ were fed lies, POLITIFACT (June 30, 2021), https://bit.ly/3MxY4w0.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 115

should expect that people will distrust what they see, and that distrust will
spill over into the other kinds of digital information they consume.
Consider again the “Moon Juice” peddler, Amanda Chantal Bacon.
Her clients are made up of the likes of “Gwyneth Paltrow, Emma Roberts,
and Shailene Woodley.”259 She is, famously, a critic of “Western
medicine.”260 Chantal Bacon proudly notes that she has “never paid
influencers.”261 Instead, “[s]ocial media, and specifically Instagram, has
always been important to me.”262 That is, the very nature of the social
media platforms allows her company to flourish. Instagram provides her
with a space to promote her products for profit, by positioning herself as
an expert on “‘wellness and longevity’” and on leading a “‘holistic
lifestyle.’”263
Consider now Alex Jones who, like Chantal Bacon, is a salesman. He
has a website, InfoWars, where he offers “organic fair-trade coffee” that
“can be purchased in an ‘Immune Support’ variety that includes cordyceps
and reishi mushroom extracts.”264 He also markets probiotics and a “Super
Female Vitality” supplement. These are made from the exact same extracts
that Moon Juice sells.265 Jones is perhaps best known as a right-wing radio
host and conspiracy theorist, who was instrumental in propagating the
false narrative that the 2020 election was stolen from Donald Trump.266
He spoke at a rally in DC on January 5, and was subpoenaed to discuss his
knowledge of, and involvement in, the January 6 attack.267
The link between Chantal Bacon, Jones, and January 6, is not as far-
fetched as it might initially seem. Chantal Bacon and Jones both traffic in,
and profit from, misinformation, and their business models depend on a
distrust of facts presented by sources external to themselves. The
background conditions of the platforms make it so that truth becomes a
commodity—with real world consequences that include harming our most
time-worn democratic institutions.268 This entwinement of profit and

259. Elana Lyn Gross, How Moon Juice’s Founder Built Her Wildly Popular
Wellness Brand, FORBES (Aug. 13, 2018), https://bit.ly/3QyIDr5.
260. Id.
261. Gross, supra note 259.
262. Id.
263. Id.
264. See Young, supra note 34.
265. See id.
266. See Frontline, What Conspiracy Theorist Alex Jones Said in the Lead Up to the
Capitol Riot, PBS (Jan. 12, 2021), https://bit.ly/3oyZK0n.
267. See Benjamin Siegel, Conspiracy theorist Alex Jones reveals he appeared
before Jan. 6 committee, ABC NEWS (Jan. 25, 2022, 6:54 PM), https://bit.ly/3oGYhVB.
268. See David Remnick, The Devastating New History of the January 6th
Insurrection, THE NEW YORKER, (Dec. 22, 2022), https://bit.ly/47t5ASt (describing the
attack as “a deliberate, coordinated assault on American democracy that could have easily
ended with the kidnapping or assassination of senior elected officials, the emboldenment
of extremist groups and militias, and, above all, a stolen election, a coup”).

Electronic copy available at: https://ssrn.com/abstract=4606149

116 PENN STATE LAW REVIEW [Vol. 128:1

politics on social media disrupts the possibility of creating any shared civic
discourse.269
The distrust of institutions that leads one to rely on the self—which
must be improved, immunized, and reinforced with various supplements
to withstand whatever may come—straddles the spectrum of political
parties as it does socioeconomic class. As individuals’ focus is pulled into
themselves and their interests, it is pulled away from the creation of a
collective reality or of a shared community.270 The result is that people feel
more alone and more siloed and therefore less engaged as members of
society. As people spend more of their lives online, and occupy spaces that
are rife with fakery, they will be less inclined to engage in a common civic
life.271

V. PLATFORM REGULATION
We have argued for a more robust assessment of the costs of shallow
fakes in both scholarship and policy. What does that mean in terms of
regulation? The first, and most obvious, regulatory move is to demand
greater transparency from social media platforms. Relatedly, the FTC
should sharpen and expand its guidelines around deception on social
media. Finally, we think there is room for voluntary initiatives by social
media firms, akin to the work being done in countering violent extremism
and child sexual abuse, though we note that some of the dominant policy
proposals today—especially antitrust policies aimed at greater
competition—are likely to be unhelpful in this context and may instead
make the problem worse.
Our focus in this Part is on the platforms, not the users, because it is
the platforms that are best situated to address the problem. They create the
market for shallow fakes, they have the most information about what is
happening on their services, and, crucially, they control users’ experiences
by incentivizing them to engage in shallow fakery. While platforms enjoy
broad immunity for much of what their users do under Section 230 of the

269. As researcher Peter Dahlgren has noted in his work on social media and
democracy, “from the standpoint of users, even if our intentions are civic or political, we
are still addressed by and embedded in dominant online consumerist discourses.” Dahlgren,
supra note 242, at 2060–61 (“These discourses offer us subject positions mostly as
consumers, rarely as citizens.”).
270. See TOLENTINO, supra note 240, at 30 (“Facebook’s goal of showing people only
what they were interested in seeing resulted, within a decade, in the effective end of shared
civic reality.”).
271. Jia Tolentino’s interview of author Naomi Klein discusses the consequences of
maintaining this hierarchy of priorities. Klein explains: “[T]he amount of labor we are
putting into optimizing our bodies, our image, our kids, is robbing from the work that needs
to be done to preserve the habitability of the planet, to preserve our humanity in the face
of those spasms.” Jia Tolentino, Naomi Klein Sees Uncanny Doubles in Our Politics, THE
NEW YORKER INTERVIEW (Sept. 10, 2023), https://bitly.ws/Uthu.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 117

Communications Decency Act, they do so only if they are not

“responsible, in whole or in part, for the creation or development of” the
offending content.272 We are centrally concerned here with what the
platforms themselves do. Moreover, as the Ninth Circuit explained in Fair
Housing Council of San Fernando Valley v. Roommates.com, LLC, “a
website may be immune from liability for some of the content it displays
to the public but be subject to liability for other content.”273

A. Transparency Reforms
Before we are ready to prescribe platform regulations with any detail,
we need to know much more about the platforms’ internal operations. How
are they offering filters and who are they targeting? How much are they
filtering by default, without giving users adequate notice? Relatedly, how
much user data are they tracking and towards what ends? We should also
know more about user behavior and how it is shaped by the platforms’
dark patterns. All of this to say: we simply do not know enough about what
social media platforms are doing and how people are using them.
For many of the most important questions, only the social media
platforms have the answers or have access to data that could provide
answers. Unfortunately, the platforms have not been terribly transparent.
As Facebook insider Frances Haugen testified, “I came forward because I
recognized a frightening truth: almost no one outside of Facebook knows
what happens inside Facebook.”274 Despite repeated calls to remedy this
problem, Facebook has refused to share its internal research. As the Wall
Street Journal reported last year, “Facebook has consistently played down
the app’s negative effects on teens, and hasn’t made its research public or
available to academics or lawmakers who have asked for it.”275 Tellingly,
the first time most of the public and regulators became aware of the scope
of the mental health crisis on social media was when a Facebook employee
blew the whistle and leaked internal research.276

272. 47 U.S.C. §§ 230(c) and (f)(3) (1998).

273. Fair Hous. Council of San Fernando Valley v. Roommates.com, LLC, 521 F.3d
1157, 1162–63 (9th Cir. 2008) (finding that Section 230 did not immunize Roommates.com
from liability under public accommodations laws for the actions the platform took,
including actions that influenced user behavior). While the Supreme Court recently
considered reviewing Section 230’s scope, the Court ultimately sidestepped the issue. See
Gonzalez v. Google LLC, 598 U.S. 617, 622 (2023) (“We therefore decline to address the
application of § 230 to a complaint that appears to state little, if any, plausible claim for
relief.”).
274. Testimony From A Facebook Whistleblower, supra note 139.
275. See Wells et al., supra note 21.
276. See Scott Pelley, Whistleblower: Facebook is misleading the public on progress
against hate speech, violence, misinformation, CBS NEWS: 60 MINUTES (Oct. 4, 2021),
https://bit.ly/3og1Xh4.

Electronic copy available at: https://ssrn.com/abstract=4606149

118 PENN STATE LAW REVIEW [Vol. 128:1

While platforms publish transparency reports and have undertaken

efforts to share their inner workings, these have been a limited and
woefully inadequate substitute for independent research.277 These
transparency reports mostly focus on content takedowns and law
enforcement requests; they largely ignore the problems created by the
platforms’ algorithms or filter tools.278 When platforms do share
information more broadly, they choose to do so only with hand-picked
researchers.279 When these efforts are more ambitious, like the Social
Science One project, which aimed to give researchers insider access to
social media data, they have stalled.280
On their own, then, platforms have done too little on transparency.
Even worse, they have aggressively blocked and threatened independent
researchers seeking to study what kinds of material people consume on
social media. For example, when researchers at New York University put
together a tool to study how political advertisements are targeted on the
platforms—something that is of considerable public interest—Facebook
sent them a cease-and-desist letter, ultimately shutting off their access to
the platform for violating the terms of service.281 Other researchers have
had similar experiences.282
Given the current landscape, the following two reforms would make
considerable inroads: (1) the platforms should be required to make more
information available to independent researchers and the public at large;
and (2) independent researchers should not fear criminal or civil liability
for engaging in non-commercial, public-interest research. Outlining the
full contours of a legislative solution to this problem is beyond the scope
of this Article, but the Platform Transparency and Accountability Act is a
good start.283 That bill would create a safe harbor for researchers engaged

277. See, e.g., Mathew Ingram, Facebook “transparency report” turns out to be
anything but, COLUM. JOURNALISM REV. (Aug. 26, 2021), https://bit.ly/43dqz8Q.
278. For example, Meta’s recently created “Widely Viewed Content Report” is
supposed to bring transparency into the way information goes viral on the platform, yet it
provides such a high-level overview, it gives only a snapshot of “what a typical feed looks
like,” and does not give granular information about how information spreads. See, e.g.,
Widely Viewed Content Report: What People See on Facebook Q1 2023 Report, META,
https://bit.ly/3OoHlxm (last visited Sept. 12, 2023).
279. See Steven Levy, It’s Time to Talk About Facebook Research, WIRED (Sept. 17,
2021), https://bit.ly/42Whv8H.
280. See Craig Timberg, Facebook made big mistake in data it provided to
researchers, undermining academic work, WASH. POST (Sept. 10, 2021),
https://wapo.st/3WjbWi2.
281. See Taylor Hatmaker, Facebook cuts off NYU researcher access, prompting
rebuke from lawmakers, TECHCRUNCH (Aug. 4, 2021, 8:17 PM), https://bit.ly/3DrXoEs.
282. See Ethan Zuckerman, Facebook has a misinformation problem, and is blocking
access to data about how much there is and who is affected, THE CONVERSATION (Nov. 2,
2021, 8:27 AM), https://bit.ly/41MtJPW.
283. See Senator Coons Press Release, Coons, Portman, Klobuchar Announce
Legislation to Ensure Transparency at Social Media Platforms (Dec. 9, 2021),

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 119

in noncommercial research and also allow the National Science

Foundation to act as a kind of gatekeeper for reviewing research proposals
and requiring platforms to provide relevant data.284 Mandating this kind of
platform transparency and protecting researchers seeking transparency are
essential first steps to exposing the incidence and the impact of mass
deception in our lives online.

B. Deceptive and Unfair Trade Practices by the Platforms

There are few legal restrictions that address shallow fakes. Where
they exist, they are difficult to interpret and sparingly enforced. Overall,
agency regulation of deception in social media has been thin, sporadic, and
entirely sectoral. Kim Kardashian, one of the most successful Instagram
influencers with hundreds of millions of followers, has been fined by a
regulatory agency for deceptive marketing only once—by the Securities
and Exchange Commission (SEC), in relation to cryptocurrency
advertising, and never by the FTC.285 This example illustrates a further
limitation of existing regulation—that it targets individual users, rather
than the social media platforms.
One obvious context where fakery clearly has the potential to be
regulated is in advertisements. The rules that have been promulgated,
however, tend to be tangential to the problem of shallow fakes, and hardly
ever enforced. In 2019, the FTC promulgated an updated memo on the pre-
existing guidelines for influencers on social media.286 The guidelines
include clarifications around when and how to disclose a paid partnership.
They indicate, for instance, that “[i]f your endorsement is in a picture on
a platform like Snapchat and Instagram Stories, superimpose the
disclosure over the picture and make sure viewers have enough to notice
and read it.”287 Additionally, the FTC asserts that “[y]ou can’t talk about
your experience with a product you haven’t tried” and that influencers
cannot say a product is great if they have tried it and thought it was

https://bit.ly/3pZgVZ3. For a more detailed account of why researchers need protected

access to social media platform data, see Alex Abdo et al., A Safe Harbor for Platform
Research, KNIGHT FIRST AMEND. INST. (Jan. 19, 2022), https://bit.ly/3MEPNaU.
284. See Jeff Horwitz, Senators Want Social-Media Apps to Share Research, WALL
ST. J. (Dec. 9, 2021), https://on.wsj.com/3ojV1zt.
285. See Press Release, SEC Charges Kim Kardashian for Unlawfully Touting
Crypto Security, U.S. SECURITIES AND EXCHANGE COMMISSION (Oct. 3, 2022),
https://bit.ly/3Wrjj7d (describing $1.26 million fine for Kardashian’s failure to disclose
that she was paid $250,000 to post her support of crypto asset EthereumMax).
286. See Disclosures 101 for Social Media Influencers, FEDERAL TRADE COMMISSION
(Nov. 2019), https://bit.ly/41STyxC; see also 16 CFR Part 255, Guides Concerning the
Use of Endorsements and Testimonials in Advertising, FEDERAL TRADE COMMISSION
(2009), https://bit.ly/3WirfHI.
287. Disclosures 101 for Social Media Influencers, supra note 286, at 4.

Electronic copy available at: https://ssrn.com/abstract=4606149

120 PENN STATE LAW REVIEW [Vol. 128:1

terrible.288 These are helpful, to be sure. But these guidelines do not apply
to the vast majority of the deception we describe, in which people are not
actively promoting a specific product. Even in the narrow situations they
are meant to cover, Alexandra Roberts has found that “[f]alse advertising
claims based on the use of editing software to improve people’s
appearance” are unlikely to succeed because the FTC looks for explicitly
misleading statements about a product’s efficacy.289 These rules are also
poorly enforced.290 As such, one study estimated that only 7% of all
sponsored influencer posts comply with FTC rules.291
Because the FTC has not been especially active in this space, some
scholars have turned to the Lanham Act as a promising option, given that
it allows for causes of action to be privately enforced.292 But making out a
successful claim is still difficult. In Lokai Holdings, LLC v. Twin Tiger
USA, LLC, Twin Tiger alleged, among other things, that competitor
Lokai’s “failure to disclose that it compensates certain influencers,
celebrities, and media outlets for their endorsement of Lokai products in
online and social media advertising is likely to deceive reasonable
consumers.”293 Yet the district court denied the claim, concluding that “the
Lanham Act does not impose an affirmative duty of disclosure.”294
Accordingly, “failure to disclose compensation to celebrities and
influencers for promoting its products is not actionable under the Lanham
Act.”295
These examples address only one small slice of the fakery taking
place online, and the focus is always on individual users or brands, as
opposed to the platforms or their policies as a whole. Our reforms are
aimed at the platforms themselves. Specifically, we call upon the FTC to
promulgate rules in this context. While we are not the first to do so,296 our

288. Id. at 6.
289. Roberts, supra note 21, at 114. Roberts writes:
[T]hose that equate to a false or misleading statement about the product’s
efficacy—like photoshopping whiter teeth in an influencer ad for a tooth
whitening product or longer lashes in an ad for a lash-lengthening mascara—
seem more likely to be fair game.
Id.
290. See id. at 120.
291. See MEDIAKIX, supra note 15.
292. See Roberts, supra note 21, at 86–88 (noting that the FTC “lacks the resources
and perhaps the authority to enforce industry-wide change” but that the Lanham Act allows
for private companies to sue one another and arguing in favor of “private actors . . . us[ing]
the Lanham Act to challenge competitors’ false influencing”).
293. Lokai Holdings, LLC v. Twin Tiger USA, LLC, 306 F. Supp. 3d 629, 639
(2018).
294. Id. at 640.
295. Id.
296. At least four student notes have identified this problem along with several
practitioners. See, e.g., Lauryn Harris, Too Little, Too Late: FTC Guidelines on “Deceptive
and Misleading” Endorsements by Social Media Influencers, 62 HOW. L.J. 947 (2019);

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 121

emphasis is much broader than what has been previously suggested: we

are concerned with holding the platforms, rather than individual users,
influencers, or even brands, accountable.297 As we have shown, the
platforms are an unusual market, but a market nonetheless—they are
advertising platforms where attention is being monetized.298 Everything
everyone posts on the platforms, even if not directly sponsoring a product
or service, is in some sense advertising, contributing directly or indirectly
to the promulgation of the advertising market run by the platforms. As
such, it would not be an extreme step beyond previous exercises of FTC
authority to take action to address the confusion, fraud, and mental health
impact on platform consumers and users.
The FTC, under the authority of the Federal Trade Commission Act,
has the necessary leeway to act.299 We can imagine two specific actions:
(1) new FTC rules around platforms that discourage deception among
users, and (2) more enforcement by the FTC against platforms around
these issues. The FTC was granted the authority, in part, to regulate
deception in the market.300 And the FTC has regulated other aspects of
consumer welfare on these platforms, including information privacy.301
What should those rules contain? The FTC’s own guidelines for
influencers offer a starting point. The FTC’s focus has been on disclosure,
so that users know that someone peddling a product is in fact doing so.
This is, unsurprisingly, precisely the opposite of what advertisers and the
platforms want.302 Thus, the FTC could similarly advocate for two

Laura E. Bladow, Worth the Click: Why Greater FTC Enforcement Is Needed to Curtail
Deceptive Practices in Influencer Marketing, 59 WM. & MARY L. REV. 1123 (2018); Tisha
James, The Real Sponsors of Social Media: How Internet Influencers Are Escaping FTC
Disclosure Laws, 11 OHIO ST. BUS. L.J. 61 (2017); Christopher Terry et al., Throw the Book
at Them: Why the FTC Needs to Get Tough With Influencers, 29 J. L. & POL’Y 406 (2021).
297. The problem has been framed as finding plaintiffs, see Roberts, supra note 21,
at 87–88, but we are ultimately concerned with identifying the correct defendant. There
are, of course, certain users, like influencers, and especially those with particularly large
followings, who should be held accountable. The problems we have identified, however,
are with platforms’ policies, and our concern here is therefore with regulating platform
behavior.
298. See generally WU, supra note 10.
299. See 15 U.S.C. § 45(a).
300. See F.T.C. v. Colgate-Palmolive Co., 380 U.S. 374, 384–85 (1965) (describing
how the Federal Trade Commission Act was significantly amended in 1938 to include a
prohibition on “deceptive acts or practices in commerce”).
301. See Daniel J. Solove & Woodrow Hartzog, The FTC and the New Common Law
of Privacy, 114 COLUM. L. REV. 583, 586 (2014) (describing how the FTC has built up an
extensive body of regulations regarding privacy, which “is functionally equivalent to a
body of common law”).
302. See Roberts, supra note 21, at 84–85 (noting that “[o]mitting sponsorship
disclosure enables paid content to masquerade as organic buzz and peer-to-peer
testimonial, rendering misrepresentations even more persuasive,” and so “the majority of
influencers and brands go out of their way to obscure the nature of their relationship”).

Electronic copy available at: https://ssrn.com/abstract=4606149

122 PENN STATE LAW REVIEW [Vol. 128:1

important kinds of disclosure. First, they could require platforms to clarify

when content on the platform is being altered and how. If TikTok videos
are filtered, the filtering should be plainly explained to users. These laws
would be directed at the platforms to address some of the harms that stem
from digital manipulation. Laws could include, for example: demanding
transparency around manipulated videos and filters; requiring users to
share whether their media has been manipulated; using automated tools to
screen for manipulated media; discouraging the use of manipulated media;
and monitoring users for mental health problems related to manipulated
media (monitoring of the kind that Facebook was undertaking in secret).303
Once the FTC knows more about the way the algorithms function, it could
make specific recommendations based on the searches the users undertake
and the algorithms that lead to specific results.
Second, users themselves could be encouraged to be transparent
about the modifications they apply to their posts. While our focus is on the
platforms, addressing the users might be an effective supplement and a
way to enhance user autonomy. The users themselves could help change
the baseline from the barely perceptible filter to the filter labeled as such,
especially for those who have large followings.304
Once these rules are in place, the FTC should step up its enforcement.
Enforcement under these new regulations has the potential to be less
involved and less costly in at least one way: rather than seek out individual
actors online, the FTC can focus its resources on the limited number of
social media platforms that control all our online activity.

C. Other Initiatives
In the absence of new regulations, industry reforms are a second-best
solution. While there are good reasons to be skeptical of voluntary industry
initiatives, there are several compelling precedents of industry-wide norms
developed by social media firms. For example, the Global Internet Forum
for Countering Terrorism (“GIFCT”) allows firms to harmonize their
efforts to combat extremist imagery, along with other counter-terrorism
steps. In 2017, the firms that make up the GIFCT created an industry
database of “perceptual hashes of known images and videos produced by

303. See facebook files, supra note 25.

304. Users with large numbers of followers are also privy to an especially close
relationship with the platforms. As Frier explains:
Today, Kim Kardashian West has 157 million followers and makes about $1
million for a single post. Paris Hilton eventually joined Instagram too, and now
has 11 million followers. Porch [who works at Instagram] now has employee
counterparts in Los Angeles who answer celebrity queries for the Kardashians
and others, solving their problems directly while most of the app’s users fend for
themselves.
FRIER, supra note 4, at 138–39.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 123

terrorist entities on the United Nations designed terrorist groups lists—

which GIFCT members had removed from their services.”305 Another
example is the hash database that the firms share with the National Center
for Missing and Exploited Children (“NCMEC”) for child sexual abuse
material (“CSAM”).306 This is a massive program, handling tens of
millions of reports of CSAM in a year, and most of these are from large
social media platforms.307
While the GIFCT and NCMEC hash databases are different in
important respects—the former is an entirely private multistakeholder
initiative while the latter is congressionally mandated—they offer lessons
for the development of industry-wide initiatives. We could imagine the
firms that make consumer products to advertise coming together in
agreement—perhaps along with advertising agencies—to self-regulate in
various ways. At a minimum, they could all endorse the FTC guidelines,
which include rules around transparency and disclosure about the products
being promoted. These firms could further agree not to use filters in their
own posts, which could encourage everyday social media users to do the
same. While this might sound far-fetched, it has happened in traditional
media, where a considerable effort has been made to reduce the use of
airbrushing and to encourage a wider acceptance of different body
types.308 For example, Olay, the cosmetics giant, has pledged to stop using
airbrushing in its advertising campaigns.309
But we have already seen that the problem goes far beyond what are
technically considered advertisements, so we would not want to limit these
initiatives to firms seeking to advertise. It would be much more impactful
to apply them to the platforms themselves. This would mean that
Instagram (and its owner Meta), Snapchat, and TikTok, among others,
would devise a set of limitations for modified content. They need not
wholly ban filters and other popular methods of altering images—it would
already be significant if they placed a label on images that were digitally
altered, much in the way that advertisers have proposed noting which ads

305. David Cohen, Global Internet Forum to Counter Terrorism Expands Scope of
Its Database, ADWEEK (July 26, 2021), https://bit.ly/44YHtsG (last visited Jul. 9, 2023).
306. See Anirudh Krishna, Note, Internet.gov: Tech Companies as Government
Agents and the Future of the Fight Against Child Sexual Abuse, 109 CAL. L. REV. 1581,
1602 (2021).
307. See id. at 1603.
308. See, e.g., Vanessa Friedman, Airbrushing Meets the #MeToo Movement. Guess
Who Wins, N.Y. TIMES (Jan. 15, 2018), https://nyti.ms/3MI6ulM (describing plans by CVS
and other firms to stop “materially altering” imagery in advertisements); see also Eric
Pfanner, A Move to Curb Digitally Altered Photos in Ads, N.Y. TIMES (Sept. 27, 2009),
https://bit.ly/3IrJzbG (describing a French law to ban airbrushing).
309. See Becky Bargh, Olay pledges to stop airbrushing advert campaigns,
COSMETICS BUS. (Feb. 24, 2020), https://bit.ly/3BKC7ov.

Electronic copy available at: https://ssrn.com/abstract=4606149

124 PENN STATE LAW REVIEW [Vol. 128:1

have been significantly altered.310 This labeling would allow users to better
distinguish between real and fake images and give them the tools to more
accurately interpret the posts.
This kind of initiative, which is ambitious on its own, would still not
remedy many of the problems described here. Users could still voluntarily
edit their photos using native photo-editing software and then upload them
to social media platforms, which likely would not be able to identify
whether the image was doctored. Even sophisticated face-recognition
software depends on a training image; if someone only posts edited images
of their face, that would be the face identified by the software. And many
of the problems described here—like a photo simply being taken out of
context—do not only concern editing software.
Any initiative, therefore, must also include media literacy training.
For more robust protection against trickery, social media users need to be
critical consumers who are versed in identifying fakes and, more
importantly, taking what they see with a healthy (rather than democracy-
defeating) dose of skepticism so that they do not think everything is fake
and can instead assess the difference between fact and fiction. Educating
users to be critical will not fundamentally change the nature of social
media platforms,311 but it will enable users to make a distinction between
the digital and the physical world in ways that give users more control and
more information about what exactly they are consuming.
One final way that industry norms could help would be to cultivate a
diversity of images across users’ visual fields. Just as people speak of
needing a balanced “information diet” to combat the filter bubble,
platforms could ensure that users receive a balanced diet of images—
filtered and unfiltered.312 This would go some way towards combatting the
unrealistic standards that are presented as the norm on social media
platforms. Being exposed to images of different body types would also put
the brakes on the common experience of tunneling down a filter bubble
where one is fed only one particular kind of image.

VI. CONCLUSION
There is a great deal of talk about the “metaverse,” a digital world
where we will be able to leave our bodies behind.313 The foundations for

310. See, e.g., Vanessa Friedman, supra note 308 (describing how CVS will mark
images that are not significantly digitally altered).
311. See TOLENTINO, supra note 240, at 19 (“The internet is engineered for this sort
of misrepresentation; it’s designed to encourage us to create certain impressions rather than
allowing those impressions to arise ‘as an incidental by-product of [our] activity.’”).
312. See Steven Leckart, Balance Your Media Diet, WIRED (Jul. 15, 2009),
https://bit.ly/45bAvS6.
313. See Eric Ravenscraft, What Is the Metaverse, Exactly?, WIRED (Nov. 25, 2021),
https://bit.ly/3MpVGHs.

Electronic copy available at: https://ssrn.com/abstract=4606149

2023] SHALLOW FAKES 125

that world are being built on today’s digital platforms.314 But today’s
platforms are awash in deceit, through deepfakes and shallow fakes alike.
Ours is a visual field marked by intense pressure to conform to a particular
ideal of the perfect self, one that can only ever exist in a fake world.315 For
most of us, though, our online images are not our reality. Let us keep it
that way.

314. See Katherine Fung, Facebook Changes Company Name to ‘Meta’ in Rebrand,
Social Network Name Will Stay, NEWSWEEK (Oct. 28, 2021, 2:38 PM),
https://bit.ly/44DLyD5 (“The metaverse is the next evolution of social connection. Our
company’s vision is to help bring the metaverse to life, so we are changing our name to
reflect our commitment to this future.”).
315. Consider @lilmiquela who is “a 19-year-old Robot living in LA.” See
@lilmiquela, INSTAGRAM, https://bit.ly/3Iwl4dI (last visited June 26, 2023). Lucy
Blakiston explains:
Miquela’s ‘life’ is the definition of unrealistic, because she’s literally an
amalgamation of a perfectly selected bunch of pixels. But between Photoshop,
Facetune, filters and all the other bullshit that exists these days to help us airbrush
our online lives, Miquela’s façade really isn’t that different from any other
influencer. And maybe that’s the point.
Lucy Blakiston, Lil Miquela and the rise of the robot influencer, THE SPINOFF (July 15,
2021), https://bit.ly/3r72ByA.

Electronic copy available at: https://ssrn.com/abstract=4606149

 DEEPFAKE
MANIPULAT SAU INFORMAT?

Notă: acest ghid a fost parțial elaborat cu

ajutorul inteligenței artificiale (IA).

_TLP:CLEAR_
Publicul țintă al ghidului
Acest ghid se adresează persoanelor cu vârsta peste 18 ani, în vederea creșterii gradului de conștientizare
și înțelegere asupra utilizării instrumentelor de inteligență artificială (IA) de tip generativ pentru crearea
de conținut video, audio sau de imagini, ori privind detectarea utilizării unor astfel de instrumente.
Deepfake, o definiție
Deepfake este o manipulare digitală a unei înregistrări video, audio sau a unei imagini, realizată cu ajutorul
inteligenței artificiale (IA) sau a altor programe specializate.
Un exemplu: Există multiple cazuri în care un videoclip cu o personalitate publică oferă sfaturi financiare.
În acest videoclip, persoana recomandă insistent investiția într-o anumită companie sau afacere,
promițând profituri mari și riscuri minime.
Detaliile contează: Videoclipul este realizat profesional, cu o calitate a imaginii și sunetului ridicată.
Persoana publică pare sinceră și convingătoare, folosind un limbaj accesibil și argumente persuasive.
Scopul campaniei: Un astfel de Deepfake ar putea fi folosit pentru a manipula populația să investească
într-o afacere ilicită (scam 1), deținută sau controlată de manipulatori.
Impact: Videoclipul este distribuit rapid pe rețelele de socializare, cu un potențial uriaș de a fi vizionat
de milioane de oameni. Mulți dintre cei care văd videoclipul ar putea fi convinși să investească în afacerea
recomandată, riscând ulterior să piardă sume semnificative de bani.
De ce este important să înțelegem Deepfake
Înțelegerea Deepfake-urilor este crucială, având implicații semnificative în societate, politică, securitate
cibernetică și în manipularea încrederii publice.
Impact asupra adevărului și încrederii: Deepfake-urile pot distorsiona realitatea, creând conținut video
sau audio extrem de realist, care poate fi greu de distins de materialele autentice. Aceasta poate submina
încrederea în mass-media și în informațiile pe care ne bazăm pentru a lua decizii informate.
Securitate și fraude: Tehnologia Deepfake poate fi folosită pentru a crea înregistrări false care implică
persoane în activități pe care nu le-au realizat sau pentru a produce dovezi false în contexte juridice. De
asemenea, poate fi utilizată în tentative de fraudă, cum ar fi falsificarea identității în apeluri telefonice
sau video pentru a obține acces neautorizat la informații sau resurse financiare.
Manipulare și dezinformare: În politică și în alte domenii, Deepfake-urile pot fi folosite pentru a manipula
opinia publică, a discredita adversarii sau a crea confuzie. Acestea pot influența alegerile, relațiile
internaționale și pot alimenta teorii ale conspirației.
Impact social și etic: Utilizarea imorală a tehnologiei Deepfake, pentru crearea de materiale
compromițătoare sau hărțuirea online, ridică probleme serioase de etică și respectare a drepturilor
individuale.
Pregătire pentru viitor: Pe măsură ce tehnologia continuă să avanseze, este probabil ca Deepfake-urile
să devină tot mai sofisticate și mai greu de detectat. Înțelegerea modului în care funcționează și a
modalităților de a le identifica este esențială pentru dezvoltarea de tehnologii și strategii care să
contracareze utilizarea lor rău intenționată.
Educație și conștientizare: Educația publicului despre existența și capacitatea Deepfake-urilor poate
ajuta la atenuarea impactului lor prin creșterea scepticismului sănătos față de conținutul dubios și
încurajarea verificării informațiilor din surse multiple.
Pentru toate aceste motive, este important să dezvoltăm un nivel colectiv de înțelegere a Deepfake-urilor
și a potențialului lor de a afecta societatea. Acest lucru ne va permite să navigăm într-o lume digitală tot
mai complexă cu discernământ și o mai mare precauție.

1
Mod general de a defini diferite tipuri de fraudă cu care se confruntă utilizatorii atunci când folosesc Internetul. Cei care desfășoară astfel de
activități folosesc instrumente tehnice sau tehnici de inginerie socială cu intenția de a obține foloase financiare.

_TLP:CLEAR_ Pagina 2 / 8
Tehnologia din spatele Deepfake
Deepfake-urile sunt create folosind o combinație de tehnici de IA și învățare automată (Machine Learning
- ML). Tehnologiile cheie implicate sunt:
1. Rețele neuronale convoluționale (CNNs): Sunt tipuri de rețele neuronale artificiale specializate în
analiza imaginilor și a videoclipurilor. Ele sunt antrenate pe seturi mari de date, imagini și videoclipuri
reale pentru a învăța caracteristicile faciale, expresiile, mișcările corpului și alte detalii vizuale.
2. Rețele neuronale generative (GANs): Rețele neuronale artificiale care pot genera conținut nou, realist,
similar cu datele pe care au fost antrenate. În contextul Deepfake, GAN-urile sunt utilizate pentru a
genera imagini și videoclipuri false care sunt foarte asemănătoare cu cele reale.
3. Învățarea automată (ML): Este utilizată pentru a antrena algoritmii Deepfake să identifice și să
manipuleze elemente specifice ale imaginilor și videoclipurilor, cum ar fi expresiile faciale, mișcările
buzelor, sincronizarea audio, etc.
Procesul de creare a unui Deepfake
Colectarea datelor: Pentru a crea un Deepfake convingător, este necesară o cantitate mare de date pentru
antrenament care să includă imagini și videoclipuri capturate din diferite unghiuri și în diverse situații. Cu
cât datele sunt mai variate și de calitate mai bună, cu atât Deepfake-ul va fi mai realist.
Antrenarea modelului: Utilizând datele colectate, algoritmi de IA și ML sunt antrenați pentru a identifica
și învăța caracteristicile unice ale persoanei țintă, cum ar fi trăsăturile faciale, expresiile și modul în care
se mișcă sau vorbește. Scopul acestei faze este de a permite modelului să reproducă aceste detalii cu o
precizie cât mai mare.
Generarea Deepfake: După ce modelul este suficient antrenat, este folosit pentru a crea conținut
falsificat, în care persoana țintă spune sau întreprinde acțiuni neadevărate. Etapa implică generarea de
imagini sau secvențe video artificiale, dar extrem de realiste, utilizând capabilitățile modelului antrenat.
Manipularea detaliilor: Pentru a spori autenticitatea și credibilitatea Deepfake-ului, detaliile fine sunt
ajustate meticulos. Aceasta include sincronizarea mișcărilor buzelor cu conținutul audio falsificat,
corectarea expresiilor faciale pentru a se potrivi cu contextul generat și finisarea altor detalii minore care
contribuie la realismul general al conținutului.
Distribuirea: Deepfake-ul final poate fi distribuit online prin intermediul rețelelor sociale, al platformelor
de sharing video sau chiar prin mesagerie. Intenția distribuirii poate fi de a înșela, de a discredita o
persoană sau de a manipula opinia publică.
Exemple reale de utilizare a Deepfake
Banca Națională a României (BNR) a avertizat publicul cu privire la o schemă de înșelăciune care implică
utilizarea tehnologiei Deepfake pentru a crea videoclipuri false cu guvernatorul BNR. În aceste videocli-
puri, guvernatorul pare să promoveze o platformă de investiții, însă BNR a declarat că acestea sunt false.
Scamul folosește IA pentru a modifica vocea și imaginea guvernatorului, cu scopul de a induce în eroare
publicul pentru a participa la investiții frauduloase, promițând câștiguri financiare rapide și ușoare.

Figura 1 SCAM | Deepfake cu guvernatorul BNR Mugur Isărescu care promovează o platformă de investiții

_TLP:CLEAR_ Pagina 3 / 8
O femeie pensionară din Vaslui a fost victima unei escrocherii postată online pe platforma YouTube.
Escrocii au creat un videoclip fals în care un cunoscut bancher și alte personalități cunoscute recomandau
o platformă de investiții. Promisiunea unor profituri rapide a convins pensionara să investească suma de
52.000 de lei, economii strânse în 20 de ani de muncă. Chiar dacă femeia a raportat această fraudă
autorităților, experții consideră că șansele de recuperare a fondurilor pierdute sunt minime i.
Actorii rău intenționați devin tot mai inventivi. Acum pot imita perfect vocea unei persoane dragi pentru
a vă păcăli. Te pot suna pretinzând că sunt un membru al familiei care are nevoie urgentă de bani.
Recomandăm ca întotdeauna să fie verificată identitatea apelantului, chiar dacă pare să fie o rudă sau un
prieten cunoscut. Nu trimite niciodată bani în grabă și anunță imediat autoritățile dacă ai suspiciuni. Doar
prin vigilență și informare te poți proteja de această formă de fraudă ii.
În 2019, un Deepfake audio a fost folosit pentru a înșela un CEO cu 220.000 de euro. Directorul general al
unei firme de energie cu sediul în Marea Britanie a crezut că vorbea la telefon cu directorul general al
companiei-mamă germane atunci când a urmat ordinele de a transfera imediat 220.000 de euro în contul
bancar al unui furnizor maghiar. De fapt, vocea aparținea unui escroc care folosea tehnologia vocală IA
pentru a-l imita pe directorul executiv german iii.
Pericolul Deepfake în contextul electoral
În era digitală actuală, în care granița dintre realitate și ficțiune se estompează în mod constant, procesul
electoral depășește simpla confruntare a ideologiilor și promisiunilor politice, transformându-se într-un
teren complex de luptă ideologică. Tehnologiile Deepfake, capabile a sintetiza realist imagini și voci , pot
influența semnificativ opinia și votul alegătorilor în timpul campaniilor electorale, având un impact major
asupra procesului democratic, din următoarele considerente:
Impactul asupra politicienilor: Reputația unui politician poate fi grav afectată de videoclipuri Deepfake
fabricate, care pot discredita imaginea sa și pot deteriora șansele de a câștiga alegerile. Răspândirea
dezinformării prin Deepfake poate manipula percepția publică asupra caracterului și programului politici-
anului, afectând negativ cariera sa politică.
Impactul asupra partidelor politice: Deepfake poate fi utilizat ca instrument strategic pentru a discredita
partidele rivale, prin crearea de materiale false care să le prezinte într-o postură negativă. Promovarea
agendei propriului partid prin Deepfake poate fi o modalitate eficientă de a influența opinia publică și de
a atrage alegători. Utilizarea necorespunzătoare a Deepfake de către partidele politice poate duce la
scandaluri și la pierderea încrederii publicului în sistemul politic.
Impactul grupurilor de interese: Deepfake poate fi utilizat de grupuri de interese din interiorul unei țări
pentru a manipula opinia publică legată de probleme civice. Crearea de conținut falsificat adaptat la
contextele locale poate intensifica sau calma nemulțumirile legate de anumite probleme sau evenimente.
Utilizarea Deepfake de către grupurile de interese poate duce la polarizarea societății și la o erodare a
discursului public democratic.
Tehnicile IA/Deepfake utilizate în influențarea campaniilor electorale:
• Falsificarea video realistă: Algoritmi avansați de IA creează videoclipuri aproape imposibil de distins de
cele reale, prezentând scene false cu politicieni sau evenimente inventate.
• Clonarea vocii și falsuri audio: Vocea unui politician este reprodusă cu o precizie uimitoare, generând
mesaje false atribuite în mod eronat persoanei respective.
• Generarea de text sintetic: Texte credibile sunt create imitând stilul și tonul unei personalități politice
sau a unei instituții, răspândind dezinformare.
• Schimbarea fețelor și transformarea: Tehnologia permite modificarea fețelor în videoclipuri, creând
scenarii false care nu s-au petrecut niciodată sau nu în contextul prezentat.
• Predicția comportamentală: IA analizează comportamentul online pentru a prezice răspunsul la anumite
mesaje, facilitând crearea de Deepfake-uri țintite pentru a influența segmente specifice de alegători.
Aceste tehnici pot fi utilizate de actori malițioși pentru a discredita candidați, a manipula opinia publică
și a submina încrederea în procesul electoral. Este esențială conștientizarea pericolelor Deepfake și
implementarea de măsuri pentru a combate dezinformarea și proteja integritatea alegerilor.

_TLP:CLEAR_ Pagina 4 / 8
Semne de identificare a unui Deepfake
Protejarea împotriva Deepfake-urilor este o provocare continuă, deoarece tehnologia se perfecționează
constant, făcând conținutul manipulat tot mai greu de distins de cel real. Cu toate acestea, există anumite
indicii care pot trăda un Deepfake, iată la ce anume trebuie să fim atenți:
• Mediul înconjurător (de exemplu, umbre inexistente, reflexii prea puternice, zone neclare)
• Imperfecțiuni ale feței (alunițe nerealiste, clipire nesincronizată, distorsiuni în interiorul gurii cum ar
fi lipsa dinților și a limbii, dinți mult prea perfecți etc.)
• Nesincronizarea vorbirii/sunetului și a mișcării buzelor, de exemplu: din cauza strănutului
Nesincronizarea vorbirii/sunetului și a mișcării buzelor poate fi observată la pronunțarea literelor b, m și
p. Uneori apar pixeli în nuanțe de gri la marginile componentelor modificate. Se poate distinge dacă este
vorba de o falsificare și atunci când persoana din înregistrare este privită dintr-un alt unghi. Dacă pentru
crearea conținutului Deepfake nu s-au folosit fotografii ale persoanei din unghiuri diferite, algoritmul nu
poate deduce aspectul persoanei din alt unghi, rezultând distorsiuni.
Massachusetts Institute of Technology (MIT) este una dintre entitățile care
dezvoltă instrumente ML care pot identifica dacă un conținut este autentic
sau un Deepfake, oferind totodată un chestionar educativ pentru a învăța
publicul cum să facă această distincțieiv.
Un detector clasic de conținut vizual fals se bazează pe detectarea erorilor
rezultate din prelucrare. Cel mai adesea, aceasta implică analiza pixelilor
pe care ochiul uman nu îi poate vedea, deoarece prin manipularea imaginii
marginile componentei modificate au caracteristici speciale. Un astfel de
algoritm este hibridul dintre Long short term memory (LTSM – o rețea
neuronală) și algoritmul Encoder-Decoder. Acesta funcționează analizând
în paralel fiecare pixel individual și/sau întreaga imagine/videoclip
comprimat(ă). În cele din urmă, rezultatele celor două funcții sunt
comparate și dacă ambele indică aceeași regiune, materialul este considerat modificat.
Există, de asemenea, diferite tipuri de detectoare de acces. Detectorul Intel v se bazează pe observarea
unor indicii subtile, invizibile ochiului uman, pentru a verifica autenticitatea conținutului. Prin analiza
semnelor precum dilatarea pupilelor, modificări ale culorii vaselor de sânge în concordanță cu bătăile
inimii etc., se poate determina dacă acel conținut este autentic vi.
Instrumente și tehnici de detectare a Deepfake
Detectarea Deepfake-urilor este o provocare în continuă evoluție, întrucât tehnologiile de IA care stau la
baza creării Deepfake-urilor devin din ce în ce mai sofisticate. Ca răspuns, cercetătorii și dezvoltatorii
lucrează la dezvoltarea de instrumente și tehnici noi pentru a identifica aceste falsuri. Iată câteva dintre
cele mai promițătoare abordări în detectarea Deepfake-urilor:
Analiza comportamentală: Această metodă se bazează pe identificarea micilor imperfecțiuni sau anomalii
în comportamentul sau mișcările fizice ale subiectului din videoclip.
Consistența iluminării: Detectarea inconsistențelor în iluminare este o altă tehnică eficientă. Algoritmii
analizează umbrele, reflexiile și modul în care lumina se reflectă pe diferite suprafețe ale feței pentru a
determina dacă imaginea a fost manipulată.
Analiza texturii pielii: Tehnicile de Deepfake adesea netezesc textura pielii sau introduc anomalii la
nivelul texturii. Detectarea acestor modificări, care sunt adesea subtile și greu de observat cu ochiul liber,
poate ajuta la identificarea manipulărilor.
Detectarea artefactelor de compresie: Videoclipurile și imaginile manipulare prin IA pot prezenta
artefacte de compresie unice datorită procesului de generare și compresie. Analiza acestor artefacte poate
oferi indicii că materialul a fost alterat.
Examinarea metadatelor: Deși Deepfake-urile în sine pot fi convingătoare, metadatele asociate cu un
fișier video sau imagine (cum ar fi data creării, tipul camerei etc.) pot fi contradictorii sau suspecte,
sugerând manipulare.

_TLP:CLEAR_ Pagina 5 / 8
Verificarea consistenței respirației și a pulsului: Unele tehnici avansate includ analiza variațiilor minore
în culoarea feței sau a umbrelor, care pot indica bătăile inimii și respirația. Modificările în aceste modele
pot indica prezența unui Deepfake.
Utilizarea rețelelor neuronale convoluționale: CNN-urile sunt folosite pentru a analiza videoclipurile
frame cu frame, învățând caracteristicile specifice videoclipurilor autentice comparativ cu cele false.
Această metodă poate fi extrem de eficientă, dar necesită o cantitate mare de date de antrenament.
Aplicații și servicii comerciale: Există și instrumente comerciale disponibile sau soluții oferite de start-
up-uri specializate în detectarea Deepfake-urilor, care combină tehnologiile menționate mai sus pentru a
oferi soluții la cheie pentru organizații sau indivizi.
Provocări și limitări: Este important de menționat că, pe măsură ce tehnologiile de detectare devin mai
sofisticate, la fel devin și metodele de creare a Deepfake-urilor. Asta înseamnă că este nevoie de o
actualizare și îmbunătățire constantă a instrumentelor de detectare. În plus, multe dintre tehnici pot
genera răspunsuri fals pozitive sau fals negative, ceea ce necesită verificări suplimentare și îmbunătățirea
continuă a algoritmilor.
Detectarea Deepfake-urilor este un câmp de bătălie dinamic între creatorii de conținut falsificat și cei
care încearcă să protejeze autenticitatea informațiilor. Continuarea cercetării și dezvoltării în acest
domeniu este crucială pentru a ține pasul cu evoluția rapidă a tehnologiei.
Sfaturi pentru a evita să fii păcălit de Deepfake-uri
Evitarea înșelăciunii prin Deepfake necesită o combinație de scepticism sănătos, atenție la detalii și
utilizarea unor instrumente de verificare. Iată câteva sfaturi utile:
• Nu crede tot ce vezi online! Internetul este o sursă vastă de informații, dar nu toate sunt veridice. Este
important să dezvolți un scepticism sănătos și să analizezi cu atenție orice conținut video sau foto
înainte de a-l accepta ca fiind real.
• Caută semne de manipulare: Deepfake-urile pot fi foarte sofisticate, dar adesea pot fi identificate prin
anumite indicii. Fii atent la discrepanțe de iluminare, erori de aliniere, nere-
guli ale pielii sau probleme de sincronizare a buzelor cu sunetul.
• Verifică sursa: De unde provine videoclipul sau imaginea? Este distribuit pe o
platformă de încredere? Caută confirmarea informației din surse credibile sau
direct de la entitățile sau persoanele implicate.
• Folosește instrumente de verificare: Există numeroase organizații și instru-
mente online care te pot ajuta să verifici dacă o informație este reală. Utili-
zează-le pentru a cerceta autenticitatea conținutului suspect.
• Nu te baza pe o singură sursă: Caută confirmare din mai multe surse credibile.
Un singur videoclip sau imagine nu este suficient pentru a verifica o informație.
• Învață despre Deepfake-uri: Cu cât înțelegi mai bine cum funcționează această tehnologie, cu atât vei
fi mai capabil să identifici falsurile. Există multe resurse online care explică principiile Deepfake-urilor
și metodele de detectare.
Prin aplicarea acestor sfaturi, poți reduce riscul de a fi înșelat de conținutul Deepfake și poți contribui la
promovarea unei culturi a verificării și a responsabilității în mediul online.
Educația în combaterea Deepfake-urilor
Informarea publicului: Educația este esențială pentru a crește gradul de conștientizare cu privire la
Deepfake-uri și la pericolele asociate. Oamenii trebuie informați despre modul în care Deepfake-urile pot
fi utilizate pentru manipulare și dezinformare, pentru a reduce impactul lor negativ.
Dezvoltarea gândirii critice: Programele educaționale pot contribui la dezvoltarea abilităților de gândire
critică. Oamenii trebuie învățați cum să evalueze sursele de informații, să recunoască semnele de conținut
falsificat și să verifice informațiile înainte de a le distribui.
Securitate digitală: Educația privind securitatea online și confidențialitatea poate ajuta indivizii să își
protejeze mai bine propriile informații și să fie conștienți de potențialele abuzuri ale tehnologiei Deepfake.

_TLP:CLEAR_ Pagina 6 / 8
Alfabetizare media: Înțelegerea modului în care funcționează mass-media și a tehnicilor de producție a
conținutului poate ajuta publicul să discearnă mai bine Deepfake-urile.
Combinația dintre eforturi tehnologice și programe educaționale solide poate construi un răspuns puternic
și eficient la provocările prezentate de Deepfake-uri. Această abordare duală este esențială pentru a
minimiza impactul negativ al Deepfake-urilor asupra societății, politicilor și vieții private a cetățenilor.
Ce să faci dacă ești victima unui Deepfake
Dacă te afli în situația neplăcută de a fi victima unui Deepfake, este important să acționezi rapid și eficient
pentru a minimiza daunele. Iată câțiva pași pe care îi poți urma:
• Documentează abuzul: Salvează copii ale conținutului Deepfake, inclusiv
URL-uri, capturi de ecran, sau orice altă formă de dovezi care ar putea fi
relevante. Acest lucru este esențial pentru orice demersuri legale sau
raportări ulterioare.
• Raportează conținutul: Majoritatea platformelor de socializare și site-
urilor web au politici stricte împotriva Deepfake-urilor și a conținutului
manipulat. Utilizatorii pot semnala cu ușurință conținutul suspect folosind
funcția de raportare integrată a platformei.
• Contactează autoritățile: În cazuri grave, unde conținutul Deepfake
încalcă legile privind defăimarea, hărțuirea sau distribuția de materiale
pornografice fără consimțământ, poate fi necesar să contactezi
autoritățile locale sau alte organisme de aplicare a legii.
• Solicită ajutor juridic: Consultă un avocat pentru a evalua opțiunile legale disponibile pentru tine.
Aceasta poate include acțiuni legale împotriva celor care au creat sau distribuit conținutul Deepfake.
• Folosește servicii de gestionare a reputației online: Există companii specializate în îmbunătățirea
prezenței online și în eliminarea sau diminuarea impactului conținutului negativ. Aceste servicii pot fi
utile pentru a-ți proteja imaginea pe termen lung.
• Comunică cu transparență: Dacă Deepfake-ul are potențialul de a-ți afecta cariera sau relațiile
personale, ia în considerare să vorbești deschis despre situație cu angajatorul, colegii sau persoanele
apropiate. Oferirea contextului și a versiunii tale poate ajuta la diminuarea impactului negativ.
• Protejează-ți informațiile personale: În urma unui incident Deepfake, este important să fii mai precaut
în ceea ce privește securitatea online. Verifică setările de confidențialitate pe rețelele sociale, schimbă
parolele și monitorizează activitatea conturilor tale pentru semne de acces neautorizat.
• Suport emoțional: Trauma psihologică suferită de victima unui Deepfake poate fi semnificativă. Nu
ezita să cauți sprijin din partea prietenilor, familiei sau a profesioniștilor în sănătate mintală.
• Educație și conștientizare: Ajută la creșterea gradului de conștientizare despre pericolele Deepfake-
urilor prin împărtășirea experienței tale, dacă te simți confortabil. Acest lucru poate ajuta la informarea
și protejarea altora.
Acționând prompt și decisiv, poți trece prin provocările asociate cu a fi victima unui Deepfake și poți
începe procesul de recuperare și protecție a reputației tale.
Regulament privind inteligența artificială adoptată de Parlamentul European
Parlamentul European a adoptat Regulamentul COM/2021/206 privind IA care are ca scop protejarea
drepturilor fundamentale, democrației, statului de drept și sustenabilității mediului. Regulamentul
clasifică sistemele IA în funcție de nivelul de risc (inacceptabil, ridicat, limitat, minim) și interzice
anumite utilizări, cum ar fi manipularea cognitiv-comportamentală a persoanelor vulnerabile, sistemele
de credit social și identificarea biometrică în timp real (cu excepții). De asemenea, regulamentul impune
obligații de transparență, trasabilitate, evaluare a riscurilor și asigurare a calității.
Regulamentul abordează și problema Deepfake-urilor, definindu-le ca tehnici de manipulare a imaginilor
sau videoclipurilor pentru a crea iluzia că un subiect spune sau întreprinde o acțiune ce nu a avut loc sau
nu în contextul prezentat. Deepfake-urile cu risc inacceptabil sunt interzise, cele cu risc ridicat trebuie să

_TLP:CLEAR_ Pagina 7 / 8
fie identificabile ca atare, iar cele cu risc limitat și minim nu sunt supuse unor restricții specifice. Furnizorii
de Deepfake-uri trebuie să se asigure că utilizatorii sunt conștienți de natura artificială a conținutului, iar
platformele online trebuie să ia măsuri pentru a preveni răspândirea Deepfake-urilor dăunătoare vii.
Concluzii
Pe măsură ce tehnologia avansează, capacitatea de a crea Deepfake-uri devine tot mai accesibilă, sporind
necesitatea unor strategii eficiente de detecție și conștientizare.
Dezvoltarea algoritmilor de detectare: Cercetătorii lucrează la dezvoltarea de algoritmi bazați pe IA care
pot identifica Deepfake-urile prin analizarea detaliilor care sunt dificil de detectat de ochiul uman, cum
ar fi anomalii în clipire sau în mișcarea naturală a pielii.
Tehnici de autentificare a conținutului: Tehnologii precum blockchain și watermarking digital
(filigranare) pot ajuta la stabilirea originii și autenticității conținutului, oferind o metodă de verificare a
surselor video și audio originale.
Instrumente de verificare la scară: Platformele de social media și companiile tehnologice dezvoltă
instrumente automatizate pentru a scana și a elimina Deepfake-urile de pe site-urile lor, contribuind la
reducerea răspândirii dezinformării.
Parteneriate între industrii: Colaborarea între sectorul tehnologic, guvern și organizațiile de cercetare
poate accelera dezvoltarea și implementarea soluțiilor de detectare a Deepfake-urilor.

Acest material a fost realizat de următorii experți ai DNSC:

Daniel Abotezătoaei, Dan Andrieș, Mihaela Dan, Alex Leoreanu,
Irina Nemoianu, Cristian Nistor, Vlad Drăguș, Mihai Rotariu

Această publicație este licențiată sub CC-BY 4.0: "Cu excepția cazului în care se specifică altfel, reutilizarea acestui
document este autorizată sub licența Creative Commons Attribution 4.0 International (CC BY 4.0) (https://creative-
commons.org/licenses/by/4.0/). Aceasta înseamnă că reutilizarea este permisă, cu condiția menționării corespun-
zătoare și a indicării oricăror modificări".

TLP:CLEAR se poate folosi atunci când informațiile prezintă un risc minim de utilizare abuzivă, în conformitate cu normele și
procedurile aplicabile pentru publicare. Sub rezerva regulilor standard ale drepturilor de autor, informațiile TLP:CLEAR pot fi
partajate fără restricții.
Informațiile și opiniile conținute în acest document sunt furnizate "ca atare" și fără garanții. Referirea din prezentul document la
orice produse, procese sau servicii comerciale specifice prin denumire comercială, marcă comercială, producător sau în alt mod
nu constituie sau implică aprobarea, recomandarea sau favorizarea acestora de către Directoratul Național de Securitate Ciber-
netică (DNSC), iar aceste îndrumări nu vor fi utilizate în scopuri publicitare sau de aprobare a produselor.

i https://www.digi24.ro/stiri/cum-a-fost-lasata-o-pensionara-fara-52-de-mii-de-lei-am-intrat-pe-youtube-unde-ascult-rugaciuni-si-am-dat-de-un-videoclip-cu-tiriac-2714127
ii https://consumer.ftc.gov/consumer-alerts/2023/03/scammers-use-ai-enhance-their-family-emergency-schemes
iii https://www.forbes.com/sites/jessedamiani/2019/09/03/a-voice-deepfake-was-used-to-scam-a-ceo-out-of-243000/
iv https://detectfakes.kellogg.northwestern.edu/
v https://www.intel.com/content/www/us/en/newsroom/news/intel-introduces-real-time-deepfake-detector.html
vi https://www.cert.hr/wp-content/uploads/2023/12/zloupotreba_umjetne_inteligencije.pdf
vii https://www.europarl.europa.eu/news/ro/press-room/20240308IPR19015/legea-privind-inteligenta-artificiala-pe-adopta-un-act-de-referinta

_TLP:CLEAR_ Pagina 8 / 8
Shallow Fakes
Source: IE

Recently, the viral video featuring US Vice President Kamala Harris making irrational remarks points
toward the threat society faces regarding shallow fakes.

Shallow fakes or cheap fakes are pictures, videos, and voice clips created without the help
of Artificial Intelligence (AI) technology but by either editing or by using other simple software
tools.
Deepfakes are synthetic media that use AI to manipulate or generate visual and audio
content, usually intending to deceive or mislead someone.
These are created using a technique called generative adversarial networks (GANs),
which involve two competing neural networks: a generator and a discriminator.
Global Risk Report 2024, by the World Economic Forum (WEF), also
highlighted AI-powered misinformation and Disinformation as the most severe
risks in the next 2 years.

//

Read More: Deepfakes, Global Risks Report 2024: WEF

 PDF Refernece URL: https://www.drishtiias.com/printpdf/shallow-fakes

Powered by TCPDF (www.tcpdf.org)

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/384064482

Cheapfakes

Chapter · September 2024

CITATIONS READS

0 82

1 author:

Dren Gerguri
University of Prishtina
27 PUBLICATIONS 43 CITATIONS

SEE PROFILE

All content following this page was uploaded by Dren Gerguri on 16 September 2024.

The user has requested enhancement of the downloaded file.

Cheapfakes

Dren Gërguri
University of Prishtina “Hasan Prishtina”, Kosovo, [email protected]

Accepted for publication in:Nai, A., Grömping, M., Wirz, D. (Eds.). Elgar Encyclopaedia
of Political Communication.Edward Elgar Publishing.

Abstract
Cheapfakes are fake audiovisual content. These manipulations are carried out through free
programs, which is why the phrases cheap and fake are used. One of the most common and
successful types of visual disinformation is still cheapfakes, and visual recontextualization is
most widespread amongst cheapfake types. Despite their potential to harm and undermine
public discourse, Cheapfakes have received relatively little attention from researchers. This
entry aims to address this knowledge gap by exploring the definition, types, detection, and
implications of Cheapfakes, and also distinguishes it from deepfakes. By examining the ways
in which Cheapfakes can be created, disseminated, and detected, we can better understand
their impact on society. Moreover, the entry recognizes the ongoing academic discussions
and open-ended concerns about the credibility and impact of visual disinformation,
highlighting the necessity of further research to comprehensively address these issues.

Keywords: Cheapfakes; Audiovisual manipulation; Social media; Non-AI manipulation;

Visual disinformation; Detection

1
Cheapfakes are audiovisual manipulations, which are produced without the inclusion of
artificial intelligence (Gërguri, 2022). At a time when the greater use of Deepfakes was
expected, especially in the 2020 Presidential elections in the USA, Cheapfakes received
greater attention during the election campaign because more cheapfakes were circulating
online than Deepfakes (Schick 2020). The main reason why cheapfakes were used more is
related to the ease of production, and the difficulty to detect using technological means (Fazio
2020).
The term ‘cheapfakes’ is a combination of two words, cheap (free) and fake, referring to the
use of free programs to create false audio or visual content. Such manipulations have been
very frequent in recent years. However, during 2024 we may see a rise in the use of deepfake
techniques due to the rapid advancements in generative AI.
This entry aims to explain the concept of Cheapfakes, their differences from Deepfakes, and
their implications for society. The lack of study on Cheapfakes is a significant gap in our
understanding of the impact of this kind of audiovisual manipulations on society, therefore
this entry explores the definition, types, detection, and implications of Cheapfakes,
highlighting their potential to spread disinformation and undermine public debate.

Cheap manipulation

Cheapfaking is non-AI-based manipulation. To make a cheapfake, a pre-existing content of a

particular person or event is needed, whereas a deepfake may be totally generative as it is
based in more advanced technological manipulations and artificial intelligence-based.
Cheapfakes have surged in recent years, as anyone can create them using basic software.
Garimella and Eckles (2020) in their study about visual misinformation in India found that
34% of manipulated images circulated in public WhatsApp groups were cheapfakes. In
another study, which was focused on visual disinformation connected to COVID-19, 24% of
the tested visual disinformation included cheapfakes and recontextualization was the
dominant type (Brennen et al., 2021). The motives to make cheapfakes vary from
entertainment to malicious intentions. Political institutions, businesses, organizations, and
people can all be threatened by audiovisual disinformation, including non-AI manipulation,
such as cheapfakes.
There may be cheapfakes for entertainment, which are harmless to society. These include
memes, which usually have sarcastic and amusing undertones, with no purpose to divert real
events. Other cheapfakes are those manipulations made for financial gain and such cases may

2
be harmless or not, it depends on the content. When a cheapfake is produced for political
purposes, then often there is an intention to harm or attack a leader, a party, or an institution.

Cheapfakes types and detection

The main forms of cheapfakes are: intervention in the context of images, intervention in the
context of videos, video acceleration, audio raising, slowing down videos, and lowering audio
(Paris & Donovan, 2019; Gërguri, 2022). Most prevalent across different platforms are the
intervention in the context of images or videos (Brennen et al., 2021).
Intervention in the context of images is the process of editing a photo's content or context in
order to fabricate a story. For instance, altering an image's background to depict people in a
different setting or adding or deleting things to change the image's meaning. For instance, a
Reuters photo from 19991 showed an Albanian woman entering North Macedonia with other
refugees during the Kosovo war. However, nearly two decades later, a manipulated version of
the image with a changed background emerged, falsely claiming to show a Serbian survivor
of the NATO bombing of Yugoslavia. The manipulated photo was used in a Russian TV
show2. Such examples of attempts to change reality by intervening in photographs are
numerous and cheapfakes of this type are made on various topics and for various purposes,
ranging from manipulating information about the war, e.g. manipulated photos from the
Russian invasion of Ukraine, to sports events or protests during the pandemic (Reuters,
2020).
The other type of cheapfake is the intervention in the context of the video. This technique,
which is similar to picture manipulation, is changing a video's content or context. To deceive
viewers about the events represented, this might involve adding or deleting components,
altering the environment, or modifying the content. Such a case was a cheapfake published in
2021 where the climate change activist Greta Thunberg, an MSNBC interview seemed to be
saying that climate change does not exist3. This is cheapfake of the video cropping type to
take words out of context.

1
Reuters, 30 years of Reuters Pictures: Part one. https://widerimage.reuters.com/story/30-years-of-reuters-
pictures-part-one
2
Centar za geostrateške studije, РИА Новости Рат у Украјини 2014. и у Југославији 1999.,
https://www.youtube.com/watch?v=6OLu4EYqEjk&t=146s
3
Reuters Fact Check, Isolated clip of Greta Thunberg saying ‘climate change does not exist’ is misleading,
https://www.reuters.com/article/idUSL2N2O52US/

3
Video acceleration is the other type of cheapfakes. This involves increasing the speed of a
video to make events appear to unfold more quickly than they did. This can be used to distort
the perception of time and influence the viewer's understanding of a situation. The first case
of cheapfakes of video acceleration occurred in November 2018, in the United States of
America, when CNN White House correspondent Jim Acosta appeared “aggressive” to a
White House worker at a briefing former President Trump's press. In fact, Acosta's reaction is
not unusual and he tells the intern, “Excuse me, ma'am” and continues the question, but the
manipulation of the video and speeding it up makes it appear that the journalist hit her with
his elbow. This was used by the Trump administration to suspend Acosta's accreditation but
was withdrawn after CNN decided to take the case to court.4
A year later, another cheapfake went viral, the manipulation of Nancy Pelosi's video, which
depicts the Democratic politician as drunk5. In Acosta's case, the interference was speeding
up the video, whereas in this case, the contrary has been done, slowing down the video and
creating the impression that Pelosi is drunk on the show. Slowing down a video can be used
to exaggerate and emphasize certain actions or events. Slowing down footage can create a
false sense of drama or make subtle movements appear more significant than they are in
reality.
Different tools could be useful to help identify a cheapfake, such as Google Lens, InVID, etc
and there are studies (Qian, 2023) showing that when people are aware of those tools, they
develop a habit of verifying cheapfakes. Disinformers can disseminate a cheapfake by
changing a video’s or image’s context, therefore it is important to develop search techniques
and tools among journalists, fact-checkers and society in general. To detect a manipulation it
is necessary to have the critical judgement for the exposed media content and different
models orientate everyone to maintain critical skills. One of the models is that of three basic
questions, developed by Gërguri (2022) during the period of the pandemic: 1. Who is the
author? 2. Have you checked the story on multiple sources?, and 3. Is the information
evidence-based and backed by reliable and official sources? This model is settled in
extraordinary circumstances, however is appropriate even in the post-pandemic information
environment. The three basic questions model highlights the importance of knowing the
authors and their credibility, the significance of having multiple sources for information to
create a wider perspective on the information, and the role of supporting sources in the news.

4
Forbes, https://www.forbes.com/sites/laurenaratani/2018/11/08/altered-video-of-cnn-reporter-jim-acosta-
heralds-a-future-filled-with-deep-fakes/?sh=430def513f6c
5
CBS Morning, https://www.youtube.com/watch?v=EfREntgxmDs

4
Cheapfakes implications for society

Cheapfakes can have different effects on society, undermining the public debate or causing
misunderstandings, misconceptions, and the propagation of false narratives. Visual
disinformation, including Cheapfakes may seem convincing (Weikmann & Lecheler, 2023),
therefore they can be used to manipulate public opinion on important issues, particularly in
political contexts (Schick, 2020), but also can be used to blackmail or defame people (Paris &
Donovan, 2019).
Another potential consequence of cheapfakes on individuals and societies is effect on
attitudes and perceptions. People are more exposed to cheapfakes than deepfakes (Diya
2024), while studies shows that cheapfakes can go undetected more often than deepfakes
(Hameleers 2024). This can create confusion, incorrect assumptions about important topics or
increase skepticism towards reliable sources (Gërguri, 2022). Hence, an additional
unanticipated consequence would be that individuals would get so confused by the existence
of visual deception that they would begin to think everything is manipulated and lose trust in
all visuals, even when it is real (Weikmann & Lecheler, 2023).
Visual disinformation, such as cheapfakes, can contribute to social and political polarization
even more than non-visual disinformation (Dan et al., 2021). They could be used to reinforce
preexisting prejudices and opinions, strengthen echo chambers, and so on.
Although researchers have looked at the consequences of various visual disinformation, such
as cheapfakes and deepfakes, there are still some unanswered questions and disagreement
between scholars in some areas. For instance, there is a disagreement on the credibility of
cheapfakes compared to deepfakes. Scholars disagree on the credibility of cheapfakes
compared to deepfakes. Some argue that cheapfakes are most widespread and more credible
(Fazio, 2020) because they may appear more authentic to viewers, while others argue that
people are more likely to believe that deepfakes are more accurate than cheapfakes, and that
people are more inclined to spread deepfakes than cheapfakes (Ahmed & Chua, 2023).
Scholarly discussion continues also about the degree to which individuals distinguish the
credibility of visual disinformation (cheapfakes and deepfakes) and non-visual
disinformation. Hameleers (2024) compared the effects of cheapfakes and deepfakes and
non-visual disinformation in a political context and he concludes that both deepfakes and
cheapfakes was not seen as more believable or with more effect than the same disinformation
presented in textual format. This is the opposite of what another study found, which
concludes that visual disinformation is more believable than text (Hameleers et al., 2020).

5
Therefore, further research and with bigger sample size is needed to fully understand the
differential effects of these two forms of visual disinformation, also in comparison with
textual disinformation.
These types of audiovisual manipulations are a very dangerous virus for journalism too. The
crisis of trust in media and journalists, and the change of information-seeking habits, often
having social media as the main source of information in different countries worldwide
(Newman et al., 2023), has created a suitable terrain for those who aim to impact people's
perceptions and beliefs through cheapfakes. Information providers' credibility is weakened by
the pervasive usage of cheapfakes. People's confidence in institutions, journalism, and even
interpersonal communications may decline as they grow less convinced about the veracity of
media content (Kavanagh & Rich, 2018). Training journalists to detect cheapfakes is
important, but the mission is not fulfilled if the public is not educated about such a
phenomenon. These trends have made the extension of digital media literacy in society much
more important, and there are studies showing that digital media literacy have improved
people's ability to distinguish between true and false news (Guess et al., 2020).

Conclusion

In the digital era, where easily available tools for modifying information are made possible
by technological breakthroughs, cheapfakes constitute a worrying aspect. As we have seen,
these dishonest techniques come in a variety of forms, such as picture and video
manipulation, slowing down footage, and video acceleration. The examples given, which
include image alterations, manipulated videos, and false statements, highlight the wide
variety of cheapfakes that might appear. Cheapfakes are harmful when intended to influence
people’s opinion, although they can be entertaining when utilized for that purpose. People
need to be careful to confirm the legitimacy of the stuff they come upon as technology
develops further. Being aware of the possibility of manipulation and being aware of the many
strategies used enables people to use digital media more responsibly. Cheapfakes will have a
negative influence on public opinion and trust, thus it will be important to stay informed and
modify our media consumption habits as technology advances.
Future research on cheapfakes should investigate the long-term consequences of exposure to
cheapfakes on individuals' trust in institutions and the media, as well as their impact on social
and political polarization. Additionally, it is crucial to expand the scope of research beyond
the political domain, as cheapfakes also exist in other areas such as health, economy, and

6
more. Understanding how social media platforms and other communication channels could
be used for cheapfakes dissemination is also crucial to grasping the full extent of their
negative consequences. Furthermore, exploring how individuals’ process cheapfakes through
systematic and heuristic processing styles can provide valuable insights into developing
effective countermeasures. By examining these aspects, future studies can shed light on the
complex dynamics of cheapfakes and their far-reaching implications.

References

Ahmed, S. & Chua, H. W. (2023). Perception and deception: exploring individual

responses to deepfakes across different modalities. Heliyon, 9(10), e20383 .
https://dx.doi.org/10.1016/j.heliyon.2023.e20383
Brennen, J. S., Simon, F. M., & Nielsen, R. K. (2021). Beyond (mis)representation: Visuals
in COVID-19 misinformation. The International Journal of Press/Politics, 26(1),
277–299. https://doi.org/10.1177/ 1940161220964780
Dan, V., Paris, B., Donovan, J., Hameleers, M., Roozenbeek, J., van der Linden, S., & von
Sikorski, C. (2021). Visual mis- and disinformation, social media, and democracy.
Journalism & Mass Communication Quarterly, 98(3), 641–664. doi:
10.1177/10776990211035395
Fazio, L. (2020). Out-of-context photos are a powerful low-tech form of misinformation. The
Conversation. http://theconversation.com/out-of-context-photos-are-a-powerful-low-
tech-formof-misinformation-129959
Garimella, K., & Eckles, D. (2020). Images and misinformation in political groups: Evidence
from WhatsApp in India. arXiv preprint. https://doi.org/10.48550/arXiv.2005.09784
Gërguri, D. (2022). Fake News, information disorder in the digital age, Prishtina: Dukagjini.
https://www.researchgate.net/publication/375229895_Fake_News_Information_Disor
der_in_the_Digital_Age (english version)
Guess, A. M., Lerner, M., Lyons, B., Montgomery, J. M., Nyhan, B.,Reifler, J., & Sircar, N.
(2020). A digital media literacy intervention increases discernment between
mainstream and false news in the United States and India. Proceedings of the
National Academy of Sciences, 117(27), 15536–15545.
https://doi.org/10.1073/pnas.1920498117
Hameleers, M. (2024). Cheap Versus Deep Manipulation: The Effects of Cheapfakes Versus

7
 Deepfakes in a Political Setting, International Journal of Public Opinion Research,
Volume 36, Issue 1, https://doi.org/10.1093/ijpor/edae004
Hameleers, M., Powell, T. E., Van Der Meer, T. G., & Bos, L. (2020). A picture paints a
thousand lies? The effects and mechanisms of multimodal disinformation and
rebuttals disseminated via social media. Political Communication, 37(2), 281–301.
https://doi.org/10.1080/ 10584609.2019.1674979
Kavanagh, J., & Rich, M. D. (2018). Truth decay: An initial exploration of the diminishing
role of facts and analysis in American public life. Rand Corporation.
Newman, N., Fletcher, R., Eddy, K., Robertson, C.T., & Nielsen, R.K. (2023). Digital News
Report 2023, Reuters Institute. https://static.poder360.com.br/2023/06/Digital-News-
Report-Reuters-2023.pdf
Paris, B., & Donovan, J. (2019). Deepfakes and cheapfakes: the manipulation of audio and
visual evidence. Data & Society Report.https://datasociety.net/library/deepfakes-and-
cheap-fakes/
Qian, S., Shen, C., Zhang, J. (2023). Fighting cheapfakes: using a digital media literacy
intervention to motivate reverse search of out-of-context visual misinformation,
Journal of Computer-Mediated Communication, Volume 28, Issue 1,
https://doi.org/10.1093/jcmc/zmac024
Schick, N. (2020). Don’t underestimate the cheapfake. Technology Review,
https://www.technologyreview.com/2020/12/22/1015442/cheapfakes-morepolitical-
damage-2020-election-than-deepfakes/
Weikmann, T., & Lecheler, S. (2023). Visual disinformation in a digital age: a literature
synthesis and research agenda.

View publication stats

 Tema:
Provocări etice online:
Conținutul deepfake și
integritatea
informațională în era
inteligenței artificiale

„Într-o eră deepfake, «ceea ce vezi»

nu mai este întotdeauna «ceea ce este»”
De ce este important să înțelegem Deepfake

Deepfake, tehnologia prin care se creează imagini și video-uri

false, folosind imaginea persoanelor reale, a devenit o
preocupare majoră pentru toți utilizatorii de internet.

Utilizarea imorală a tehnologiei Deepfake, pentru crearea de

materiale compromițătoare sau hărțuirea online, ridică
probleme serioase de etică și respectare a drepturilor
individuale.
Ce este un un deepfake și cum poți recunoaște tehnologia
care poate face pe oricine să spună și să facă orice
DEEPFAKE
estompează granița dintre ceea ce este REAL și ceea ce este FALS.

În doar câteva minute, oricine poate crea un deepfake, îl poate posta

online și poate înșela nenumărate persoane.

Termenul “deepfake” provine de la:

“deep learning” (învățare profundă)
“fake” (fals),
și este utilizat pentru a descrie conținutul video sau audio
generat artificial, care prezintă persoane reale în situații false sau
în contexte în care acestea nu au fost implicate în realitate
Tipuri de conținut deepfake și aplicațiile sale

 Deepfake video: manipularea

imaginilor
 Deepfake audio: imitarea vocii
Deepfake textual: generarea
automată de conținut scris
Deepfake imagistic (fotografic):
crearea de fotografii false
Deepfake biometric: manipularea
datelor de recunoaștere facială
Deepfake în Social Media
Deepfake-urile pot fi clasificate în mai multe categorii, în funcție de
tehnologia utilizată și scopul lor. Iată principalele tipuri de deepfake:

Deepfake Video
• Face Swap (Schimbarea feței) –
Înlocuirea feței unei persoane cu cea a altei
persoane într-un videoclip. Exemple includ Aplicații deepfake Video în afara Social Media:
utilizarea AI pentru a pune fața unei
celebrități pe corpul altei persoane. 1. Industria filmului și divertismentului – pentru
• Lip Sync (Sincronizarea buzelor) – dublarea actorilor, recrearea digitală a persoanelor
Manipularea buzelor unei persoane într-un decedate sau rejuvenarea personajelor (ex.:
videoclip astfel încât să pară că spune tehnologia folosită în Star Wars pentru Leia și Tarkin).
altceva decât în realitate.
• Face Reenactment (Imitarea 2. Publicitate și marketing – în crearea
expresiilor faciale) – Modificarea expresiilor reclamelor cu ambasadori digitali sau portretizarea
faciale ale unei persoane într-un videoclip unor personalități fără filmări reale.
pentru a le schimba reacțiile emoționale.
3. Escrocherii financiare și fraude – Deepfake-
urile video sunt utilizate pentru a imita directori de
companii sau personalități publice în atacuri
 Deepfake Audio - imitarea vocii unei persoane prin IA

Clonarea vocii – Utilizarea

IA pentru a sintetiza vocea
unei persoane și a genera
fraze pe care aceasta nu le-a
rostit niciodată.

 Voice Transfer (Transferul

vocii) – Convertirea unei
înregistrări audio într-o voce
diferită, păstrând intonația și
emoția inițială.

 Deepfake voice phishing

(vishing), clonarea vocală în
filme, podcasturi false
 Deepfake Text
generarea de texte false sau manipulate prin tehnologia IA, utilizate pentru a induce în
eroare, pentru manipularea opiniei publice sau pentru alte scopuri malițioase

• Generarea automată de texte false –

Crearea de articole, postări pe rețele sociale sau
mesaje imitative pentru dezinformare.
• Chatbots avansați – AI care poate conversa
în stilul unei anumite persoane, imitând tonul și
stilul acesteia.
 Imitarea unui discurs sau a unui email oficial –
deepfake text poate fi folosit pentru a crea
mesaje false, care par a veni din partea unui
oficial guvernamental sau a unei companii
renumite. De exemplu, un atacator ar putea crea
un email care pare să provină de la o autoritate
importantă (cum ar fi un ministru sau un
director executiv), pentru a manipula acțiunile
cuiva sau pentru a obține informații sensibile.
 ...Fraudele deepfake, identităţile sintetice şi escrocheriile
bazate pe Inteligenţă Artificială fac furtul de identitate mai
greu de detectat şi prevenit...
În analiza Deepfakes in the global election year of 2024: A
weapon of mass deception? februarie 2024, evidențiază impactul
deepfake-urilor în contextul alegerilor globale.

Expertul ESET evidențiază în mod special riscuri sporite privind

utilizarea tot mai frecventă a imaginilor, videoclipurilor și clipurilor
audio fabricate pentru a răspândi dezinformări.
expert în securitate cibernetică,
ESET
Blogul de securitate ESET este un
Într-un alt studiu „How AI-driven identity fraud is causing havoc",
portal de editoriale, știri despre 2025, Muncaster analizează modul în care inteligența artificială
securitate, puncte de vedere, predicții este utilizată pentru a facilita fraudele de identitate prin fraudele
anuale privind riscurile și direcțiile de
evoluție în atacurile cibernetice (de ex. de tip deepfake (crearea de conținut audio și video falsificat),
deepfake, AI-driven threats, fraude de identități sintetice (combinarea informațiilor reale și fictive pentru
identitate).
a crea noi identități) și escrocherii alimentate de IA.
 ... Aceasta se dezvoltă mult mai rapid
decât am crezut. În curând, va ajunge
în punctul în care nu vom mai putea
detecta efectiv [deepfake-urile], așa
că va trebui să căutăm alte tipuri de
soluții...
Hao Li, Pionier în detectarea
deepfake-urilor
„Suntem deja în punctul în
care nu poți face diferența
între deepfake-uri și
lucrurile reale,” Hao Li,
Universitatea din California
de Sud
 Ce este deepfake-ul
DEEPFAKE este un termen folosit pentru producerea, manipularea și modificarea
artificială a datelor în mod deliberat prin utilizarea algoritmilor de inteligență artificială.
Scopul acestor produse este inducerea în eroare a oamenilor sau schimbarea unui sens
original și crearea de imagini cu evenimente false.

Potrivit definiției Directoratului Naţional de Securitate Cibernetică

(România), deepfake-ul este o formă de manipulare digitală care
utilizează tehnici avansate de inteligență artificială pentru a crea
conținut de tip imagini sau audio-video falsificat.

Dan Cîmpean, directorul DNSC „Acest

fenomen poate avea consecințe
semnificative asupra societății, inclusiv
impactul asupra securității și a încrederii
publicului în informațiile din mediul online.”
 Conform Agenției Uniunii Europene pentru
Securitatea Cibernetică (ENISA), deepfake-urile se
referă la manipulări digitale ale înregistrărilor
video, audio sau ale imaginilor, realizate cu
ajutorul inteligenței artificiale sau a altor programe
specializate.

1: Blockchain, Deepfakes, and

Cybercrime
Deepfake-urile evoluează mult mai rapid decât abilitatea noastră de a le identifica

Un studiu realizat de Christopher Doss, publicat în 2023 în revista Scientific Reports,

a arătat că doar 46% dintre adulți pot diferenția un videoclip real de unul deepfake.

Cercetarea a implicat mai multe grupuri, inclusiv elevi de liceu, profesori, directori de
școli și studenți universitari, pentru a evalua capacitatea acestora de a distinge între
videoclipuri autentice și deepfakes.
Rezultatele au arătat că între 27% și 50% dintre participanți nu au reușit să
identifice corect autenticitatea videoclipurilor, indicând o vulnerabilitate
semnificativă la deepfakes în diverse segmente ale populației.
https://www.nature.com/articles/s41598-023-39944-3
Un sondaj realizat în 2024 de
compania Regula arată că
pierderile financiare provocate de
fraudele cu deepfake-uri variază
considerabil de la o țară la alta: Pierderi financiare legate de deepfake: Țări

 Mexic: înregistrează cele mai

mari pierderi medii, de
aproximativ 627.000 USD
 Singapore: se situează pe locul
al doilea, cu 577.000 USD.

 Statele Unite ale Americii:

raportate pierderi medii de
438.000 USD.
 Germania: pierderi de 394.000
USD.

 Emiratele Arabe Unite (UAE):

aproximativ 379.000 USD.
 Apariția media sintetică în era deepfake:
provocări etice în manipularea informațiilor

 Deepfake-urile reprezintă, de fapt, un subset al categoriei generale de „media

sintetică” sau „conținut sintetic”.

 Mai multe studii definesc MEDIA SINTETICĂ drept orice tip de conținut creat sau
modificat cu ajutorul inteligenței artificiale sau al învățării automate (AI/ML).

 Misreport: Media sintetică este un termen general care se referă la orice tip de
conținut media generat sau modificat cu ajutorul inteligenței artificiale.

 Dintr-o perspectivă practică, MEDIA SINTETICĂ este, în general, definită ca orice

tip de conținut creat pe cale digitală sau artificială (de exemplu, persoane generate
pe computer) ori conținut care a fost modificat sau manipulat prin mijloace
tehnologice, fie ele analogice sau digitale.
 ChatGPT și etica generării de conținut sintetic
ChatGPT, un subiect foarte controversat din mai multe perspective, reprezintă un model de limbaj
bazat pe inteligență artificială dezvoltat de OpenAI, poate genera conținut sintetic într-o varietate de
contexte. Acest instrument este capabil să creeze articole, postări pe rețele sociale, conversații,
recenzii și chiar narațiuni complet fabricate, dar care pot părea autentice datorită modului în care
sunt construite.

ChatGPT Plus are peste 12 mil. de abonați plătitori la 20 USD pe lună, față de cele 10 milioane de abonați în 2024.
Retenția abonaților este remarcabil de puternică, 89% dintre utilizatori continuându-și abonamentul după un
trimestru și 74% rămânând abonați după trei trimestre.
 Acoperire globală:

ChatGPT este disponibil în 161 de țări.

Interdicții:
ChatGPT este interzis în 15 țări, inclusiv Italia
(prima care a interzis), China, Iran și Rusia.

Alte țări care au acces restricționat la

ChatGPT includ Cuba, Coreea de Nord și Siria.

Statele Unite sunt lider în adoptarea

ChatGPT = 77,2 de mil. de utilizatori
activi lunar, reprezentând
aproximativ 19,0% din baza globală
de utilizatori ChatGPT.
 Statistici ChatGPT 2025:

 ChatGPT are pete 400 de milioane de utilizatori activi săptămânal în

întreaga lume

 Este o creștere de 33% de la 300 de milioane la sfârșitul anului 2024.

 În doar două luni (dec 2024 – februarie 2025), 100 de milioane de noi
utilizatori,
 marcând cea mai rapidă creștere de până acum.

 Platforma vede în jur de 122,58 milioane de utilizatori zilnic.

 În ianuarie 2025, ChatGPT a înregistrat 3,8 miliarde de vizite.
 În fiecare zi, ChatGPT procesează peste 1 miliard de interogări.
 OpenAI este pe cale să atingă venituri de 11 miliarde de dolari în 2025.
DemandSage, este o companie fondată în 2019, cu sediul în Boston, SUA.
Furnizează și date statistice actualizate și analize detaliate într-o varietate de domenii.

Pe lângă soluțiile de raportare și analiză a datelor, DemandSage publică articole și

rapoarte cuprinzătoare despre subiecte precum:

Inteligența artificială: Statistici despre adoptarea și creșterea AI.

​Comerț electronic: Date despre vânzările globale și tendințele pieței e-commerce.

​Social media: Informații despre numărul de utilizatori și ratele de creștere ale
platformelor sociale.
Big Data: Statistici privind volumul și impactul datelor mari în diverse industrii.

https://www.demandsage.com
https://www.demandsage.com/chatgpt-statistics/
 Mai mult decât „deepfakes”- „Media sintetică” și dezinformare
MEDIA SINTETICĂ devine o problemă atunci când este folosită pentru a
genera conținut ce poate induce în eroare publicul, fiind folosită pentru a
genera conținut ce dezinformează.

INTERPOL: În raportul "Beyond Illusions

(Dincolo de Iluzii)" (2024), Interpol definește
media sintetică ca fiind „media generată sau
manipulată utilizând inteligența artificială”,
respectiv include imagini, sunete, voci, texte
și chiar caracteristici biometrice care pot fi
manipulate sau clonate
 În raportul „Dincolo de Iluzii”, INTERPOL analizează impactul media sintetice, precum
deepfake-urile, asupra securității publice și a integrității informațiilor.
 Raportul subliniază modul în care aceste tehnologii pot fi utilizate pentru a crea conținut
fals, având implicații grave în diverse domenii, de la fraude financiare la manipularea
opiniei publice.

În urma acestui raport, INTERPOL a emis o Notificare Violetă (195 state)

Scopul notificării - informarea și avertizarea autorităților naționale despre

amenințările reprezentate de media sintetice, tehnologiile utilizate și modul
în care acestea pot fi identificate și contracarate.
În raportul „Dincolo de Iluzii”, INTERPOL analizează diverse tipuri de media
sintetice și tehnologiile asociate acestora. Printre cele detaliate se numără:

1. Deepfakes: Acestea sunt create prin utilizarea algoritmilor avansați

de inteligență artificială, în special a Rețelelor Adversariale Generative
(GANs);
2. Modele de limbaj AI: Tehnologii precum GPT-3 și ChatGPT;
3. Sisteme de sinteză vocală: Tehnologiile de sinteză vocală.

INTERPOL a mai emis și alte tipuri de notificări, precum cele Albastre și Roșii,
care se concentrează pe identificarea persoanelor și localizarea infractorilor
internaționali. Adăugarea Notificării Violetă reflectă adaptarea continuă a
organizației la noile provocări tehnologice...
EUROPOL: În documentul
Confruntarea cu Realitatea?
Aplicarea Legii și Provocarea
Deepfake-urilor” (2023),
Europol menționează că media
sintetică se referă la "media
generată sau manipulată
folosind inteligența artificială",
subliniind utilizările sale atât în
gaming, cât și în îmbunătățirea
serviciilor sau a calității vieții.
În Raportul Europol privind Amenințările Grave și
Criminalitatea Organizată în UE 2025, un document de
referință elaborat la fiecare patru ani, bazat pe datele
colectate de la forțele de poliție din întreaga Europă.
Acest raport va juca un rol crucial în orientarea politicilor
de securitate ale UE în următorii ani.

Catherine De Bolle, directorul executiv al Europol, a

subliniat că „Prin crearea unor medii sintetice extrem de
realiste, infractorii pot înșela victimele, se pot da drept alte
persoane și pot șantaja ținte importante”. Tehnologii precum
deepfake-urile video și clonarea vocală AI au amplificat riscul
de fraudă, extorcare și furt de identitate. Atacurile susținute de
AI devin tot mai precise și devastatoare, iar multe dintre
acestea au atât un scop infracțional, cât și unul geopolitic,
transformându-se într-o adevărată cursă a înarmării digitale,
cu ținte precum guvernele, afaceri și cetățeni”.
 (Rețeaua Europeană a Serviciilor de
Tehnologie pentru Aplicarea Legii, un
consorțiu de 29 de membri din întreaga
Europă, dedicat abordării priorităților
comune în aplicarea legii și a
tehnologiilor și capabilităților adecvate)

ENLETS definește media sintetică drept "media generată sau

manipulată cu inteligența artificială (IA)", incluzând imagini,
sunete, voci și texte. fețe sau chiar oameni în mișcare.

...Dar nu se limitează la asta, fiecare caracteristică biometrică poate fi

manipulată sau clonată sintetic, la fel ca orice semnal care este pus în
aplicațiile de detectare și luarea deciziilor automate, de fapt, toate
informațiile și semnalele digitale .
https://enlets.eu/wp-content/uploads/2021/11/Final-Synthetic-Reality-
Deep-fakes-Impact-on-Police-Work-04.11.21.pdf
 CENTRUL PENTRU ȘTIRI, TEHNOLOGIE ȘI INOVARE, un centru
independent de cercetare politică globală, încearcă să încurajeze
mass-media independentă și durabilă, să mențină un internet
deschis și să încurajeze conversații informate privind politicile
publice.

CNTI definește media sintetică drept conținut media (audio, imagini, texte
și videoclipuri) creat sau manipulat cu ajutorul inteligenței artificiale (IA).
Experții CNTI subliniază că, Numărul deepfake-urilor online a crescut de
zece ori din 2022 până în 2023. https://sumsub.com/fraud-report-2024/
Experții CNTI subliniază că, deși tehnicile de manipulare a imaginilor există
de peste 150 de ani, deepfake-urile reprezintă un nivel nou de realism
datorită avansurilor în AI, având un impact semnificativ asupra încrederii
publice și integrității informațiilor.
https://innovating.news/article/synthetic-media-deepfakes/
 În loc de concluzii

Într-o eră digitală marcată de deepfake-uri,

protejarea identității noastre devine o
responsabilitate. Așadar, să nu uităm:

Atenție la neatenție în spațiul digital!

Appearance vs. Reality: The Effect of Deep Fake and Shallow Fake on
the Law of Evidence in Common Law Jurisdictions.
1.0. ABSTRACT 2
2.0. INTRODUCTION 2
3.0. THE CONCEPT OF DEEP AND SHALLOW FAKE 3
4.0 PRINCIPLES OF EVIDENCE IN DIFFERENT COMMON LAW JURISDICTIONS 4
4.1. Authenticity 5
4.2. Standard of Proof 5
4.3. Best Evidence 6
5.0. THE EFFECTS OF DEEP AND SHALLOW FAKE IN THE LEGAL EVIDENTIAL SYSTEM 6
6.0. RECOMMENDATIONS 8
7.0. CONCLUSION 9

1
1.0. ABSTRACT

Throughout history, philosophers have argued that things are not always as they appear. With the rise of
Machine Learning, we are faced with the proliferation of highly convincing forgeries known as Deep
Fake and Shallow Fake. Indeed, this trend raises concerns about the consequences as the line between
reality and fabrication becomes increasingly blurred. Unfortunately, the legal system, the precursor of
justice to curb these practices, has now fallen victim to passing off Deep Fakes and Shallow Fakes
evidence in courts.

This paper endeavours to analyse the effect of Deep Fake and Shallow Fake in the evidential litigation
system in common law countries. It also focuses on evidential principles in litigation, analysing Deep
Fake and Shallow Fake technology, discussing their dangers in court, and proposing recommendations
to address this issue.

Key Words: Deep Fake, Shallow Fake, Evidence, Technology, Machine Learning, Artificial Intelligence

2.0. INTRODUCTION

A feeling of certainty or conviction, even one of great conviction, may be misleading _ David Lund
(Philosopher).

In 2017, a seemingly harmless Redditor with a subreddit account1 began a trend that would question
reality as the world sees it. The Redditor, with the account name Deep Fake, started to post
pornographic videos that swapped the faces of porn actors with those of celebrities and well-known
persons.2 This trend became widespread and notoriously practised across different platforms despite
several attempts to shut it down. Shallow Fakes have been around for a while. They are manipulated or
doctored audio-visual representations used for deceptive purposes through simple, intentional editing.3

1
Subreddits are niche communities within Reddit, a social news website forum for stories. These communities have their
rules. Subscribers, and posts, depending on the subject matter and as indicated by the URL structure. <
https://www.shopify.com/ng/blog/how-to-use-reddit#:~:text=Subreddits%20are%20niche%20communities%20within,%2C%
20Controversial%2C%20and%20Top%20submissions. > accessed 21 September 2022
2
‘Deep Fake’ (Dictionary.com, January 2020) < https://www.dictionary.com/e/tech-science/Deep Fake/ > accessed 21
September 2022
3
Kalev Leetaru, ‘The Real Danger Today is Shallow Fakes and Selective Editing not Deep Fakes’ (Forbes, 26 August 2019)
< https://www.google.com/amp/s/www.forbes.com/sites/kalevleetaru/2019/08/26/the-real-danger-today-is-Shallow
Fakes-and-selective-editing-not-deep-fakes/amp/ > accessed 21 September 2022

2
Although this was not the first instance of technological face adjustment techniques, Deep Fake and
Shallow Fake schemes have gone past the malicious use of AI learning for pornography; it has
transcended into various aspects of society and has become an issue of concern for several stakeholders.
With the widespread use of Deep Fakes and Shallow Fakes, experts have posited that audio-visual
technology may be tools of deceit and disinformation that can set the legal system, which has attempted
to incorporate electronic video and audio evidence back for several years.

3.0. THE CONCEPT OF DEEP AND SHALLOW FAKE

Ironically, for a technique hinged on deceit with potential dastardly consequences in the wrong hands, an
average person knows so little about the intricate methods of Deep and Shallow Fakes. This may be why
people have become highly susceptible to them. Deep Fake involves audio-visual manipulation using
machine learning algorithms. The technique leverages Artificial Intelligence to create realistic
impersonation by tweaking the words and actions of video and audio recordings of actual people to
simulate reality.4

Deep Fake's technical workings are pretty simple; it uses the primary technique called the Generative
Adversarial Network (GAN). The GAN contains a generator and a discriminator. The generator creates a
new fake image drawing from the real one and then runs the picture or audio through the discriminator.
The latter checks for similarity with the original image and continues to rate the authenticity of the fake
image while making recommendations to improve the similarity between both pictures. These
recommendations continue until the photos are similar and the discriminator can no longer distinguish
between the actual entry and the fake one.5

On the other end of the spectrum are Shallow Fakes. Unlike Deep Fakes, Shallow Fakes do not require
complex AI and Machine Learning Algorithms. Instead, creators of Shallow Fakes use image and video
editing software to effect minimal or subtle changes to the media concerned.6 There can be very little

4
Bobby Chesney & Danielle Citron, ‘Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security’
(2019) 107:1753 California Law Review < https://bit.ly/2C78jnQ > accessed 18 September 2022
5
Jeffrey Westling, ‘Are Deep Fakes a Shallow Concern? A Critical Analysis of the Likely Societal Reaction to Deep Fakes’
(The 47th Research Conference on Communication, Information and Internet Policy, USA, 25 July 2019)
6
Arnold, ‘What is the Difference between a Deep Fake and Shallow Fake?’ (Deep FakeNow, 21 April 2020) < https://Deep
Fakenow.com/what-is-the-difference-between-a-Deep Fake-and-Shallow Fake/ > accessed 18 September 2022

3
difference between Deep and Shallow Fake save for the techniques involved in causing the changes in
the audio-visual representations.7

Surprisingly, Deep and Shallow Fakes have some legitimate uses; they can be used for works of art,
education, entertainment, business, and, in some instances, investigations.8 Sadly, their malicious
services far outweigh the legitimate ones, and this has caused increasing worries, disinformation, and
widespread panic. Deep Fake and Shallow Fake technologies are getting more accessible and advancing
considerably with more realistic results. These technologies currently deceive people nearly 30% of the
time.9 Moreover, internet accessibility makes sharing this information faster and more far-reaching.10

The detrimental effect of these fakes is felt even more in the legal aspect. They threaten the rights to
privacy, democracy, and other personal rights and may occasion irreversible injustice. Hence, its effects
will be considered in subsequent sections.

4.0 PRINCIPLES OF EVIDENCE IN DIFFERENT COMMON LAW JURISDICTIONS

A party proffers evidence to a suit to prove that the existence of a fact is more or less probable.11 This
proof can take any form - documents, testimony, electronic recordings, and other tangible objects. 12
Every country's legal system has a set of laws or principles that guide its receipt of evidence. These
systems may vary in form and structure, but this paper seeks to give attention to only one: the Common
Law system of Evidence.

Under the common law system of evidence, which still serves as the inspiration for the law of evidence
in many jurisdictions, evidence must fulfil specific rules before admissibility in a court proceeding.
Although several countries have broken the strains of common law and adjusted to an independent

7
Ibid
8
Supra, 3
9
Kyle Wiggers, ‘Carnegie Mellon Researchers Create the Most Convincing Deep Fakes Yet’ (The Machine, 16 August 2018)
<https://venturebeat.com/ai/carnegie-mellon-researchers-create-the-most-convincing-Deep Fakes-yet/ > accessed 20
September 2022
10
Supra, 2
11
‘Evidence’ Cornell Law School Wex < https://www.law.cornell.edu/wex/evidence > accessed 20 September 2022
12
‘Basic Principles and Rules of Law of Evidence’ (Law Corner, 27 July 2021) <
https://lawcorner.in/basic-principles-and-rules-of-law-of-evidence/ > accessed 24 September 2022

4
system of law, there are similarities in rules and principles of evidence such as Authenticity, Standard of
Proof, and Best Evidence.13

4.1. Authenticity

Authenticity is a condition precedent to the admissibility of evidence in the common law system and this
paper's crux. It involves satisfying the court, through sufficient evidence, that a person's claim is as they
say. If a judge is satisfied that there is a possibility that evidence is authentic, no matter how little, s/he is
bound to advance its admissibility.14 While there are rules to ensuring that the evidence (oral and
documentary)15 tendered is authentic, the court may need help concerning audio-visual representation.
This issue is compounded when one considers that, unlike the civil law system, the common law does
not allow inquisitorial actions by the court. The court may need help to evaluate the weight of the
evidence by inquiry, and this option is only available by tactical cross-examination on the part of a
knowledgeable counsel. There is also no leeway to subject such evidence if admitted, to appeals for
factual errors.16

4.2. Standard of Proof

The standard of proof in common law necessitates the party who accuses to establish that the facts they
claim are actual.17 A recognised principle of evidence in common law is that facts admitted need not be
proven.18 The party adducing evidence directs its proof to confirm that the act alleged was committed
beyond all reasonable doubts, at least for a criminal proceeding. This principle begs the question: what
happens if a litigant submits an incriminating Deep Fake video without the knowledge of the court?

13
Caslav Pejovic, ‘Civil Law and Common Law: Two Different Paths Leading to the Same Goal’ (2001) 32 VUWLR pp.
817-841
14
See R v Randall (2003) UKHL 69 @ 20. Rosemary Pattenden, ‘Authenticating Things in English Law: Principles for
Adducing Tangible Evidence in Common Law Jury Trials (2009) 12 The International Journal of Evidence & Proof <
https://ssrn.com/abstract=1704625 > accessed 23 September 2022
15
Documentary Evidence here should be taken to be Written documents as some common law jurisdictions define electronic
audio-visual evidence as documentary, see section 258 of the Evidence Act, 2011 Laws of the Federation of Nigeria 2004
16
Supra, 13
17
Kevin Clermont and Emily Sherwin, ‘A Comparative View of Standards of Proof’ (2002) 50:2 The American Journal of
Comparative Law < https://scholarship.law.cornell.edu/facpub/222 > accessed 25 September 2022
18
Supra, 12

5
Would the court recognise that as proof beyond a reasonable doubt? Indeed, one party is incriminated,
and proof that the evidence was tampered with may not readily come to light.

4.3. Best Evidence

The best evidence rule was directly inspired by antique common law. The principle necessitates that the
original and highest or best evidence is presented to the court. The essence is to present legible and
concrete evidence before the court.19 This rule can be directly traced to 1745 when Lord Harwicke stated
that no evidence was admissible except it was "the best that nature will allow."20 If anything is clear, the
very nature of Deep and Shallow Fake goes against the principles of the evidence above. This clash has
dire consequences for the legal system, which will be analysed below.

5.0. THE EFFECTS OF DEEP AND SHALLOW FAKE IN THE LEGAL EVIDENTIAL
SYSTEM

Previously, the legal system had to rely heavily on witnesses and circumstantial evidence for proof of an
act or omission. Indeed, the go-to proof method was primarily written evidence in the early eighteenth
century.21 The trend of oral and written evidence continued well into the nineteenth and twentieth
centuries and was riddled with confusion, controversy, hearsay, and falsehoods.22 One can only imagine
the ease audio-visual evidence has brought to the judicial system. Various experts have regarded the
paradigm shift to using audio-visual technological evidence as reasonable and necessary for the legal
profession.23

Sadly, while audio-visual technologies have set the steps of justice in the right direction, the danger of
evidential deceit arising from fake audio-visual is becoming quite evident. The legal system has become

19
Osaro Eghobamien , Samuel Ehiwe, and Chike Okpaleke, ‘Revisiting the Best Evidence Rule: Matters Arising’ (Mondaq,
16 August 2022) <
https://www.mondaq.com/nigeria/contracts-and-commercial-law/1221804/revisiting-the-best-evidence-rule-matters-arising#:
~:text=The%20best%20evidence%20rule%20is,the%20truth%20of%20its%20content > accessed 23 September 2022
20
See Omychund v. Baker (1745) 26 ER 15
21
John Langbein, ‘Historical Foundations of the Law of Evidence: A View from the Ryder Sources (1996) 96:1168 Columbia
Law Review.
22
Thomas Gallanis, ‘The Rise of Modern Evidence Law’ (1999) 84 Iowa Law Review p. 513
23
Molly Mullen, ‘A New Reality: Deep Fake Technology and the World Around Us’ (2022) 48:1 Mitchell Hamline Law
Review p.219

6
swamped with various technological evidence and has been left to grapple with receiving and
authenticating them.

Experts argue that the danger of fakes in the court system goes far deeper than passing off phoney
representations. The proliferation of Deep Fake and Shallow Fake has made it relatively easy to create
and forge counterfeit evidence. Moreover, technological advancement has made it somewhat
challenging to spot the differences between a fake and accurate audio-visual representation.

Evidence tampering is one of the foremost concerns regarding the malicious use of fakes in court
systems. The danger becomes evident in the mistrust it breeds; it engenders the mistake of dismissing
what is real and cripples the basis of evidence authenticity. Herein lies the dilemma: a judicial officer
with limited insights into Machine Learning and Artificial Intelligence has to determine whether the
evidence presented is accurate enough to be admitted into the court's records.

The practical effect of phonies on the legal system was seen in a custody case in the United Kingdom,
where one of the parties presented a Deep Fake of the other party threatening and harassing the children.
It took the admission of falsehood by the lawyer of the erring party for the court to order an in-depth
analysis of the recording's metadata.24 Without the vehement denial by the accused party and the
vigilance of his counsel, the court would not have realised the forgery of the evidence.

This incident and others question the authenticity of evidence admitted by the court and the court's
capacity to sift through the originality and authenticity of electronic audio-visual representations. It is
clear that while it is impossible to stop fakes' menace completely, their effects can be limited on the legal
system through sifting and authentication processes. Unfortunately, many court systems have shallow
standards for electronic evidence authentication.25

Conversely, even if authentication processes are initiated, it may cost an arm and leg, one that many
litigation systems do not have stowed away. Perchance the court uses testimonies to validate the

24
Bari Weinberger, ‘Looking Out for Manipulated Deep Fake Evidence in Family Law Cases’ (New Jersey Law Journal, 22
February 2021) < https://www.law.com/njlawjournal/2021/02/22/looking-out-for-manipulated-Deep
Fake-evidence-n-family-law-cases/?slretun=20220523104512 > accessed 23 September 2022
25
George Blum, ‘Authentication of Social Media Records and Communications (2019) 70:1 ALR

7
audio-visual evidence, but it may not make any difference as Deep Fake and Shallow Fake have
implications on oral testimony; a study has shown that falsified videos can induce false testimony.26

Deep and Shallow, fake questions the burden of proof before the court; can an artificial but
undiscernible audio-visual representation presented as evidence before the court be taken as proof
beyond a reasonable doubt? The answer is glaringly negative, and the same would be true if the issue of
best evidence arises.

6.0. RECOMMENDATIONS

Several attempts have been made to develop technologies to detect Deep and Shallow Fakes and combat
their widespread use. Sadly, technology is fluid; the closer we are to discovering an efficient fake
detection technique, the more the existing fake technologies are further developed. The available Deep
Fake detection technologies are expensive and often need to be more efficient. Regardless, the cruel
effect of the Deep and Shallow Fake cannot be ignored, especially in the legal system. Therefore, the
question on everyone's mind is: what is the best way to combat Deep and Shallow Fake representations?

The menace of Deep and Shallow Fakes has been proven to cut across the board; the effects are evident
for individuals, law enforcement agencies, the government, and the legal system. Fake detection
techniques have never caught up with these phonies because of the stakeholders' lack of collaboration
and incentive. Collaboration among stakeholders is the most prominent and reasonable solution.
Collaboration would mean increased interest, funding, and development of detection technologies. The
combined strength of crime labs, private sectors, academic research institutions, public companies, and
all forensic-interested persons will surely advance Deep and Shallow Fake detection technology.27

Authentication methods to affirm the chain of custody and validity of an audio-visual representation is a
great way to begin a new regime of authenticity. Times have changed; by virtue, traditional authentic
measures may not be enough to test the reality of fakes, as these fakes can mislead even a witness. As
mentioned earlier, a solution to top those would be reinforcing authenticity techniques in court,
especially regarding electronic audio-visual evidence.

26
Kimberly Wade, et al, ‘Can Fabricated Evidence Induce False Eyewitness Testimony?’ (2010) Applied Cognitive
Psychology 24:7 < https://onlinelibrary.wiley.com/doi/10.1002/acp.1607 > accessed 22 September 2022
27
Federick Dauer, ‘Law Enforcement in the Era of Deep Fakes’ (Police Chief Online, 29 June 2022) <
https://www.policechiefmagazine.org/law-enforcement-era-Deep Fakes/ > accessed 24 September 2022

8
Lastly, one must uphold the importance of a rigorous regulatory approach to the use and pass-off of
Deep and Shallow Fakes. This approach will include educating the general public and legal officials on
the dangers and recognising mid-level fakes that can be easily discovered. The legislation of different
jurisdictions may also update the authenticity of electronic mechanisms to include an identification or
watermark to distinguish original from counterfeit. The last resolution would be to criminalise the use
and passing of fakes within the court's territory.

7.0. CONCLUSION

The bitter truth is that Deep Fake and Shallow Fakes have come to stay; with every development in AI
and Machine Learning, there is a simultaneous growth of these phonies. One cannot stress the danger it
presents to a rigid, often archaic profession like law, especially the common law system. Regardless, this
author advises that the legal system acknowledges the menace of Deep Fake and Shallow Fake and does
not wait until there are consistently ill-delivered judgments brought about by admitting phoney evidence
produced by these technologies.

9
 Curs: Etică și integritate profesională în online
Specialitatea Producție multimedia, an II
Test II. Conținutul deepfake și integritatea informațională în era
inteligenței artificiale
Subiecte:
1. Provocări etice online: tehnologiile deepfake
- Ce este un un deepfake. Deepfakes – realism artificial periculos
- Cheap fakes (falsuri ieftine) - manipulare simplă, impact mare
- Shallow fakes (falsuri superficiale) – denaturarea prin context
- Diferențele dintre deepfakes -cheap fakes și shallow fakes.

2. Media sintetică în era deepfake: provocări etice în manipularea

informațiilor

!!!. Întrebările testului vor fi elaborate în conformitate cu sursele bibliografice de

referință.
Deepfakes vs Shallow Fakes. https://www.blogs.opengrowth.com/deepfakes-vs-
shallow-fakes-understanding-the-layers-of-digital-deception