Organic or Diffused: Can We Distinguish Human Art from

AI-generated Images?
Anna Yoo Jeong Ha∗ Josephine Passananti∗ Ronik Bhaskar Shawn Shan
University of Chicago University of Chicago University of Chicago University of Chicago
Chicago, IL, USA Chicago, IL, USA Chicago, IL, USA Chicago, IL, USA

Reid Southen Haitao Zheng Ben Y. Zhao

Concept Artist University of Chicago University of Chicago
Detroit, MI, USA Chicago, IL, USA Chicago, IL, USA

Abstract the 2024 ACM SIGSAC Conference on Computer and Communications Security
The advent of generative AI images has completely disrupted the (CCS ’24), October 14–18, 2024, Salt Lake City, UT, USA. ACM, New York, NY,
USA, 15 pages. https://doi.org/10.1145/3658644.3670306
art world. Distinguishing AI generated images from human art is a
challenging problem whose impact is growing over time. A failure
to address this problem allows bad actors to defraud individuals
paying a premium for human art and companies whose stated 1 Introduction
policies forbid AI imagery. It is also critical for content owners to Creative expression through artwork is intrinsic to the human ex-
establish copyright, and for model trainers interested in curating perience. From cave paintings by Neanderthals and Homo sapiens,
training data in order to avoid potential model collapse. to modern abstract masters like Kandinsky, Pollock and Mitchell,
There are several different approaches to distinguishing human human art connects us by evoking shared experiences, trauma, and
art from AI images, including classifiers trained by supervised learn- hopes and dreams.
ing, research tools targeting diffusion models, and identification Yet this might all be changing with the proliferation of output
by professional artists using their knowledge of artistic techniques. from generative image models like Midjourney, DALL-E 3, Stable
In this paper, we seek to understand how well these approaches Diffusion XL (SDXL) and Adobe Firefly. Given prompts as short
can perform against today’s modern generative models in both as a single word, these models can generate glossy images that at
benign and adversarial settings. We curate real human art across a glance, resemble the work of a professional artist or fine pho-
7 styles, generate matching images from 5 generative models, and tographer. As they continue to evolve, it is becoming increasingly
apply 8 detectors (5 automated detectors and 3 different human difficult to distinguish art produced by human creatives and images
groups including 180 crowdworkers, 3800+ professional artists, and produced by generative AI.
13 expert artists experienced at detecting AI). Both Hive and expert Identifying if a piece of art is human-made or AI-generated is
artists do very well, but make mistakes in different ways (Hive critical for a number of reasons. First, individuals and companies
is weaker against adversarial perturbations while Expert artists are often willing to pay a premium for human art over AI content.
produce higher false positives). We believe these weaknesses will Also, companies or creative groups have policies restricting the use
persist, and argue that a combination of human and automated de- of AI-generated imagery in competitions, work product, or ad cam-
tectors provides the best combination of accuracy and robustness. paigns. Yet recent news is littered with examples of fraud, where
AI-generated images are sold as human art to individuals [48, 77]
CCS Concepts and publishers [15, 62], and used in ad campaigns or submitted to
• Security and privacy → Human and societal aspects of se- creative competitions against AI policies [51, 58]. This has resulted
curity and privacy. in numerous controversies [7, 22, 47], retracted ads and publica-
tions [14, 70], and public apologies [45, 62].
Keywords Second, identification of AI images is also a legal and regulatory
AI-generated Art Detection, Automated and Human Detectors issue. Commercial companies want to copyright their creative con-
ACM Reference Format: tent, but the US Copyright Office has ruled that only human created
Anna Yoo Jeong Ha, Josephine Passananti, Ronik Bhaskar, Shawn Shan, Reid artwork (or human contributions to hybrid artwork) can be copy-
Southen, Haitao Zheng, and Ben Y. Zhao. 2024. Organic or Diffused: Can righted [64]. Thus businesses using generative AI might try to pass
We Distinguish Human Art from AI-generated Images?. In Proceedings of off AI images as human art to obtain copyright. Finally, multiple
∗ Both projects have shown evidence that both text and image AI models
authors contributed equally to the paper
will degrade if only trained on output of AI models [2, 10, 63]. Thus
This work is licensed under a Creative Commons Attribution- AI model trainers also need to distinguish AI-generated images
NonCommercial-NoDerivs International 4.0 License. from human art for training purposes.
All of this begs the question, do we have the tools today to
CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. reliably and consistently distinguish AI-generated imagery from
© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0636-3/24/10 human-created art? There are multiple potential solutions. First,
https://doi.org/10.1145/3658644.3670306 human artists are often quite good at recognizing human art, and
CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. Anna Yoo Jeong Ha et al.

Human Art CIVITAI DALL-E 3 MidjourneyV6 Firefly SDXL

Figure 1: Samples of human art and matching images produced by generative AI models. Copyright held by respective artists,
©Kirsty (@kirue_t), ©Nguyen Viet, ©Liam Collod

some experts have demonstrated a consistent ability to detect AI im- • We find that normal, non-artist users are generally unable to tell
ages trying to pass as human art [25, 48, 58]. Alternatively, specific the difference between human art and AI-generated images. Pro-
companies like Hive, Optic and Illuminarty have trained supervised fessional artists are more confident and perform much better on
classifiers to distinguish AI imagery from human art. Online me- average, and expert artists more so.
dia has raised questions on the accuracy of these detectors amid • Supervised classification does surprisingly well, and Hive out-
their growing impact on news media [44]. Finally, recent research performs all detectors (human and ML), and produces zero false
results like DIRE and DE-FAKE [60, 75] suggest specific techniques positives. Unsurprisingly, accuracy seems to correlate with ex-
to recognize images produced by diffusion models, including all of pected training data availability: biggest classifier (Hive) performs
the major generative models today. best; all classifiers perform the worst on Firefly, the newest of the
The goal of our work is to systematically and comprehensively major generative models.
explore the effectiveness of these detectors at distinguishing AI- • Fine tuned models that mimic human artists, and real images
generated images and human art1 . We consider a wide range of modified with AI upscaling posed no significant challenges to
imagery, including 7 distinct art styles, each represented by sam- human or ML detectors.
ples of human art, images generated by each of 5 generative AI • Adversarial perturbations did have significant impact on ML de-
models, AI images painted over by humans (hybrid), and human tectors such as Hive, with feature space perturbations being the
photography enhanced by AI. We also consider a range of “detection most effective.
methods,” including 5 automated tools (3 deployed classifiers Hive, • Expert human artists perform very well in detecting AI-generated
Optic, Illuminarty and 2 research detectors DIRE and DE-FAKE) images, but often attribute mistakes and lower skill by human
and 3 different populations of “human detectors” (crowdsourced artists as evidence of AI images, thus producing false positives.
non-artists, professional artists, and expert artists). Finally, we also • A combined team of human and automated detectors provides the
consider adversarial scenarios, where AI-generated images are aug- best combination of accuracy and robustness.
mented with noise and adversarial perturbations with the intention
of bypassing detection. 2 Background and Related Work
In total, we curated a dataset of 280 real human art images across
We begin by providing background on generative AI image models
7 different styles, and 350 generated AI-based images from genera-
using the diffusion model architecture, and on currently available
tive models using prompts automatically extracted from each of the
automated detectors of generative AI images. We then discuss ex-
human art images. One component of our study tests the efficacy of
isting work related to our study.
automated detectors on these images and their perturbed variants;
and the other part evaluates human-based detection. The latter 2.1 Generative Image Diffusion models
involves 3 separate user studies on: a) 180 crowdworkers on the
Prolific platform, b) 3800+ professional artist volunteers recruited First introduced in 2020, diffusion models quickly replaced GANs
from social media artist groups, and c) 13 expert professional artists as the state-of-the-art models for image synthesis [21]. This was
who have experience identifying generative AI images. quickly followed by extension to text-to-image generation [46, 54].
Our study produces a number of significant findings: Later instances included multiple open-source models from Stable
Diffusion [52, 56, 67–69], and commercial models from Midjourney,
1 Unlikemost prior studies, our focus is not identifying deepfake images from real DALL-E, and Adobe Firefly.
photographs. Instead our focus is on determining the provenance of creative art
imagery, and our inclusion of photography focuses on photographs as art, not as Diffusion models face a growing number of social and ethical
records of real events. concerns. Base models require enormous amounts of training data,
Organic or Diffused: Can We Distinguish Human Art from AI-generated Images? CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA.
often obtained without consent through web-scraping. Midjourney and photorealistic images from Midjourney V5. They also create a
trained their model using data from over 16,000 artists, the vast dataset of roughly two million fake images to train ML detectors.
majority without consent [31]. Stability AI trained on datasets While humans misclassify 37% of images, the best-performing ML
from LAION [56, 59], containing millions of copyrighted works. detector misclassifies 13% of the same evaluation set.
These copyright infringement issues have led to multiple class- Several projects explore robust evaluation and robust training
action lawsuits [17, 35]. Even in smaller volumes, artists are finding techniques to improve detection accuracy. [73] proposes training
their works being used without consent in finetuned models using data augmentation using flipping, blurring, and JPEG compression;
techniques such as LoRA [3]. [65] evaluates detection under perturbations of color contrast, color
As these models continue to improve in quality, many of their saturation, blurring, and pixelation; [4] performs data augmentation
users have attempted to pass AI-generated images as human art. AI- with JPEG compression; and [32] uses an ensemble of detectors
generated images have been used to win art competitions, fooling over the frequency domain to improve detection robustness.
judges of digital art, photography, and book covers [27, 57, 58]. Explainability in Image Identification. Some have explored
Companies that promote human art have found themselves using explainability in detecting AI images. [55] studies distributions
AI content provided to them by third-party vendors [50]. of GAN, diffusion, and real images, showing greater overlap be-
tween diffusion and real distributions than between GANs and
2.2 Automated AI-Image Detectors real distributions. [6] creates a counterfactual, generated dataset to
A number of software and web services offer the ability to detect if CIFAR-10 [37] and uses gradient heatmaps to visualize important
an image is generated by generative AI image models. We group features for detection. [16] performs forensic analysis on the fre-
these detectors into two categories: deployed commercial detectors quency domain distributions of various diffusion and GAN models.
and research-based detectors. AI Images and Art. The abundance of prior work has almost
Research-based detectors. These come from published research entirely focused on detecting deepfakes and photorealistic images,
papers that often offer source code and training/testing datasets, including some very large fake image benchmarks [42, 80]. DE-
sometimes also with pretrained models. While they lack the same FAKE briefly mentions detecting art but only tests on 50 pieces of
public reach and influence as commercial detectors, they offer the human art and 50 AI-generated images. Deepart [74] is an art-based
benefit of transparency in methodology. Two such detectors are dataset composed of a random selection of images selected from
the DIRE detector [75] and DE-FAKE [60]. DIRE, or Diffusion Re- LAION-5B [59], designed as a training dataset for a classifier to
construction Error, aims to exploit the forwards and backwards detect AI-generated art.
diffusion processes to identify images generated by that model. The most related work on this topic was presented recently at
DE-FAKE uses both image-only and multimodal embeddings [53] IEEE S&P 2024 [24]. Where our work focuses entirely on creative
to create a model-agnostic detector [60]. visual art, this prior study covered generative AI detection broadly
Commercial detectors. These detectors are deployed online, across images, audio and text across Internet users in multiple
generally as web services with a tiered pricing model and a web- countries.
based non-transparent (black-box) detection API. They provide
easy access to image classification with minimal computational
3 Methodology
requirements, Among the most popular are Hive AI Detector (Hive), AI-generated images have already become exceptionally good at
Optic AI or Not (Optic), and Illuminarty [30, 33, 49]. All three mimicking human art. Distinguishing these generated images from
services advertise free demo plans via a web UI with high accuracy, human art is critical for individual and institutional consumers, for
and are well covered by popular media [23, 71, 72]. copyright reasons, and for AI models seeking to curate their training
Beyond DIRE and DE-FAKE, other techniques have been pro- datasets. The goal of this study is to understand how feasible this
posed to detect AI-generated images. Diffusion models can embed task is today given recent advances in these generative models, how
invisible watermarks into images during generation [8]. Diffusion and why current detectors make mistakes, and what that portends
watermarks involve manipulating the initial noise vector, creating for the future.
watermarks robust to some perturbations [76, 79]. However, this
approach requires modifying the diffusion model. Finally, other 3.1 Overview: Goals and Challenges
detection methods make use of frequency domain analysis to detect In tackling this multifaceted problem, our goal is to try to explore
AI-generated images as outliers [5]. several broad questions on this topic:
(1) Are there detection methods today, human or automated,
2.3 Related work that can accurately distinguish between human art and AI-
Detecting Deepfake Photographs. While our study focuses generated images? How do artists using their knowledge of
on distinguishing human art from AI-generated images, several art fundamentals fare against semantically-agnostic super-
prior studies have focused on human detection of deepfake pho- vised classification and research tools designed specifically
tos generated by machine learning models. [9] evaluates users’ to detect diffusion model output?
ability to detect deepfakes of human faces using StyleGAN2 [36], (2) What are limitations of current detectors, and why do they
and finds that human participants have below 65% accuracy in make mistakes?
all experiments, even when taught how to recognize deepfakes. (3) How well do detectors perform under adversarial conditions,
Similarly, [42] evaluates human detection on a set of real photos i.e. against images altered to avoid identification?
 CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. Anna Yoo Jeong Ha et al.

(4) Are there fundamental trends in performance of detection We evaluate both automated detectors and human detection
approaches, and what are implications as models continue on the same core test dataset of images (280 human art pieces,
to evolve? 350 AI images, 40 hybrid images), described in more detail in Sec-
As the first research study to perform a comprehensive analysis tion 4. However, we also test automated detectors against a variety
of classifying human art and AI-generated images, our most sig- of adversarial perturbations including Gaussian noise, JPEG com-
nificant challenge is how to capture the numerous dimensions of pression, adversarial perturbations, and the Glaze style mimicry
this problem. Most specifically, we consider and incorporate five protection tool [61].
different dimensions into our study. We summarize these here and
present further details as we describe our experimental methodol- 3.3 Evaluating Human Detection: User Studies
ogy in the remainder of this section.
Recent events have shown human artists to be exceptionally suc-
• Art Styles. Generative AI image models have a wide range of cessful at identifying AI-generated imagery masquerading as hu-
success when mimicking different styles of art. Therefore, our man art [25, 48, 58]. Instead of looking for statistical properties of
evaluation must cover a wide range of art styles, from anime to images, human artists look for inconsistencies in artistic technique,
sketches to fine photography. flaws in logic/composition, and other domain-specific properties
• Sources/Types of AI-Generated Images. Different AI models that diffusion models do not understand.
vary in their ability to mimic human art. Thus we must consider Our study evaluates how well skilled artists can use their un-
a representative set of current diffusion models, as well as more derstanding of art to detect AI-generated images, by performing
unorthodox image types such as hybrid (AI-generated images separate user studies for 3 separate user populations.
painted/altered by humans) and upscaled (human-generated pho- • Baseline Participants. We recruited 180 crowdworkers through the
tography expanded in resolution using AI models). Prolific online crowdsourcing platform (177 completed and passed
• Range of Automated AI Image Detectors. We include results attention checks). Participants were compensated $2/10min and
of the most popular available automated detectors (Hive, Optic, this group took on average 8 minutes to complete. This group
Illuminarty) as well as research prototypes (DIRE, DE-FAKE). included no full-time professional artists and 7 part-time artists.
• Range of Human AI Image Detectors. Humans will vary sig- • Professional Artist Volunteers. We asked for artist volunteers on
nificantly in their ability to identify human art vs AI images, de- social media to participate. Of more than 4000 who responded,
pending on their knowledge and experience in producing art. 3803 completed the survey and passed all attention checks.
We consider three user groups: regular users (non-artists), pro- • Expert Participants. We recruited 13 high-profile professional
fessional artists, and expert artists experienced in identifying AI artists known by members of the research team to have experience
images. identifying AI imagery. These expert artists are compensated $25
• Range of Adversarial Countermeasures. Multiple factors in- for completing the initial survey and detailed feedback, and $25
centivize AI model users to alter their images to escape identifica- more for participating in the Glaze perturbation user study.
tion as AI images. Thus our study also considers multiple types
of adversarial perturbations and explore their ability to confuse Procedure. The basic user survey included a randomized sample
different detectors (both automated and human). set of real human artwork, hybrid images, and generative AI images.
We ask participants to classify each image as human-generated,
unsure, or AI-generated. We also ask if they have seen the image
3.2 Evaluating Automated Detectors
displayed before, and answers to previously seen images are dis-
For automated software-based detectors, we consider both deployed carded. We ask questions about their artistic expertise, what styles
commercial systems, as well as research-based systems. There are of art they found easier to classify than others, and factors that
three well-known deployed commercial systems: influenced their classification.
• Hive: AI content detection using supervised classification provided We also presented the expert team with a small fixed sample set
by thehive.ai. of AI generated images that produced the most misclassifications in
• Optic: “AI or Not” is a free service (for limited queries) running a the other user studies. In an interactive chat setting, we asked the
proprietary algorithm to detect AI images and audio. experts for detailed feedback on techniques and specific examples
• Illuminarty: an AI detection service running a proprietary algo- applicable to each of these difficult images.
rithm including an implementation of DIRE.
3.4 Data Collection
For research-based systems, we selected two recent systems that
We curated our own dataset of real human-created artwork, AI-
had code (and models) available for testing.
generated images, and hybrid images. We define real images as
• DIRE [75]: DIRE (Diffusion Reconstruction Error) pushes a test original artwork drawn or created by human artists. AI-generated
sample forwards and backwards through a diffusion pipeline and images are images that are generated using AI models like Mid-
measures its changes to detect if the image came from that pipeline. journey, Stable Diffusion and DALL-E 3 from text prompts. Hybrid
DIRE has pretrained models with a public implementation. images are images that are AI-generated, retouched, and partially
• DE-FAKE [60]: DE-FAKE uses both image-only and multimodal drawn over by humans. One of the coauthors is a professional artist
embeddings to create a model-agnostic detector. We trained a with over 30 years of experience. He scanned numerous social me-
model based on techniques from the paper. dia sites and art platforms and collected a set of 40 images whose
Organic or Diffused: Can We Distinguish Human Art from AI-generated Images? CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA.

creators admitted they were AI-generated images altered by human each AI generator to produce 10 images, for a total of 50 images
artists later. We describe our data collection process in detail next. across all five generators.
Configurating Prompts for Each Art Style. We create prompts
4 Constructing the Dataset for AI generators by running BLIP [41] on human artworks sub-
mitted by artists, generating captions that effectively capture both
We consider images of diverse art styles and sources. We curate the artwork’s style and content. BLIP stands out as the state-of-the-
a dataset consisting of four different groups of images: artworks art model for image captioning. We apply this method to improve
handcrafted by human artists (§4.1), AI-generated images (§4.2), consistency, because artworks of the same style often display large
perturbed versions of human artworks and AI images (§4.3), and variation in content type and scene. For each art style, we randomly
unusual images created by combining human and AI efforts (§4.4). select 10 human artworks, making sure to include at least one piece
per contributing artist. The chosen images are input to BLIP to
4.1 Human Artworks extract the captions.
Human artworks are novel creations by artists that capture their Here we encounter an issue where, for some artworks, BLIP
personal touch and emotions. They showcase the unique techniques, struggles to extract the correct art style or any style at all. For
styles and perspectives of individual artists that only come from example, for some anime artworks, BLIP generates captions accu-
years of training and experiences. With help from the artist com- rately describing the content but fails to include any style. When
munity, we collected artworks across 7 major art styles, including prompted by this caption, the AI generators consistently produce
anime, cartoon, fantasy, oil/acrylic, photography, sketch, and wa- images in the photohumanistic style instead of the intended anime
tercolor. We recruited artist volunteers from Cara [11], a major style, despite the substantial difference between the two. Similarly,
portfolio platform dedicated to human-created art which uses fil- BLIP also fails to extract the watercolor style. In our study, we ad-
ters to detect AI images and peer-based validation between artists. dress this issue by adjusting the BLIP-generated captions to include
When recruiting volunteers, we provided artists with a detailed the style of the artwork, for which we have ground truth. Table 14
explanation of the study’s scope and operations. We sought their in Appendix summarizes the modification made for each art style.
consent to use their artworks in the study and offered them the Customizing Prompts per AI Generator. We also make cus-
option to opt out if they were not comfortable with their works tomized adjustments on BLIP-generated captions to address unique
being included. restrictions and configurations that each AI generator impose on
Overall, we recruited 53 artists and received 280 distinct artworks, prompts. Specifically, Adobe’s Firefly and OpenAI’s DALL-E 3 mod-
mapping to 40 images per style. For each style, we recruited 5 artists els do not respond to prompts that contain certain content. For
specialized for this style and each artist sent us 8 digital images instance, Firefly does not generate any image when prompted with
of their own artworks. The only exception is the watercolor style, “a fantasy style image of a woman in black holding a knife in the
where we recruited 7 artists and the number of images sent per snow,” but responds properly when the word “knife” is replaced with
artist varies between 4 and 11. “sword.” Similarly, DALL-E 3 does not react to prompts containing
Many artists choose to protect their intellectual property by copyrighted materials such as Marvel character names (e.g., Spider-
adding digital signatures or watermarks onto the images of their Man) and Nintendo game names (e.g., The Legend of Zelda). To
original artworks. However, many AI-generated images do not have address this, we manually substitute such content with more generic
these distinctive marks. Therefore, the presence of signatures or terms like “ superhero-themed action figures” or “a fantasy-themed
watermarks can potentially influence the perception of human art, action figure on a horse.” We verify that the modified prompts do
introducing unwanted bias and susceptibility to manipulation. To produce images that aligned with the intended description.
address this issue, we obtained consent from the artists to crop Another issue is the inconsistent aspect ratio of generated images.
out any signatures or watermarks from their submitted images. Four out of the five generators consistently produce square images.
In cases where the mark was too adjacent to the art subject, we DALL-E 3, on the other hand, generates images with random aspect
communicated with the artists to request the original artwork image ratios (e.g. 1024×1792). DALL-E 3 also tends to self-elaborate on
free of such markings. Finally, all images were cropped to achieve the input prompt, producing extraneous intricacies. To address
a square shape, with efforts made to minimize any potential loss of these artifacts, we include, in each input prompt to DALL-E 3, the
content. This is to maintain consistency between human artworks additional phrase of “square image prompt the text to Dall-e exactly,
and AI-generated images, since the latter is of a square shape. with no modifications.” Doing so effectively restricts its operation
Ethics. Aside from obtaining consent from artists, we take great to adhere to the original prompt and return a square image.
efforts to minimize exposure of human artworks to external sources. Selecting Art Style from CIVITAI. Unlike other models, CIVI-
We provide details in §9. TAI hosts instances of SDXL fine-tuned on specific art styles. For
each art style, we locate the most frequently downloaded model
4.2 AI-Generated Images from CIVITAI with that style. For instance, we use “Anime Art
Diffusion XL” to generate anime style images.
For AI-generated images, we take effort to cover the 7 art styles
(listed above) and different AI generators. We consider the five
most popular AI generators: CIVITAI [13], DALL-E 3, Adobe Firefly, 4.3 Perturbed Images
MidjourneyV6, and Stable Diffusion XL (SDXL). All were the latest Users of AI-generated images can intentionally add perturbations
release at the time of submission. For each art style, we prompt to images to deter their identification as AI images. We consider
CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. Anna Yoo Jeong Ha et al.

four representative types of perturbations and describe each below. choice. We then consider advanced scenarios where the detectors
Appendix (Fig 9) provides visual samples of perturbed images. face different types of perturbed images, benign or malicious.
Perturbation #1: JPEG Compression. Existing work has shown
that compression artifacts can reduce the accuracy of image clas- 5.1 Experiment Setup
sifiers [29, 39, 40]. To study its impact on AI image detectors, we As discussed in §3, we consider five classification-based detectors,
follow prior work to apply JPEG compression of a quality factor including three commercial detectors (Hive, Optic, Illuminarty),
15 [66] to AI-generated images before querying these classifiers. and two detectors built by academic researchers (DIRE, DE-FAKE).
Perturbation #2: Gaussian Noise. Similarly, digital noises can Our experiments use both original, unperturbed imagery (280 hu-
be introduced to disrupt classification-based detectors. For our man artworks and 350 AI-generated images) and their perturbed
study, we apply zero-mean Gaussian noise to each pixel value, with versions. We delay the study of unusual images to §7.3 due to their
a standard deviation limited to 0.025, a parameter sweetspot with decision complexity.
maximum impact and minimal visual disturbance. Detector Decisions. We study the ability of automated detectors
Perturbation #3: CLIP-based Adversarial Perturbation. A to identify a human artwork as human and an AI-generated image
more advanced (and costly) approach is to apply adversarial per- as AI, mapping to a binary decision. However, today’s automated
turbations on AI images. Adversarial perturbations [28, 43] are detectors all output a probabilistic score indicating the likelihood
carefully crafted pixel-level perturbations that can confuse ML clas- or confidence of the input being AI-generated. A score of 100%
sifiers. Automated AI image detectors are known to rely on the implies absolute certainty that the input is AI-generated, while 0%
public CLIP model [18, 60] for detection, and thus, we leverage indicates it is definitely human art. To convert this score into a
the CLIP model to craft our adversarial perturbations to maximize binary decision, we need to establish a boundary that distinguishes
their transferability to AI detectors [20]. Specifically, we compute between the two classes. Relying on a single threshold, such as 𝑥%
LPIPS-based adversarial perturbation [26] on each AI-generated (e.g., 50%), is obviously too fragile for this purpose.
image. We ensure that the perturbation is sufficient to confuse the Instead, we leverage the widely used 5-point Likert scale in user
CLIP model (i.e., LPIPS budget = 0.03). studies [34], designed to obtain quantitative measures of percep-
Perturbation #4: Glaze. Glaze [61] is a tool for protecting human tion/decision. Specifically, scores ranging from 0-20% are associated
artists from unauthorized style mimicry. It introduces imperceiv- with category human artwork (very confident), 20-40% with hu-
able perturbations on each artwork, which transforms the image’s man artwork (somewhat confident), 40-60% with “not sure,” 60-80%
art style to a completely different one in the feature space. The with AI-generated (somewhat confident), and 80-100% with AI-
widespread use of Glaze by artists has sparked extensive online generated (very confident). It is important to note that our user
discussions focused on instances where the use of Glaze on human studies maintain consistency by adopting the same rating scale.
art results in detection as AI images, while applying Glaze on AI- Next, to produce binary decisions, we designate any score below
generated images can evade detection. To understand its impact, 40% as a decision of human art and any score above 60% as a deci-
we use the public WebGlaze [38] tool to perturb both human art sion of AI-generated. Those inputs yielding scores between 40-60%,
and AI images. We choose both the default medium intensity and indicating uncertainty (“not sure”), are excluded from the experi-
also the high intensity, as artists often employ strongest protection ment. This exclusion is based on two practical considerations. First,
to safeguard their online images. there is no equitable method for comparing a “not sure” decision
against the definitive ground truth [12]. Second, the occurrence
of “not sure” is minimal across the machine detectors, e.g., 0.54%
4.4 Unusual Images for Hive and 4.29% for Optic, thus their removal has a negligible
Hybrid Images. Users can create “hybrid” images by painting impact on overall performance.
over AI-generated images. When posting them online, many include Evaluation Metrics. We report detector performance using four
in the caption the generative models used. One of the coauthors, a metrics. For easy notation, let [H→H] represent the # of human
professional artist over 30 years of experience, collected 40 hybrid artworks detected as human-made, and [AI→AI] represent the #
images to include in our dataset and verified their sources. The of AI-generated images detected as AI-generated. Let [H] and [AI]
images cover a variety of styles and subjects, including anime, represent the total # of human artworks and AI-generated images
cartoon, industrial design, and photography. included in this test, respectively.
Human Artworks with Upscaling. Some artists use tools like
• Overall accuracy (ACC) measures the accuracy of the detector re-
image upscalers to enhance the quality of photography images, e.g.,
gardless of the data origin. ACC= ([H→H]+[AI→AI])/([H]+[AI]).
reducing blur or noise introduced during image capturing. We have
• False positive rate (FPR) represents the ratio of human artworks
70 images in this group, upscaled using the baseline function of
misdetected as AI-generated. FPR=1- [H→H]/[H].
MagnificAI [1], a web upscaling tool endorsed by artists.
• False negative rate (FNR) measures the ratio of AI-generated
images misdetected as human artworks. FNR=1 - [AI→AI]/[AI].
5 Accuracy of Automated Detectors • AI Detection success rate (ADSR) captures the detector accu-
Using the dataset outlined in §4, we examine the efficacy of auto- racy on AI-generated images. ADSR= [AI→AI]/[AI]. We use this
mated detectors in detecting AI-generated images. We first report metric to examine how generation-related factors would affect
the results on unperturbed imagery and the impact of AI generator the detection outcome of AI-generated images.
Organic or Diffused: Can We Distinguish Human Art from AI-generated Images? CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA.

Tested on Human Artworks + AI-generated Images ADSR (%) ↑

Detector ACC (%) ↑ FPR (%) ↓ FNR (%) ↓ CIVITAI DALL-E 3 Firefly MJv6 SDXL
Hive 98.03 0.00 3.17 Hive 100.00 98.57 91.04 94.29 100.00
Optic 90.67 24.47 1.15 Optic 100.00 97.14 97.06 100.00 100.00
Illuminarty 72.65 67.40 4.69 Illuminarty 94.03 100.00 92.42 91.67 98.41
DE-FAKE 50.32 41.79 56.00 Table 2: The impact of AI-generator choice on detector perfor-
DIRE (a) 55.40 99.29 0.86 mance, represented by ADSR, the % of AI-generated images
DIRE (b) 51.59 25.36 66.86 correctly detected as AI-generated.
Ensemble 98.75 0.48 1.71
Table 1: Performance of automated detectors tested on un-
using images generated by SDXL and MSCOCO captions. When
perturbed human artworks and AI-generated images. The
tested on SDXL images produced from MSCOCO prompts, the clas-
Ensemble detector (Hive+Optic+Illuminarty) takes scores
sifier reproduces a high ACC of 92.44% similar to [60]. However,
from Hive, Optic, and Illuminarty, using the highest confi-
when tested on images generated using artwork prompts, the accu-
dence value as the determining score.
racy drops to 50.3% for SDXL images and 46.43% for those produced
by other generators. We attribute performance discrepancy to the
5.2 Results of Unperturbed Imagery issue of transferability. Training images generated from MSCOCO
We start from unperturbed human artworks (280 images) and AI- prompts do not follow the art dataset distribution, so these open-
generated images (350 images). Table 1 summarizes the detection source detectors did not transfer well to the art dataset.
performance in terms of ACC, FPR and FNR. Combining Detectors (Hive+Optic+Illuminarty). We also
Hive. Hive is the clear winner among all five detectors, with a study an Ensemble detector that leverages decisions from Hive,
98.03% accuracy, 0% FPR (i.e., it never misclassifies human artworks), Optic, and Illuminarty. In case of a disagreement between the de-
and 3.17% FNR (i.e., it rarely misclassifies AI-generated images). tectors, it opts for the decision marked with highest confidence.
Optic and Illuminarty. Both perform worse than Hive, except The confidence is calculated relative to the classification, by com-
that Optic has a lower FNR (1.15%) and thus is more effective at flag- puting | detector score - 0.5 |. The result in Table 1 shows that the
ging AI-generated images. However, this comes at the expense of a improvement over Hive is minor: 0.6% increase in ACC while FNR
high 24.47% FPR, where human art is misclassified as AI-generated. reduces from 3.17% to 1.71% and FPR increases from 0% to 0.48%.
Illuminarty demonstrates even harsher treatment towards human
5.3 Impact of AI Generator Choice
art, with a very high FPR of 67.4%.
We also investigate the factors that may affect detection perfor-
DE-FAKE and DIRE. We experiment with all 6 versions of DIRE,
mance. While one might anticipate that the art style could have
representing its 6 checkpoints published online. The detailed per-
an impact, we do not observe any notable effect, at least for the
formance is listed in Table 12 in Appendix. The overall accuracy is
seven major styles considered by our study. Instead, we observe
consistently low for all 6 models (< 55.5%). The top-2 models’ per-
a considerable impact by the choice of AI-generator. This can be
formances are listed in Table 1 as DIRE (a) and DIRE (b), one with
seen from both Table 2 and Figure 2. Table 2 lists ADSR, the %
nearly 100% FPR, and another with 66% FNR. DE-FAKE shows a sim-
of AI-generated images correctly detected as AI, while Figure 2
ilar pattern, with a low 50% accuracy and high FPR and FNR values.
displays the raw confidence score produced by the detectors.
Given their poor performance, we do not conduct any additional
Across the five AI generators, images by CIVITAI and SDXL are
experiments with both detectors.
the “easiest” to detect by Hive and Optic, i.e., 100% ADSR and not
Variation across Automated Detectors. The large performance a single “not sure” decision. Since CIVITAI models are fine-tuned
variations across detectors could be attributed to the diversity and versions of SDXL, this suggests fine-tuning has minimal impact on
coverage of their training data. According to its website, Hive uti- AI image detection.
lizes a rich collection of generative AI datasets and can identify the On the other hand, Firefly images are the least detectable – Hive
generative model used for the current input from a pool of nine marks 6 out of 70 Firefly images as human art and 3 as “not sure”
models 2 . Similarly, Optic can pinpoint an input image to one of while Optic marks 2 as human art and 2 as “not sure” (Fig. 2).
the 4 generative models. In contrast, Illuminarty’s training data We hypothesize this is due to lack of training data. Firefly is a
coverage is limited, particularly since it does not support image relatively new model, so Hive and Optic are likely to have much
files exceeding 3MB3 . Illuminarty and Optic are smaller companies less training data relative to other models
with less training data 4 .
DE-FAKE and DIRE’s training data are constrained and lack rep- 5.4 Impact of Perturbations
resentation from art. DIRE models are trained on interior design Our goal is to understand whether adding perturbations to images,
images (LSUN-Bedroom), human faces (CelebA-HQ) and ImageNet. whether benign or malicious, could change detection outcomes.
The detection accuracy on art images is around 50%. For DE-FAKE, This triggers two questions below.
we follow the methodology described by [60] to train the classifier
Question 1: How do perturbations affect the detection of
2 As of April when conducting additional tests on Hive, it now detects 27 models. AI images? We explore four perturbations: JPEG compression,
3We downsample images generated by Firefly from 2048x2048 to 1024x1024 to meet Gaussian noise, adversarial perturbations on CLIP, and Glaze at
this restriction.
4 Illuminarty’s creator informed us the model has not been updated for 6 months in medium and high intensity. The details of each perturbation are
Novemeber 2023. discussed at length in Section §4.3.
 CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. Anna Yoo Jeong Ha et al.

1 ACC (%) ↑ FPR (%) ↓ FNR (%) ↓

Detector score (%)

0.8 Hive 80.81 / 98.03 3.23 / 0 32.44 / 3.17

0.6 Optic 61.92 / 90.67 33.59 / 24.47 42.00 / 1.15
0.4 Illuminarty 68.66 / 72.65 56.91 / 67.40 10.78 / 4.69
0.2
Hive Ensemble 82.70 / 98.75 3.21 / 0.48 28.57 / 1.71
Optic
0
Illuminarty Table 4: Detection performance on human art and AI-
CIVITAI DALL-E 3 Firefly MJv6 SDXL
generated images with and without Glaze (at high intensity),
Generative Model
shown as: Glazed / Unperturbed.
Figure 2: The confidence score produced by automated detec-
tors on images generated by 5 generators. Detecting images
generated by Firefly is the hardest. increase in intensity only resulting in a drop in ADSR ranging from
0.5% to 4%. Glaze has the least effect on Illuminarty, dropping the
ADSR (%) ↑
ADSR from 95.31% to 89.22% for high-intensity Glaze. In compar-
JPEG Gaus. Glaze Glaze
Adver. Unperturbed ison, Glaze reduces the ADSR for both Hive and Optic to below
comp. noise (med.) (high)
Hive 91.88 88.73 93.00 69.73 67.56 96.83 70% while the Ensemble detector is only able to achieve 71.43%. We
Optic 97.98 52.63 80.42 62.62 58.00 98.85 explore the effects of Glaze further below.
Illuminarty 93.19 61.43 94.34 89.80 89.22 95.31 Question 2: Does Glaze Affect Accuracy Similarly on Human
Ensemble 94.29 88.86 94.29 73.71 71.43 98.29 Artwork vs AI-generated Images? To investigate if the de-
Table 3: The impact of perturbations on AI-generated images, tection of human artwork is impacted similarly to AI-generated
represented by ADSR, the % of AI-generated images correctly images when Glazed, we applied high-intensity Glaze to both. The
detected as AI-generated. detection outcomes, including ACC, FPR and FNR are presented
in Table 4 comparing results with and without Glaze. Our findings
reveal a large reduction in ACC with the use of Glaze, as expected.
Table 3 reports ADSR, the success rate of detecting AI-generated However, the FNR increases across all detectors, while the FPR
images as AI when they include one of the four perturbations. As a remains relatively consistent. On Hive, ACC is decreased around
reference, we include the ADSR for unperturbed images. We discuss 20% and yet FPR increases only 3.23% while FNR increases almost
each of the perturbations individually below, analyzing their impact 30%. Our Ensemble detector is able to achieve the highest accuracy
on each detector. on Glazed images, but maintains a FNR of 28.57%. This suggests
Figure 3 plots a detailed view of the impacts of perturbations that glazing human artwork typically does not affect classification
on the scores assigned by Hive to each image. While we exam- success, but glazing AI-generated images often leads to misclassifi-
ined these plots for each detector, we observed relatively similar cation as human artwork. We attribute this to the scarcity of Glazed
trends across all, except that Optic and Illuminarty exhibited higher images online and thus the lack of Glazed images in the detector’s
levels of noise. Hive’s plots are easier to interpret, both for the training datasets. Additionally Glaze was created to protect human
perturbed and unperturbed data, thus we only present them for artwork and has since been adopted by many artists, therefore the
clarity. Additionally, we identify a set of images that demonstrate distribution of Glazed human art vs. Glazed AI-generated images
‘robustness’ to any perturbation against Hive and provide a more online is likely skewed. Once again this demonstrates the impact
in-depth discussion of these images in Table 13 in Appendix. of insufficient training data on image-based classifiers.
JPEG Compression. JPEG compression shows minimal impact
on performance across all detectors, as they all remain above 91%. 5.5 Summary of Findings
The lossy compression artifacts don’t hinder the detectors’ ability Our study leads to three key findings.
to detect AI images. • Commercial detectors perform surprisingly well, and Hive per-
Gaussian Noise. Gaussian noise has little impact on Hive, it does forms the best. It is highly accurate (98.03% accuracy) and never
drop both Optic and Illuminarty in ADSR. Optic’s ADSR decreases misclassifies human artwork in our test. The other two detectors
to 52.63%, and Illuminarty’s ADSR decreases to 61.43%, both of (Optic and Illuminarty) tend to misclassify human art as AI.
which are near random guessing. On the other hand, Hive’s perfor- • Commercial detectors are heavily affected by feature space pertur-
mance remains relatively high at 88.73% ADSR. These classifiers bations (i.e., Glaze) added to AI-generated images. On the other
may have already adapted to the presence of mild noise in training hand, human artworks with Glaze are mostly unaffected, as they
images and learned to suppress the effects of noise. are still largely detected as human.
Adversarial Perturbation. Of all the perturbations, the adver- • Poor performance on Firefly and Glaze indicates the results are
sarial perturbations on CLIP have the least impact on performance correlated with training data. Performance of supervised classifica-
for Hive and Illuminarty. Optic’s ADSR drops to 80.42%, but Hive tion depends heavily on the availability of training data. Detectors
and Illuminarty both remain above 93%. This may indicate rela- are more vulnerable to newer models with less available training
tively low transferability of targeted attacks in the CLIP space in data (Firefly) and adversarial inputs they have not seen before.
black-box settings.
Glaze. Across detectors, Glaze at both intensities consistently 5.6 Followup Robustness Tests on Hive
has a significant impact on ADSR. Additionally Glaze affects each After the paper was accepted, we discovered a Reddit post that
detector similarly between medium and high intensity, with the claimed overlaying a real image of a white wall onto an AI-generated
Organic or Diffused: Can We Distinguish Human Art from AI-generated Images? CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA.

Unperturbed JPEG Compression Gaussian Noise Adv. Perturbation Glaze (medium) Glaze (high)
1
Detector score (%)

0.8

0.6

0.4

0.2 Unperturbed
Perturbed
0
0 100 200 300 0 100 200 300 0 100 200 300 0 100 200 300 0 100 200 300 0 100 200 300

Figure 3: Impact of five different perturbations on the Hive confidence score, for 350 AI-generated images. In each figure, the
images are indexed by the increasing Hive score of unperturbed versions.
1 0.2

0.8

0.6
CDF

CDF 0.1
0.4

0.2
April 2024 Adobe Photoshop
June 2024 Our Overlay
0 0
10 20 30 40 50 60 70 80 90 10 15 20 25 30 35 40
Overlay Intensity Overlay Intensity

Figure 4: CDF of overlay intensity required Figure 5: CDF of overlay intensity required to
to change Hive’s decision over the period of change Hive’s decision using different over- Figure 6: User study in-
3 months. lay methods in June 2024. terface.

image could bypass Hive’s detection [78]. We investigate this as has a significantly stronger effect against Hive at higher intensi-
a new adversarial approach to bypassing detection, and conduct ties. This gap is somewhat unexpected, and just confirms that even
a large-scale test on 105 AI-generated images, randomly sampled small variations in adversarial attacks can have large unpredictable
from our dataset to include 3 images from each of the 7 styles and impact on Hive’s detection robustness.
5 AI-generators. We implement a python script to overlay each
image with a white wall image from Adobe Shutterstock, at in- 6 User Studies on Human Detection
tensity levels varying from 10% to 90% in increments of 10. Hive Next, we measure the ability of human users and artists in identi-
confirms the Adobe’s white wall image as not AI-generated with fying AI-generated images. As discussed in §3.3, we perform user
100% confidence. studies with 3 separate user populations, including crowdworkers,
Figure 4 illustrates the CDF graph of the distribution of overlay professional artists and expert artists.
intensities required to flip Hive’s decision from AI-generated to To ensure uniformity, each group is provided with the same user
not AI-generated. The red curve from April 2024 shows there is a study, but the expert group received extended followup questions
17.5% chance that Hive’s decision will change at or below intensity asking for detailed examples (Section 6.4). Our study is approved by
levels of 50%. Yet, there are corner cases where images (2.91% in our local Institutional Review Board (IRB). We omit the IRB number
April and 3.85% in June) were able to fool Hive with the white wall for anonymity.
overlaid at only 10% intensity. Most images require around overlay
intensity of 60-80% to change Hive’s classification. 6.1 Study Setup
We also note that Hive’s performance improved over time. Fig-
ure 4 displays Hive results collected in April 2024 and June 2024. Participants. As discussed in §3.3, we recruited three partici-
The shift in the CDF shows that Hive’s accuracy and robustness pant groups, and a total of 3993 participants completed our study
increased in the 2 month period, possibly due to adversarial training and passed all attention checks. These include 177 baseline par-
on inputs or updates to its supervised classification algorithm. ticipants recruited from Prolific, referred to as general users, 3803
We conduct another smaller-scale test to see the effect of dif- professional artist volunteers, and 13 high profile experts.
ferent overlay methods on Hive’s output. We apply and compare Task. Our study takes the form of a user survey. We added
Adobe Photoshop’s “multiply blend” overlay option (the method attention-check questions in both the middle and concluding sec-
used in the Reddit post) against our scripted blending algorithm, tions of the survey to filter out low attention participants.
varying intensities from 10% to 40% in increments of 10 across 63 After a brief introduction on generative models and the current
AI-generated images5 . Figure 5 shows Adobe’s Photoshop method issue of distinguishing between human artworks and AI-generated
images, we present the participant with a sequence of 20 images,
5 Dataset is narrowed down to 3 images from MJv6, Firefly, and SDXL across 7 styles. shown one at a time, and ask them to decide whether each image
CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. Anna Yoo Jeong Ha et al.

is human-made or AI-generated. For each image, we ask two ques- ACC (%) ↑ FPR (%) ↓ FNR (%) ↓
tions. The first question is “Have you seen this image before?” If General user 59.23 40.81 40.75
yes, we disregard the response for this image, since the participant Professional artist 75.32 23.53 25.37
Expert artist 83.00 20.78 14.63
has likely seen this image and is possibly aware of the image’s true
source. The second question asks them to rate the current image Table 5: Performance of human detection on both human art
with one of the five choices: “human art (very confident),” “human and AI-generated images.
art (somewhat confident),” “not sure,” “AI-generated (somewhat
Anime Cartoon Fantasy Photo. Oil/Acrylic Sketch Watercolor
confident),” “AI-generated (very confident).” For each participant, Gen. 57.43 63.58 62.80 63.23 56.81 54.99 56.16
we randomly sample 20 images from a collection of 670 images, Pro. 79.16 82.42 81.72 76.05 66.01 70.27 70.51
consisting of 210 human artworks, 70 human artworks after upscal- Table 6: Impact of art style on detection accuracy (ACC) for
ing, 350 AI-generated images, and 40 hybrid images. Every image general users (Gen.) and professional artists (Pro.). We ex-
is seen by five participants. clude the result of expert artists due to insufficient coverage.
Next, we ask the participant, for each of the 7 art styles, whether
they are confident at distinguishing human art from AI-generated ADSR (%) ↑
images; the user study interface is presented in Figure 6. If confident, CIVITAI DALL-E 3 Firefly MJv6 SDXL
General user 66.77 60.63 51.18 50.00 67.56
we ask them to describe the properties of the art that contributed to
Professional artist 83.56 67.56 75.40 61.53 84.43
their decisions. Here we present seven options: “content,” “complex- Expert artist 90.32 86.96 65.22 86.96 95.65
ity,” “technical skill,” “perspective,” “lighting,” “consistency,” and an
Table 7: ADSR of human detection on AI-generated images by
additional text response section for additional details. Finally, we
different AI generators. ADSR is the success rate of detection
ask them if they self-identify as full-time or part time artists.
AI-generated images as AI-generated.
Performance Metrics. Same as the evaluation of automated
detectors in §5.1, we convert each 5-point Likert scale rating into
a binary decision. That is, both “human art (very confident)” and that the image’s artistic style has a clear impact on the performance
“human art (somewhat confident)” map to a decision of human art, of professional artists. Among the 7 styles, the top-3 “easier-to-
and “AI-generated (somewhat confident)” and “AI-generated (very detect” styles are anime, cartoon and fantasy, and the bottom one
confident)” map to AI-generated. Not decision is produced for the is oil/acrylic for which the detection accuracy drops nearly 20%.
“not sure” responses, and we ignore them when computing ACC, This aligns with the participants’ feedback on the styles that they
FPR, FNR and ADSR (same as §5.1). feel most confident on detection, where 82.42% selected cartoon,
followed by fantasy and anime.
6.2 Detection Accuracy for General Users For these top-3 styles, the artists select “consistency” as the most
dominant decision factor. Specifically, in the text response section
Table 5 shows detection performance of general users, which is
on anime images, the most frequently entered words are hands,
only slightly better than random coin-flip. This shows that general,
hair, details, eyes and lines. Similarly, in the case of fantasy images,
non-artist users are unable to tell the difference between human
artists observe specific details such as asymmetric armor or symbols
art and AI-generated images.
that should be symmetric. This shows that professional artists can
Table 6 examines the impact of art style on detection accuracy
apply their knowledge and experiences of art creation to identify
(ACC). For general users, accuracy is slightly higher on cartoon,
inconsistencies in AI-generated images, with particular focus on
fantasy and photography styles. These three represent a collection
fine-grained details.
of “digital” artworks more frequently accessible to general users,
compared to “physical” art styles like oil/acrylic, watercolor, and Impact of AI generator. Table 7 lists the detection success rate of
sketch, and other “digital” artworks like anime. We attribute the AI-generated images broken down by source generator model. For
slightly higher accuracy to this increased familiarity. Finally, Table 7 professional artists, accuracy clearly varies across generator models.
reports detection success rate on AI-generated images, which is The top-2 “easiest-to-detect” are CivitAI and SDXL, with detection
around 60% and varies slightly across the five AI generators. Images success rate above 83%. Interesting to note that these two are also
generated by Firefly and MidjourneyV6 are harder for non-artist the most easy-to-identify generators for automatic detectors. Next,
users to recognize. detection accuracy reduces to below 70% for images produced by
MidjourneyV6 and DALL-E 3, suggesting that MidjourneyV6 and
DALL-E 3 produce better copies of human art styles, making it
6.3 Detection Accuracy for Professional Artists
harder for artists to spot inconsistencies.
Compared to general users, professional artists take more time to
inspect images and are more effective at distinguishing between 6.4 Detection Accuracy for Expert Artists
human art and AI images. They produce a detection accuracy of Our 13 expert artists show greater proficiency in the detection task,
75.32% (Table 5). Their 23.53% FPR and 25.37% FNR indicate that raising overall detection accuracy (ACC) to 83% (see last row in Ta-
their detection performance is not skewed toward either human ble 5). They also produce slightly imbalanced FPR (20.78%) and FNR
art or AI-generated images. (14.63%), indicating that they are better at spotting AI-generated
Decision Factors. To understand why professional artists are images than human art. When we gave feedback to the experts
better at evaluating art images, we study the key factors that influ- on their results, many were frustrated that they committed errors
enced their decisions. The first is the image’s art style. Table 6 shows by marking human art as AI (false positive). In retrospect, they
 Organic or Diffused: Can We Distinguish Human Art from AI-generated Images? CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA.

Original Detail 1 Detail 2 Original Detail 1 Detail 2 Original Detail 1 Detail 2

a) b) c)

d) e) f)

Figure 7: Six hard-to-detect AI-generated images, and their artistic errors/inconsistencies discovered by our expert artists. In
each 3-image group, the left one is the full image, and the right two are zoomed-in view of discovered artifacts.

explained that they identified detailed mistakes and inconsisten- round moon looks like a digital art (detail 2), inconsistent with
cies which were likely due to inexperience and human error by the overall messy painting style. The white is “too clean” (detail
the human artists. For example, in a painting of a bedroom bathed 1) when transitioning into the blue background.
in moonlight, the shadow of a window pane had slightly offset • Intentionality in details. In art featuring human figures, human
position of the latch compared to the window itself. This was seen artists dedicate considerable effort to convey precise details of
by expert artists as an inconsistency, but later attributed to lack of human features. In image b), the light caught in the eyes do not
attention to detail by a human artist. This attests for the drop in match and the hair ends flow in opposite directions from the rest
ADSR for Firefly images. Since expert artists look at fine-grained of the hair (detail 2). Similarly, in image c), hairs behind the neck
detail in AI-generated images, they are overfitted to spot irregulari- are floating and doing completely different things from the rest of
ties from popular generators and have yet been accustomed to the her hair. Human artists also avoid unusual tangents with the bangs
newer style of images. and eyebrow, something that this image completely overlooked.
Decision factors. Our expert artists are generally very confident • Limitations of medium. Experienced artists know that certain pat-
at judging images from more than 2 art styles. The most frequently terns and details are impossible to produce in real life due to
selected one is fantasy art. The primary decision factors are intrinsic physical limits of the medium. In image e), since watercolor is
artistic details, which often go beyond the “(in)consistency” element transparent and it bleeds after each brush stroke, the “white over
used by professional artists in their detection efforts (as discussed dark in the quills” (detail 1) is impossible to physically produce.
in §6.3). Specifically, our expert artists point out that AI-generated The image is also too smooth to be hand painted on paper (detail
images generally “look too clean, rendered, and detailed” and “have 2), since watercolor bleeds in random directions.
no variety in composition, edges, distribution of detail,” and had • Domain knowledge. There are specific rules when drawing specific
“design elements are nonsensical or blend into each other in telltale subjects that are easy to validate. Wet paintings have an order
ways,” while human-made fantasy art “contains components that of application, from light to dark, transparent to opaque. Thus
are novel such as armor or jewelry.” all white spaces must be subtractive. Yet in image f), the white
Focus Study on “Hard-to-Detect” AI Images. For a compre- highlight is added after a dark area, which is wrong. Also the
hensive view of how experts identify AI-generated images, we water flow shows the wrong diffusion pattern. A trained artist
presented our expert group with a fixed set of the six AI-generated should not make these mistakes.
images that produced the most false negative errors by the pro- We consider these techniques in aggregate, and note that most of
fessional (non-expert) artist group. They cover 5 styles: sketch, them require significant training and external knowledge to apply.
oil/acrylic, cartoon, watercolor, and anime. For each of these dif- In this sense, factors such as intentionality and domain knowledge
ficult images, we asked the experts for detailed feedback on what seem like the most difficult for AI models to capture from training
exactly they would use to identify these images as AI. Figure 7 data. But as a whole, these domain-specific filters clearly operate
shows all six images, each followed by two regions of the image very differently from statistical approaches used in automated de-
zoomed in to show details of artifacts identified by our expert artists. tectors. Thus we believe these methods will continue to be effective
Next we summarize general techniques identified by experts, even as AI models continue to evolve.
and use specific images to illustrate these methods.
• Consistency in medium. For any specific artwork, a trained artist
typically employs only a single consistent medium, e.g. pencil, 6.5 Summary of Key Findings
charcoal, and rarely combine multiple mediums. For example, in Our multi-tier user study finds that general (non-artist) users are
the sketch in image a), our experts locate not only a “weird halo unable to distinguish between human art and AI-generated images.
effect” (detail 1) due to the use of both pencil and charcoal, but also Professional artists are more confident and make more accurate
a “crunchiness” (detail 2) to the lines that associate with neither decisions, and experienced experts are the most effective. Diving
pencil nor charcoal. AI models are associating lines from multiple deeper into concrete examples, we learn that experts leverage ex-
mediums with the same style and fail to differentiate between tensive knowledge of art mediums and techniques to identify what
them. Similarly, oil and acrylic paintings can display messy or features are physically impossible, and inconsistencies and mistakes
smooth styles but never both. In image image d), the perfectly that professional artists would avoid.
CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. Anna Yoo Jeong Ha et al.

Human detector Machine detector

Metric General
General Professional Expert Hive Optic Illuminarty
Professional
ACC (%) ↑ 59.23 75.32 83.00 98.03 90.67 72.65
FPR (%) ↓ 40.81 23.53 20.78 0.00 24.47 67.40 Expert
FNR (%) ↓ 40.75 25.37 14.63 3.17 1.15 4.69 Hive
Table 8: Performance of human and machine detectors on Optic
unperturbed imagery. Illuminarty
0% 20% 40% 60% 80% 100%

(a) AI-generated images

7 Human vs. Classifier Detectors General
In this section, we present our findings by comparing the perfor- Professional
mance of human and machine detectors. Our analysis starts from Expert
the baseline results, where all detectors are tested using the same Hive
set of usual images, covering unperturbed human artworks and Optic
AI-generated images. Next, we investigate how human detectors Illuminarty
respond to Glazed images, for which machine detectors have strug- 0% 20% 40% 60% 80% 100%

gled to handle (as shown in §5.4). Finally, we explore both human (b) Human artworks
human art (very confident) human art (somewhat confident)
and machine detectors’ responses to unusual images, including not sure AI-generated (somewhat confident)
both hybrid images (when human edits AI-generated images) and AI-generated (very confident)

upscaled human art (where artists polish the digital image of their
Figure 8: Distribution of detection decision represented by
artworks using enhancement tools).
the 5-point Likert rating on (a) AI-generated images and (b)
human artworks.
7.1 Decision Accuracy and Confidence ACC (%)↑ FPR (%) ↓ FNR (%) ↓
We start from comparing human and machine detectors on the Expert Artist 83.44 23.53 8.97
baseline task of differentiating Hive 87.76 4.04 20.62
Detection accuracy. Table 8 lists the detection performance Optic 61.20 38.30 39.33
of both human detectors and classifier-based detectors, on unper- Illuminarty 66.11 59.09 9.78
turbed images. Among the six detectors, we have Hive > Optic > Table 9: Detection performance on Glazed version of human
Expert Artist > Professional Artist > Illuminarty > Non-Artist. artworks and AI-generated images.
Decision confidence. We are also interested in understanding
the distribution of decision confidence among human and machine Here we randomly select 100 AI-generated images (20 images per
detectors. Figure 8 shows, for AI-generated images and human AI generator) and 100 human artworks (15 images per style, except
artworks, the distribution of the “raw” decision represented by the 10 images for cartoon), and use the WebGlaze tool and the medium
5-point Likert score used by our study. For AI-generated images intensity setting to Glaze all 200 images. For each expert artist, we
(shown by the top figure), the dark blue bar represents the ratio of randomly select 20 Glazed images and ask them to decide between
correct decisions that also carry high confidence, while for human human art and AI-generated.
artworks (the bottom figure), the dark orange bar captures the ratio Table 9 lists the overall performance across Glazed images. We
of correct decisions made with high confidence. see that human experts largely outperform both Optic and Illu-
Across the three groups of human detectors, general users are minarty at judging Glazed images, whether it is human art or AI-
the least accuracy and also show the lowest confidence in their generated. While Hive is still the best-performing detector in the
decisions, while expert artists are the most accurate and the most overall accuracy (87.76%), our expert artists are not far off (83.44%).
confident. This is as expected. Across the three machine detectors, But more importantly, human experts achieve a much lower FNR
Hive is highly confident (and accurate), followed by Optic. Addi- (8.97%) compared to Hive (20.62%). This implies that human artists
tionally, Optic is more confident when facing AI-generated images do outperform machines at judging Glazed versions of AI-
than human artworks. Overall, the two machine detectors (Hive generated images.
and Optic) show higher confidence (and higher accuracy) than ex-
pert artists, while the third machine detector (Illuminarty) performs 7.3 Judging Unusual Images
worse than both expert and professional artists.
As discussed in §4.4, we also consider two unusual types of images:
(i) hybrid images produced by human users editing AI-generated
7.2 Are Human Artists Better at Judging Glazed images, and (ii) upscaled human artworks where artists apply digital
Images? touchups to polish the image of their artworks. By evaluating them
As shown by Table 4 in §5.4, machine detectors, especially Hive using both human and machine detectors, our goal is to identify, if
and Optic, are much less effective at judging Glazed versions of any, notable differences in how they are evaluated by human and
AI-generated images. Thus a natural question is “would artists who machine detectors.
know or use Glaze be more effective at judging Glazed images?” In total, we collect 70 human artworks after applying upscaling
To answer this question, we conduct an additional user study with and 40 hybrid images (see §4.4). Given the limited member size
our expert group, who all use Glaze on their published artworks. of our expert group, we do not have sufficient coverage on these
Organic or Diffused: Can We Distinguish Human Art from AI-generated Images? CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA.

Unperturbed Human Artworks and AI Images 9 Ethics

Detector ACC (%)↑ FPR (%) ↓ FNR (%) ↓
Hive 98.03 0.0 3.17 Our user study was reviewed and approved by our institutional
expert 83.00 20.78 14.63 review board (IRB). In our study, we prioritized consent and protec-
Hive + expert 92.19 8.11 7.63 tion of all participants, especially human artists and their artworks.
Table 10: Detection performance on unperturbed human Consent for Human Art. Our study necessitates the use of
artworks and AI-generated images. Three detectors: Hive, artwork by human artists. To obtain consent, we identified artists
one expert (per image), Hive + expert (tiebreak=confidence). and reached out with request for permission. Many responded. We
waited roughly 4 weeks, and reached out again to everyone else.
Glazed Human Artworks and AI Images Once downloaded, images are anonymized and stored on private,
Detector ACC (%)↑ FPR (%) ↓ FNR (%) ↓ secure servers.
Hive 87.12 6.06 19.70 Since we asked for artwork from human artists with signa-
expert 84.85 23.08 7.46
tures/watermarks removed, we were extremely sensitive to po-
Hive + expert 92.54 6.06 8.82
tential unauthorized exposure. We took efforts to minimize the
Table 11: Detection performance on Glazed human artworks
exposure of human artwork to external sources. They were avail-
and AI-generated images. Three detectors: one expert (per
able to participating crowdsourced workers only for a short time
image), Hive, Hive + expert (tiebreak = confidence).
through the Prolific platform. In the regular artist user study, im-
ages were available to participants for a total of 14 hours, after
images by expert artists and thus omit their results. Figure 10 in which we shut down the study to minimize uncontrolled exposure.
Appendix plots, for both types of images, the distribution of the Exposure to Web Services. We took careful steps to ensure that
“raw” detection decision represented by the 5-point Likert score, for images of human art were not misused by external AI detection
both human and machine detectors. services. We reached out to both Optic and Illuminarty, and were
Hybrid Images. For these images, the decision distribution assured that images are never used for training and deleted after
is similar to that of AI-generated images shown in Figure 10(a), process (with a max of 4 days for Optic). Hive’s terms of service
suggesting that both human and machine detectors frequently label states they can train AI models using images uploaded via the free
these images as AI-generated. web service, but not images classified through paid APIs. Thus we
Upscaled Human Art. The decision distribution is very sim- obtained access to a paid Hive account, and ensured all images of
ilar to that of human artworks in Figure 10(b). This implies that human art were classified using this paid Hive account.
upscaling does not have a significant impact of human artworks in
10 Discussion and Takeaways
terms of their decision outcomes from both human and machine
detectors. As with any real-world study, there are limitations in our study
that need to be considered.
8 Combining Human and Automated Detectors • Category of styles: our dataset only included a few fixed art styles.
Our study shows that both human artists and automated detec- More diverse styles might provide more comprehensive results.
tors face challenges in distinguishing between human art and AI- • Cropping of images: cropping was applied to a small number of
generated images. Tools like Hive are highly effective at evaluating human artworks. We avoided samples with highly irregular aspect
unperturbed images, but perform poorly when AI-generated im- ratios and ensured meticulous cropping to minimize irregularities.
ages are intentionally perturbed (e.g., Glaze or image overlays) to • Curating Likert Scale: we discarded “not sure” responses from the
evade detection. On the other hand, human experts can still identify user study. This was done to maintain consistency with measur-
perturbed AI-generated images. Thus we believe a mixed team of ing the confidence level (40-60%) in automated detectors and to
human artists and machine classifiers will be the most effective. prevent guesses from affecting our metrics.
Teaming up Hive and Expert Artists. We evaluate a scenario
Our results also suggest takeaways for different audiences.
that combines Hive scores with one expert artist. If Hive and the
expert disagree, the score with higher confidence wins. For artists. Proving human authenticity will be come increas-
We evaluate the combined detector on both unperturbed art- ingly important, and increasingly difficult. No single method wil
works/images and their Glazed versions. Table 10 shows that, for be foolproof, and artists should consider incorporating work-in-
unperturbed images, the combined detector exhibits a slightly lower progress (WIP) or timelapses into their process.
accuracy compared to Hive. Next, Table 11 shows that the combined For researchers. Developing highly accurate detection tools re-
detector is highly effective in judging Glazed images, outperform- quires continued investment in ethically obtaining diverse training
ing both Hive and expert. Notably, it outperforms Hive by lowering sets. To enhance robustness, incorporating perturbed and adversar-
FNR from 19.70% down to 8.82%. Thus it is more effective at identi- ially altered images during training is crucial.
fying AI-generated images that have applied Glaze in an attempt to For policy makers. Implementing standards that mandate mixed
evade detection. At the same time, the combined detector achieves detection teams in critical applications can enhance both detection
a low FPR like Hive (6.06%), remarkably lower than that of expert accuracy and robustness.
(23.08%). Finally, the equilibrium between FPR and FNR values, on As AI evolves, generative AI users seeking to avoid detection will
both Glazed and original images (Table 10), suggests “unbiased” adapt to exploit vulnerabilities. Some are using Adobe’s Photoshop
detection accuracy for human artworks and AI-generated images. to add imperfections to avoid the smooth AI finishing look. Others
CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA. Anna Yoo Jeong Ha et al.

are using one model to generate foreground objects and another to [31] Karen Ho. 2024. Database of 16,000 Artists Used to Train Midjourney AI, Including
create backgrounds [19]. These evolving techniques will continue 6-Year-Old Child, Garners Criticism. https://www.artnews.com/art-news/news/
midjourney-ai-artists-database-1234691955/.
to present challenges for future AI detection systems. [32] Ashish Hooda et al. 2024. D4: Detection of Adversarial Diffusion Deepfakes
Using Disjoint Ensembles. In Proc. of WACV. IEEE.
Acknowledgements [33] Illuminarty. 2023. Is an AI Behind Your Image? https://illuminarty.ai/.
[34] Heon Jae Jeong and Wui Chiang Lee. 2016. The level of collapse we are al-
We thank our anonymous reviewers for their insightful feedback. lowed: comparison of different response scales in safety attitudes questionnaire.
Biometrics Biostatistics International Journal (2016), 128–134.
Sincere thanks also go to the thousands of artists who participated [35] Joseph Saveri Law Firm LLP. 2023. Class Action Filed Against Stability AI,
in our user study. This work is supported in part by NSF grants CNS- Midjourney, and DeviantArt for DMCA Violations, Right of Publicity Violations,
2241303 and CNS-1949650. Opinions, findings, and conclusions or Unlawful Competition, Breach of TOS.
[36] Tero Karras et al. 2020. Analyzing and Improving the Image Quality of StyleGAN.
recommendations expressed in this material are those of the authors arXiv preprint:1912.04958 (2020).
and do not necessarily reflect the views of any funding agencies. [37] Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features
from tiny images. University of Toronto (2009).
References [38] SAND Lab. 2023. Web Glaze. https://glaze.cs.uchicago.edu/webglaze.html.
[39] K. W. K. Lam, W. L. Lau, and Z. L. Li. 1999. Effects of JPEG compression on
[1] Magnific AI. 2023. Magnific. https://magnific.ai. accuracy of image classification. In Proc. of ACRS.
[2] Sina Alemohammad et al. 2023. Self-Consuming Generative Models Go MAD. In [40] W-L Lau, Z-L Li, and KW-K Lam. 2003. Effects of JPEG compression on image
arXiv preprint:2307.01850. classification. Proc. of IJRS (2003).
[3] ANDY BAIO. 2022. Invasive Diffusion: How one unwilling illustrator found [41] Junnan Li et al. 2023. BLIP-2: bootstrapping language-image pre-training with
herself turned into an AI model. frozen image encoders and large language models. In Proc. of ICML.
[4] Quentin Bammey. 2020. Synthbuster: Towards Detection of Diffusion Model [42] Zeyu Lu et al. 2023. Seeing is not always believing: Benchmarking Human and
Generated Images. IEEE Open Journal of Signal Processing 5 (2020), 1–9. Model Perception of AI-Generated Images. arXiv preprint:2304.13023 (2023).
[5] Xiuli Bi et al. 2023. Detecting Generated Images by Real Images Only. arXiv [43] Aleksander Madry et al. 2017. Towards deep learning models resistant to adver-
preprint:2311.00962 (2023). sarial attacks. arXiv preprint:1706.06083 (2017).
[6] Jordan J. Bird and Ahmad Lotfi. 2023. CIFAKE: Image Classification and Explain- [44] Emanuel Maiberg. 2023. AI Images Detectors Are Being Used to Discredit the
able Identification of AI-Generated Synthetic Images. arXiv preprint:2303.14126 Real Horrors of War. 404Media.
(2023). [45] MaxonVFX. 2024. We extend our apologies to the community. https://twitter.
[7] Isabelle Bousquette. 2023. Companies Increasingly Fear Backlash Over Their AI com/MaxonVFX/status/1748826148858208286.
Work. WSJ. [46] Alex Nichol et al. 2022. Hierarchical text-conditional image generation with clip
[8] G. W. Braudaway. 1997. Protecting publicly-available images with an invisible latents. arXiv preprint:2204.06125 (2022).
image watermark. In Proc. of ICIP. [47] Travis Northup. 2023. Wizards of the Coast Repeats Anti-AI Art Stance After
[9] Sergi D. Bray et al. 2023. Testing human ability to detect ‘deepfake’ images of Player’s Handbook Controversy. IGN.com.
human faces. Journal of Cybersecurity (2023), 1–18. [48] Jie Yee Ong. 2023. Scooby-Doo: Daphne Voice Actor Fell Victim To $1,000 AI Art
[10] Martin Briesch, Dominik Sobania, and Franz Rothlauf. 2023. Large Language Scam. The Chainsaw.
Models Suffer From Their Own Output: An Analysis of the Self-Consuming [49] Optic. 2023. AI or Not. https://www.aiornot.com.
Training Loop. In arXiv preprint:2311.16822. [50] Kyle Orland. 2024. Magic: The Gathering Maker Admits it Used AI-generated
[11] Cara. 2023. https://cara.app/. Art Despite Standing Ban. Ars Technica.
[12] Seung Youn Chyung et al. 2017. Evidence-Based Survey Design: The Use of a [51] Susannah Page-Katz. 2023. Introducing Our AI Policy. Kickstarter.com.
Midpoint on the Likert Scale. Performance Improvement (2017), 15–23. [52] Dustin Podell et al. 2023. SDXL: Improving Latent Diffusion Models for High-
[13] Civitai. 2022. What is Civitai? https://civitai.com/content/guides/what-is-civitai. Resolution Image Synthesis. arXiv preprint:2307.01952 (2023).
[14] Linda Codega. 2023. Dungeons & Dragons Updates Bigby to Replace AI-Enhanced [53] Alec Radford et al. 2021. Learning transferable visual models from natural
Images. Gizmodo. language supervision. In Proc. of ICML.
[15] Linda Codega. 2023. New Dungeons & Dragons Sourcebook Features AI Gener- [54] Aditya Ramesh et al. 2022. Hierarchical text-conditional image generation with
ated Art. Gizmodo. clip latents. arXiv preprint:2204.06125 (2022).
[16] Riccardo Corvi et al. 2023. On The Detection of Synthetic Images Generated by [55] Jonas Ricker, Simon Damm, Thorsten Holz, and Asja Fischer. 2023. Towards the
Diffusion Models. In Proc. of ICASSP. Detection of Diffusion Model Deepfakes. arXiv preprint:2210.14571 (2023).
[17] CourtListener. 2024. Andersen v. Stability AI Ltd.(3:23-cv-00201). https://www. [56] Robin Rombach et al. 2022. High-resolution image synthesis with latent diffusion
courtlistener.com/docket/66732129/andersen-v-stability-ai-ltd/. models. In Proc. of CVPR.
[18] Davide Cozzolino et al. 2023. Raising the Bar of AI-generated Image Detection [57] Kevin Roose. 2022. An A.I.-Generated Picture Won an Art Prize. Artists
with CLIP. arXiv preprint:2312.00195 (2023). Aren’t Happy. https://www.nytimes.com/2022/09/02/technology/ai-artificial-
[19] Katie Deighton. 2024. How the Ad Industry Is Making AI Images Look Less Like intelligence-artists.html.
AI. Wall Street Journal. [58] Mia Sato. 2023. How AI art killed an indie book cover contest. The Verge.
[20] Ambra Demontis et al. 2019. Why do adversarial attacks transfer? explaining [59] Christoph Schuhmann et al. 2022. Laion-5b: An open large-scale dataset for
transferability of evasion and poisoning attacks. In Proc. of USENIX Security. training next generation image-text models. arXiv preprint:2210.08402 (2022).
[21] Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion Models Beat GANs on [60] Zeyang Sha, Zheng Li, Ning Yu, and Yang Zhang. 2023. DE-FAKE: Detection and
Image Synthesis. In Proc. of NeurIPS. Attribution of Fake Images Generated by Text-to-Image Generation Models. In
[22] Maggie H. Dupre. 2023. Sports Illustrated Publisher Fires CEO After AI Scandal. Proc. of CCS.
Futurism. [61] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and
[23] Salako Emmanuel. 2023. AI Tools for Combating Deepfakes. https://ijnet.org/ Ben Y. Zhao. 2023. Glaze: Protecting artists from style mimicry by text-to-image
en/story/ai-tools-combating-deepfakes. models. In Proc. of USENIX Security.
[24] Joel Frank et al. 2024. A Representative Study on Human Detection of Artificially [62] "Eons Show". 2024. Eons Show apologies for violating own AI policy. Twitter.
Generated Media Across Countries. In Proc. of IEEE S&P. San Francisco, CA. https://x.com/EonsShow/status/1751327424556544451.
[25] Ethan Gach. 2023. Amazon’s First Official Fallout TV Show Artwork Is an AI- [63] Ilia Shumailov et al. 2023. The Curse of Recursion: Training on Generated Data
Looking Eyesore. Kotaku.com. Makes Models Forget. arXiv preprint:2305.17493 (2023).
[26] Sara Ghazanfari et al. 2023. R-LPIPS: An adversarially robust perceptual similarity [64] Zachary Small. 2023. As Fight Over A.I. Artwork Unfolds, Judge Rejects Copyright
metric. arXiv preprint:2307.15157 (2023). Claim. NY Times.
[27] Paul Glynn. 2023. Sony World Photography Award 2023: Winner Refuses Award [65] Haixu Song, Shiyu Huang, Yinpeng Dong, and Wei-Wei Tu. 2023. Robustness and
After Revealing AI Creation. BBC News. Generalizability of Deepfake Detection: A Study with Diffusion Models. arXiv
[28] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and preprint:2309.02218 (2023).
harnessing adversarial examples. arXiv preprint:1412.6572 (2014). [66] Kenneth R. Spring, John C. Russ, Matthew J. Parry-Hill, and Michael W. Davidson.
[29] Genki Hamano, Shoko Imaizumi, and Hitoshi Kiya. 2023. Effects of JPEG 2016. JPEG Image Compression. National High Magnetic Field Laboratory.
Compression on Vision Transformer Image Classification for Encryption-then- [67] Stability AI. 2022. Stable Diffusion Public Release. https://stability.ai/blog/stable-
Compression Images. Sensors 23, 7 (2023). diffusion-public-release.
[30] Hive. 2023. AI-Generated Content Classification. https://thehive.ai/apis/ai- [68] StabilityAI. 2022. Stable Diffusion v1-4 Model Card. https://huggingface.co/
generated-content-classification. CompVis/stable-diffusion-v1-4.
Organic or Diffused: Can We Distinguish Human Art from AI-generated Images? CCS ’24, October 14–18, 2024, Salt Lake City, UT, USA.

[69] StabilityAI. 2022. Stable Diffusion v1-5 Model Card. https://huggingface.co/

Modified Prompt for Each Style
runwayml/stable-diffusion-v1-5. Modified Prompt:
Style
[70] "Portal Staff". 2023. League of Legends AI-Generated LATAM Anniversary Video added the phrase
Gets Taken Down. ZLeague The Portal. Anime "anime"
[71] Chandra Steele. 2023. How to Detect AI-Created Images. https://www.pcmag. Cartoon "a cartoon style image of"
com/how-to/how-to-detect-ai-created-images. Fantasy "a fantasy style image of"
[72] Stuart A. Thompson and Tiffany Hsu. 2023. How Easy Is It to Fool A.I.-Detection
Oil/Acrylic "an oil and acrylic painting of"
Tools? NY Times.
[73] Sheng-Yu Wang et al. 2020. CNN-generated images are surprisingly easy to spot... Photography "a photography of"
for now. arXiv preprint:1912.11035 (2020). Sketch "a sketch drawing of"
[74] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. 2023. Benchmarking Deepart Watercolor "a watercolor painting of"
Detection. arXiv preprint:2302.14475 (2023). Table 14: Phrases added to BLIP captions extracted from
[75] Zhendong Wang et al. 2023. DIRE for Diffusion-Generated Image Detection. In
Proceedings of ICCV. human arts to correct art styles.
[76] Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. 2023. Tree-
Rings Watermarks: Invisible Fingerprints for Diffusion Images. In Proc. of NeurIPS.
[77] Cam Wilson. 2024. AI is producing ’fake’ Indigenous art trained on real artists’
work without permission. Crickey.com.au.
[78] YentaMagenta. 2023. Hive AI image "detection" is inaccurate and easily defeated.
Reddit.
[79] Lijun Zhang et al. 2024. Robust Image Watermarking using Stable Diffusion.
arXiv preprint:2401.04247 (2024).
[80] Mingjian Zhu et al. 2023. GenImage: A Million-Scale Benchmark for Detecting
AI-Generated Image. arXiv preprint:2306.08571 (2023).

Appendix
Unperturbed Glaze (medium) Glaze (high)
Detection Results of DIRE Models. Table 12 shows the de-
tection results using 6 DIRE checkpoints, using on our dataset of
human artwork and AI-generated images. We test each model with
the same set of 630 unperturbed images: 280 human artworks and
350 AI-generated images.
Additional Results on Hive. Table 13 shows the distribution
of images across art style and generative model, which are “easy-
to-detect” by Hive and unaffected by all five perturbations. JPEG Compression Gaussian Noise Adv. Perturbation
Additional Information on Data Collection. Table 14 lists the
modifications made to the extracted BLIP captions, which are then Figure 9: Samples of five different perturbations considered
used to prompt individual AI generators. Figure 9 shows examples by our study.
of five types of perturbations considered in our study.
Results on Unusual Images. Figure 10 plots, for both hybrid im- General
ages and upscaled human art, the distribution of the “raw” detection
Professional
decision represented by the 5-point Likert score.
Hive
Optic
DIRE model
ACC(%)↑ FPR(%)↓ FNR(%)↓ Illuminarty
Training Generation
0% 20% 40% 60% 80% 100%
dataset model
(a) Hybrid images
ImageNet ADM 54.44 96.43 4.86
LSUN PNDM 54.76 100.00 1.43 General
LSUN StyleGAN 55.40 99.29 0.86 Professional
LSUN ADM 55.08 99.29 1.42 Hive
LSUN iDDPM 54.45 97.86 3.71 Optic
CelebA-HQ SD-v2 51.59 25.36 66.86
Illuminarty
Table 12: Performance of six DIRE checkpoint models on our 0% 20% 40% 60% 80% 100%
dataset. (b) Upscaled human art
human art (very confident) human art (somewhat confident)
CIVITAI DALL-E 3 Firefly MJv6 SDXL Total not sure AI-generated (somewhat confident)
Anime 4 4 9 2 6 25 AI-generated (very confident)
Cartoon 5 6 8 1 6 26
Fantasy 10 4 5 0 6 25 Figure 10: Distribution of detection decision represented by
Oil/Acrylic 6 6 1 0 1 14 the 5-point Likert rating on (a) hybrid images and (b) upscaled
Photography 1 3 0 0 2 6 human artworks.
Sketch 2 1 1 2 3 9
Watercolor 5 5 7 1 6 24
Total 33 29 31 6 30 129
Table 13: The number of AI-generated images for which Hive
is >99% confident across all perturbations.