Scribd
Upload
2 views25 pages

Capstone Presentation

Uploaded by

laplluve
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views25 pages

Capstone Presentation

Uploaded by

laplluve
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 25

DeepFake Image

Detection
By
Jeff Spagnola
Let’s Play a Game
Can you determine which of these images isn’t real?
Let’s Play a Game
Can you determine which of these images isn’t real?

Trick question...NONE of them are real!


Introduction
⩥ Image editing technology has improved
to the point where it’s nearly impossible
to tell what’s real.

⩥ Deepfakes have already become a


serious issue on social media, in politics,
and in society at large.

⩥ Goal of this project is to create a system


that can tell the difference between a
real image and a high quality deepfake.

⩥ Deepfake Image Detection can be used


in social media companies, security
organizations, and news agencies. You can test out the Deepfake Detection App at the following URL:
<INSERT URL HERE>
4
1. The Process
Steps Taken in this Project
OSEMN PRocess
Throughout this project, we will be following the OSEMN Data Science Process

6
2. The Data
Obtaining & Cleaning the Data
The Data
⩥ Dataset of images was obtained
by combining several collections
of real and deepfake images.

⩥ Criteria for the images is that


they had to be high quality & at
least 150px square.

⩥ 142, 286 images in total.

⩥ Nearly an equal distribution of


real and deepfake images.

8
Scrubbing
⩥ Import the folders of images
⩥ Rescale & Resize images

⩥ Define the target classes:


1 = real, 0 = deepfake

⩥ Convert the image into an array


(series of numbers)

⩥ Create training, test, and


validation sets for modeling

9
4. Modeling
Convolutional Neural Network
⩥ Image is scanned by a deep
neural network
⩥ Convolutional (Conv2d) layer
analyzes groups of pixels in
sequence

⩥ Convolutional “weights” are fed


into pooling layers, dense layers,
and normalization layers.

⩥ Experimented with a finely tuned


CNN, pretrained CNN & an
ensemble of both.
11
Tuned CNN
⩥ Iterated through many
combinations of layers and
parameters.
⩥ Able to achieve a 92% Accuracy
⩥ Achieved 92% weighted Recall

⩥ Faster training time than


Pretrained CNN.

12
Pretrained CNN
⩥ Used a pretrained CNN (Xception)
as a convolutional base
⩥ Able to achieve a 96% Accuracy
⩥ Achieved 96% weighted Recall

⩥ Slowest training time among the


models we used.

13
Ensemble CNN
⩥ Combined the layers of the Tuned
CNN & Pretrained CNN into a new
model.
⩥ Able to achieve a 97% Accuracy
⩥ Achieved 97% weighted Recall

⩥ Slow training time plus this model


required added processing of the
data.

14
Models By the Numbers
Accuracy Weighted Recall Training Time

Tuned CNN 92% 92% 1:16:48

Pretrained CNN 96% 96% 2:39:08

Ensemble CNN 97% 97% 0:27:28


(plus 1:16:48 & 2:39:08)

The pretrained & ensemble CNNs both had higher scores but the training & loading time of
the pretrained CNN makes it a difficult choice for deployment for the Deepfake Detection
App. The ensemble model requires an extra step in preprocessing and does add a bit of
loading time in the final app. This is something we’re still experimenting with. 15
THe App
Test the functionality of the app here.
5. Interpret
What have we learned?
How does this Work?
Earlier, we mentioned the CNN works by scanning layers. Below is
an original image, a single layer of a CNN and multiple layers of a
CNN.

18
Lime Explainer
The Lime Package gives us additional insight into how the model is making
predictions. We can see that the model seems to “notice” the area around the eyes.

19
Results
⩥ Tuned CNN - 92% accuracy
Pretrained - 96% accuracy
Ensemble - 97% accuracy
⩥ Pretrained networks are a valuable
tool, but only when retraining
parameters.
⩥ Ensemble model is fastest & most
accurate, but requires the training
time of previous models.

⩥ Lime Explainer shows that eyes are


a focal point for making a prediction.
20
6. Recommendations
How to Proceed
RecoMMEndations
⩥ For pretrained networks, retrain the
parameters
⩥ Use an ensemble of tuned CNN &
Pre-trained CNN for for highest accuracy
⩥ For model deployment, use a finely
tuned CNN for speed & solid accuracy
⩥ The Deepfake Image Detection App is
recommended for Social Media
Companies to weed out bots & fake
profiles.
22
Future Work
More Data Expand the Scope
Add additional images to Modify the app to be
the dataset. More data = able to scan for video
higher accuracy. files as well as images..

More User-Friendly Adapt for Poor Quality


Update the associated Expand the capability of
app to be more the model to account for
user-friendly and have poor quality fake images.
more image
classification features.

23
thanks! @spags093
[email protected]

Any questions?

24
Appendix
For those interested in the original datasets that were sampled to
make the dataset for this project, links can be found below:

⩥ 140k Real and Fake Faces


⩥ Deepfake_faces
⩥ Real and Fake Face Detection

Additional Resources:
⩥ This Person Does Not Exist
⩥ Generated Photos
25

You might also like