Deep Fake Detection in Videos

Rajath R Pai Rakeen Harmain Khan

Dept. of Artificial Intelligence and Machine Learning Dept. of Artificial Intelligence and Machine Learning
PES Institute of Technology and Management PES Institute of Technology and Management
Shivamogga, India Shivamogga, India
Email: [email protected] Email: [email protected]

Shreya N S Suchith A
Dept. of Artificial Intelligence and Machine Learning Dept. of Artificial Intelligence and Machine Learning
PES Institute of Technology and Management PES Institute of Technology and Management
Shivamogga, India Shivamogga, India
Email: [email protected] Email: [email protected]

Abstract—The last few decades have seen a significant rise I NTRODUCTION

in Artificial Intelligence (AI) and Machine Learning (ML),
promoting the development of deepfake technology. Deepfakes
A deep fake is an AI-based technology that used deep
are synthetic media created using AI techniques, altering audio, learning technology to create or alter audio, video, and im-
images, and videos to appear authentic but are fabricated. ages to make them appear real; more specifically, it usually
Em ploying concepts like Generative Adversarial Networks requires a neural network that has been trained on vast data
(GANs), deepfake creation involves a competitive process where based on the target person’s voice, facial expressions, and
one model produces forgeries while another aims to identify
them. The consequences of deepfakes are extensive, ranging
mannerisms[1]. Risks of deepfakes: Although deepfakes have
from misinformation campaigns by terrorist organizations to limitless promising avenues in entertainment and education, its
indi vidual harm through fabricated pornography and financial dangers are major; they will support misinformation and mak-
scams. Detecting deepfakes presents a significant challenge, ing non-consensual lewd content based on privacy violations;
prompting exploration into various methodologies. Convolutional fraud and impersonation are supported[2]. Current research is
Neural Networks (CNNs) stand as a prominent tool, with studies
comparing different architectures such as DenseNet, VGGNet,
ongoing into the development of effective detection methods
ResNet, and custom CNNs. Additionally, autoencoders, partic and technologies that might mitigate such risks[3].
ularly Variational Autoencoders, offer utility in reducing data Deepfake technology is a product of the rapid development
dimensionality and editing facial features in images. Trans of artificial intelligence which can be traced back only a
fer learning, where pre-trained models like InceptionV3 are few years ago. Deepfake technology is whereby a person’s
adapted for deepfake detection, and multi-model approaches
likeness is reproduced to create movies or audio content that
combining CNNs with Long-Short Term Memory Networks
(LSTMs) have shown promise. Furthermore, novel techniques are difficult to differentiate from actual footage. However,
like CapsuleNets and ensemble methods, which combine multiple this invention has been abused in several ways and has given
base models, are explored for their efficacy. CapsuleNets facilitate rise to numerous ethical and legal issues[4]. Repeatedly and
spatial inconsistency detection, while ensembling aims to enhance ironically, the absence of ethical standards has permitted
model robustness. Two-stream networks, integrating spatial and
people to create more harm than good using the fantastic tools
frequency streams, offer a solution for handling low-quality
videos. Despite there being various advancements in detection available to them. Drawing on existing literature, the efficacy
models, deepfake detection faces numerous challenges. Adverse of the prevalent deepfake technology will be challenged by
serial attacks targeting detection models, the emergence of new embracing an automated approach for detection and analysis.
deepfake variants evading existing detection mechanisms, and Using state-of-the-art technologies including computer vision,
computational resource requirements hindering real-time imple
machine learning, and forensic analysis, the system aims to
mentation are significant hurdles. Moreover, concerns regarding
privacy, fairness, and biases in training data persist. As deepfake identify false media content accurately[5]. Made up of three
generation technology progresses, detection models may struggle broad components which are technical, legal and ethical, this
to keep pace, requiring ongoing research and adaptation. While project seeks to mitigate the proliferation of deepfake abuse
detection models play a crucial role, a comprehensive approach and safeguard the purity of digital content.
to addressing deepfake-related issues is imperative. This involves
not only refining detection techniques but also fostering public Addressing the Growing Challenge of Deepfake Detection and
awareness, implementing regulatory measures, and promoting Mitigation
responsible usage of AI-generated content. By addressing these
versatile challenges, society can reduce the negative impacts of The generation of deepfakes has raised some noteworthy
deepfake technology. concerns as they can be abused. These videos and audio
Index Terms—Machine learning,Deep fake,Convolution Nureal recordings that are synthesized by an AI can as well be used to
Network,Generative Adversarial Networks relay false information, alter public opinion, meddle with the
electoral process, and create fake details that may incriminate be able to process the content live as it is steamed[22][31][50].
certain people, more so the politicians. Even more, the issue User Interface Development: Social media platforms such as
is compounded by the availability of deepfake production a phone application that will enable every user including
tools that enable people of low-level technical expertise to institutions and even law enforcement agencies to upload
generate credible fake media. It can be noted that the issue videos for deepfake detection will be developed[25][33].
of deepfake detection is harder and harder with every passing Generation process of deep fakes:
day owing to the development of techniques used to create
them. Present technologies that aid in the identification of
also considering legal approaches do not support enough
contemporary DeepFake technology which deals with artificial
and real content merging. This begs the question of how
to build a system capable of detecting deep fake content,
limiting its distribution, and protecting the integrity of the
media in a reality where its security is increasingly threatened
by manipulative schemes.

Developing an Adaptive ML-based Framework

The major goals in the framework of this project are as
follows: Detect deepfakes in video content: Create a solid
mitigation measure that seeks to develop technologies for
the verification of manipulated media with an emphasis on
video forgery detection especially face forgery[6][1]. Face
extraction and forensic analysis: Methodically proceed with
the extraction of faces from the video every second for one
frame and study the forensic content for any possible manipu-
lation of the videos[4][47]. Machine learning uses: Formulate
an optimized model to separate real and fabricated image or
video media utilizing machine learning especially deep learn- Fig. 1. Exploring the Role of GANs in Creating Realistic Media
ing techniques such as CNNs[11][14]. Ensure adaptability: This figure depicts the step-by-step procedure in the making
Develop a system that can be modified and adapted faster than of a deepfake. It begins with Data Collection in which an
the advancement of new, complex deepfake procedures in the enormous volume of the datasets are collected to train the
near future[8][18].Mitigate misinformation: Help in combating neural networks. Then is the Training Phase where the end
misinformation by developing a tool to easily detect such defining is done based on the data available allowing the
media for use in the various sectors including the media, networks to decrease the errors in the quality of output.
government and law enforcement[5]. The Adversarial Process features two competing networks en-
Scope of this project hancing realism—usually in Generative Adversarial Networks
(GANs)—through iterative processes. The next one is the
The scope of this project covers the following areas: Data Refinement stage where the output media images are edited
Collection and Preprocessing: Textual information will be and modified over a period to enhance quality. To wrap it up
obtained from different databases such as newspapers, ethno- is the Output Generation where the production is deepfake as
graphic interviews and field notes. The data was cleaned by the final visual is produced aiming for the highest realistic
scanning through the video clips, locating the facial features quality. Hence each of these steps plays a big role towards the
and cropping them out, changing the angle of these faces, development of photorealistic and realistic deepfakes.
and so on to enhance the data[1][5]. Model Training and
Validation: A CNN model will be trained using the processed Methodology
data which aims at discrimination of the real and the fake 1) Data Preprocessing: Preprocessing includes such simple
faces. Should the efficacy of the model be established using the actions as cutting the faces from the video frames, which is
so collected data[4][9][21]. Integration of Temporal Consis- the first step. To achieve homogeneity across the dataset, video
tency Checks: The system will incorporate consistency in time frames are captured via OpenCV at constant intervals of one
checks which are annotations that search for motion dynamics frame per second. Thereafter, the MTCNN is used to locate
in order to circumvent the deceptions of deepfake synthesis the faces on these frames, where the faces are then scaled and
through checking frames in a video[17][18]. Implementation graduated to the same size[1][2]. In this section, the methods
of RealTime Detection: The first system will be prohibiting the of augmentation that include but are not limited to image
analysis of video content, aiming at only post-processing video flipping, rotation and brightness changes are in order to make
content for the deepfake identification, Whereas subsequent the training data more interesting and increase representative
releases shall focus on building the detection systems that will ability of the model[30][37].
 2) Feature Extraction : The face images, which have process is Video Frame Extraction and involves processing
been preprocessed, are passed into the CNN model, where video files contained in the “Deepfake Detection Challenge”
the specific features forensic manipulations are targeted. It dataset. The videos include both real face and face simulated
encompasses the study of the texture, lighting editing, twisted contents. Each of the frames is extracted and saved in a specific
edges and others that reveal a sign of deepfake media[6][9]. folder that is the main source of data. Then the Resizing
Convolutional layers of the model via multiple filters ir cat- Frames step makes sure that every frame, both the initial
egorically learn how to discriminate between real and fake and any further processed, have the same size and resolution
images. which is important to maintain the consistency of the model
3) Model Training and Validation : The dataset includes training[1][2].Where the facial regions are removed from the
three subsets namely training dataset, validation dataset and standard frames resulting in cropped images which are targeted
test dataset. The CNN model is trained on the training dataset more on the region of interest for deep fake detection[3].
where the model is able to differentiate between real and
2024 Market Analysis and Applications
fake faces based on features extracted. The validation Set is
basically employed for tuning the model and its parameters in
such a way that one does not overfit the model to the training
data. Once the model has been still improved on, the test set is
applied to assess the deepfake detection in terms of accuracy,
precision, and recall[3][20][47].
4) Temporal Consistency Analysis: The other approach
that serves to increase the detection accuracy is temporal
consistency analysis that works by detecting sudden changes
between two consecutive video frames that are unexpected.
There are many deepfake videos which have a lot of artifacts
such as light and shadows and facial features change from
frame to frame, and this step helps to rectify the model from
such changes making the model more accurate in fake media
detection[17][18][19].
Fig. 3. Market Research Report 2024
Frame Extraction through MTCNN Algorithm
The illustration shows the deepfake applications market
analysis for the year 2024 along the vertical line and across
different verticals along the horizontal line. Entertainment has
the highest adoption with 35% first place, which is followed
by Fraud Detection second place at 25% that gives insight into
its application in combating deceit. Cyber security takes over
20%, which underscores the application of the technology in
the solution of security complications. Advertising comes in
at 15% whereby it has been established that this technology
is used to develop interactive adverts and Healthcare is at a
dismal 5%, suggesting that the use of this technology in the
medical sector is still developing and not widely embraced.
Each sphere, in turn, showcases different examples of deepfake
technology commercialization and explains the diverse market
scope of the deepfake technology.
Result and Discussion
Technology for the detection of deepfake is key to ev-
erybody in the contemporary world where content can be
synthesized and used to commit fraud. This project provides
evidence that Machine learning as well as computer vision
techniques can be employed for the purpose of the deep fake
Fig. 2. MTCNN detection[4][5].
By effective pre processing, forensic detection and provision
The figure provides details of the first phase architecture of temporal consistency checks, this system gave satisfactory
of the deep fake detection project, which is based on video results in terms of deepfakes detection. However, deepfake
frames and how the dataset is created. The first step in the technology is constantly evolving, necessitating continuous
improvements to detection techniques. Future work must focus [9] Kharbat, F. F., Elamsy, T., Mahmoud, A., & Abdullah, R. (2019,
on keeping detection systems up-to-date with the latest de- November). Image feature detectors for deepfake video detection. In
2019 IEEE/ACS 16th International Conference on Computer Systems
velopments in deepfake generation while ensuring the system and Applications (AICCSA) (pp. 1-4). IEEE.
remains computationally efficient[7][8]. [10] Caldelli, R., Galteri, L., Amerini, I., & Del Bimbo, A. (2021). Optical
Technology for the detection of deepfake is key to ev- Flow based CNN for detection of unlearnt deepfake manipulations.
Pattern Recognition Letters, 146, 31-37.
erybody in the contemporary world where content can be [11] Shad, H. S., Rizvee, M. M., Roza, N. T., Hoq, S. M., Monirujjaman
synthesized and used to commit fraud. This project provides Khan, M., & Singh, Bourouis S. (2021). Comparative analysis of
evidence that Machine learning as well as computer vision deepfake image detection method using convolutional neural network.
Comput. Intell. Neurosci., 2021.
techniques can be employed for the purpose of the deep fake [12] Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., & Frey, B. (2015).
detection. By effective pre processing, forensic detection and Adversarial autoencoders. arXiv preprint arXiv:1511.05644.
provision of temporal consistency checks, this system gave [13] Li, D., Zhang, M., Chen, W., & Feng, G. (2018, August). Facial attribute
editing by latent space adversarial variational autoencoders. In 2018
satisfactory results in terms of deepfakes detection. However, 24th International Conference on Pattern Recognition (ICPR) (pp. 1337-
deepfake technology is constantly evolving, necessitating con- 1342). IEEE.
tinuous improvements to detection techniques. Future work [14] Suratkar, S., Johnson, E., Variyambat, K., Panchal, M., & Kazi, F. (2020,
July). Employing transfer-learning based CNN architectures to enhance
must focus on keeping detection systems up-to-date with the the generalizability of deepfake detection. In 2020 11th International
latest developments in deepfake generation while ensuring the Conference on Computing, Communication and Networking Technolo-
system remains computationally efficient. gies (ICCCNT) (pp. 1-9). IEEE.
[15] Sanghvi, B., Shelar, H., Pandey, M., & Sisodia, J. (2021, April). De-
tection of machine generated multimedia elements using deep learning.
In 2021 5th International Conference on Computing Methodologies and
Communication (ICCMC) (pp. 1238-1243). IEEE.
[16] Nasar, B. F., Sajini, T., & Lason, E. R. (2020, December). Deepfake
detection in media files-audios, images and videos. In 2020 IEEE Recent
Advances in Intelligent Computational Systems (RAICS) (pp. 74-79).
IEEE.
[17] Guera, D., & Delp, E. J. (2018, November). Deepfake video detection
using recurrent neural networks. In 2018 15th IEEE International Con-
ference on Advanced Video and Signal Based Surveillance (AVSS) (pp.
1-6). IEEE.
[18] Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., & Natara-
jan, P. (2019). Recurrent convolutional strategies for face manipulation
detection in videos. Interfaces (GUI), 3(1), 80-87.
[19] Mehra, A. (2020). Deepfake detection using capsule networks with long
short-term memory networks (Master’s thesis, University of Twente).
[20] Bonettini, N., Cannas, E. D., Mandelli, S., Bondi, L., Bestagini, P., &
Tubaro, S. (2021, January). Video face manipulation detection through
ensemble of CNNs. In 2020 25th International Conference on Pattern
Recognition (ICPR) (pp. 5012-5019). IEEE.
[21] Kohli, A., & Gupta, A. (2021). Detecting deepfake, faceswap and
face2face facial forgeries using frequency CNN. Multimedia Tools Appl.,
80, 18461-18478.
Fig. 4. Train Data Distribution [22] Liang, Y., Wang, M., Jin, Y., Pan, S., & Liu, Y. (2023). Hierarchical
Supervisions with Two-stream Network for Deepfake Detection. Pattern
Recognit. Lett..
[23] Li, C., Huang, Z., Paudel, D. P., Wang, Y., Shahbazi, M., Hong, X.,
& Van Gool, L. (2023). A continual deepfake detection benchmark:
R EFERENCES Dataset, methods, and essentials. In Proceedings of the IEEE/CVF
Winter Conference on Applications of Computer Vision (pp. 1339-
[1] Rana, M. S., Nobi, M. N., Murali, B., & Sung, A. H. (2022). Deepfake 1349).
detection: A systematic literature review. IEEE Access, 10, 25494-25513. [24] Neekhara, P., Dolhansky, B., Bitton, J., & Ferrer, C. C. (2021).
[2] Wubet, W. M. (2020). The deepfake challenges and deepfake video Adversarial threats to deepfake detection: A practical perspective. In
detection. Int. J. Innov. Technol. Explor. Eng, 9. Proceedings of the IEEE/CVF Conference on Computer Vision and
[3] Chadha, A., Kumar, V., Kashyap, S., & Gupta, M. (2021). Deepfake: Pattern Recognition (pp. 923-932).
an overview. In Proceedings of Second International Conference on [25] Gowrisankar, B., & Thing, V. L. (2024). An adversarial attack approach
Computing, Communications, and Cyber-Security: IC4S 2020 (pp. 557- for eXplainable AI evaluation on deepfake detection models. Computers
566). Springer Singapore. Security, 139, 103684.
[4] Singh, A., Saimbhi, A. S., Singh, N., & Mittal, M. (2020). DeepFake [26] Xiao, S., Lan, G., Yang, J., Lu, W., Meng, Q., & Gao, X. (2023).
video detection: a time-distributed approach. SN Comput. Sci, 1, 212. MCSGAN: A Different Understanding for Generalization of Deep
[5] Katarya, R., & Lal, A. (2020, October). A study on combating emerging Forgery Detection. IEEE Trans. Multimedia.
threat of deepfake weaponization. In 2020 Fourth International Confer- [27] Campbell, C., Plangger, K., Sands, S., Kietzmann, J., & Bates, K. (2022).
ence on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC) How deepfakes and artificial intelligence could reshape the advertising
(pp. 485-490). IEEE. industry: The coming reality of AI fakes and their potential impact on
[6] Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega- consumer behavior. J. Advert. Res., 62(3), 241-251.
Garcia, J. (2020). Deepfakes and beyond: A survey of face manipulation [28] Liu, J., Zhu, K., Lu, W., Luo, X., & Zhao, X. (2021). A lightweight 3D
and fake detection. Information Fusion, 64, 131-148. convolutional neural network for deepfake detection. Int. J. Intell. Syst.,
[7] Akhtar, Z. (2023). Deepfakes Generation and Detection: A Short Survey. 36(5), 2313-2329.
Journal of Imaging, 9(1), 18. [29] Prashanth, K., & Kundu, H. (2021). An adaptive neural network ap-
[8] Mirsky, Y., & Lee, W. (2021). The creation and detection of deepfakes: proach for deepfake detection. J. Ambient Intell. Humaniz. Comput., 12,
A survey. ACM Comput. Surv., 54(1), 1-41. 5437-5445.
[30] Bondi, L., Cannas, E. D., Bestagini, P., & Tubaro, S. (2020, December). [53] Yang, J., Li, A., Xiao, S., Lu, W., & Gao, X. (2021). MTD-Net:
Training strategies and data augmentations in CNN-based deepfake Learning to detect deepfakes images by multi-scale texture difference.
video detection. In 2020 IEEE International Workshop on Information IEEE Trans. Inf. Forensics Security, 16, 4234-4245.
Forensics and Security (WIFS) (pp. 1-6). IEEE. [54] Xu, Y., Zhang, D. D., & Sun, C. (2023, June). Frequency domain
[31] Suratkar, S., Kazi, F. (2023). Deep fake video detection using transfer deepfake detection based on two-stream neural network. In Fourteenth
learning approach. Arabian J. Sci. Eng., 48(8), 9727-9737. International Conference on Graphics and Image Processing (ICGIP
[32] Ramadhani, K. N., Munir, R., & Utama, N. P. (2024). Improving 2022) (Vol. 12705, pp. 910-918). SPIE.
Video Vision Transformer for Deepfake Video Detection using Facial [55] BR, S. R., Pareek, P. K., Bharathi, S., & Geetha, G. (2023, February).
Landmark, Depthwise Separable Convolution and Self Attention. IEEE Deepfake Video Detection System Using Deep Neural Networks. In
Access. 2023 IEEE International Conference on Integrated Circuits and Com-
[33] Yu, Z., Cai, R., Li, Z., Yang, W., Shi, J., & Kot, A. C. (2024). munication Systems (ICICACS) (pp. 1-6). IEEE.
Benchmarking joint face spoofing and forgery detection with visual and [56] Hu, Y., Zhang, X., & Ren, J. (2023). Automatic detection of deepfake
physiological cues. IEEE Trans. Dependable Secure Comput.. videos using convolutional neural networks in frequency domain. Mul-
[34] Leyva, R., Sanchez, V., Epiphaniou, G., & Maple, C. (2024). Detect- timedia Tools Appl., 82(2), 1827-1847.
ing Face Synthesis Using a Concealed Fusion Model. arXiv preprint [57] Heidari, A., Jafari Navimipour, N., Dag, H., Unal, M. (2024). Deepfake
arXiv:2401.04257. detection using deep learning methods: A systematic and comprehensive
[35] Nawaz, M., Javed, A., & Irtaza, A. (2023). ResNet-Swish-Dense54: a review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge
deep learning approach for deepfakes detection. The Visual Computer, Discovery, 14(2), e1520.
39(12), 6323-6344.
[36] Killi, C. B. R., Balakrishnan, N., & Rao, C. S. (2023). Deep Fake
Image Classification Using VGG-19 Model. Ingénierie des Systèmes
d’Information, 28(2).
[37] Waqas, N., Safie, S. I., Kadir, K. A., Khan, S., & Khel, M. H. K. (2022).
DEEPFAKE image synthesis for data augmentation. IEEE Access, 10,
80847-80857.
[38] Zhang, T., Deng, L., Zhang, L., & Dang, X. (2020, August). Deep
learning in face synthesis: A survey on deepfakes. In 2020 IEEE 3rd
International Conference on Computer and Communication Engineering
Technology (CCET) (pp. 67-70). IEEE.
[39] Guarnera, L., Giudice, O., Guarnera, F., Ortis, A., Puglisi, G., Paratore,
A., ... & Battiato, S. (2022). The face deepfake detection challenge.
Journal of Imaging, 8(10), 263.
[40] Fischer, E. M. Formulation of the Variational Autoencoder and Evidence
Lower Bound and an Application to the MNIST Dataset.
[41] Zhao, Y., & Linderman, S. (2023, July). Revisiting structured variational
autoencoders. In International Conference on Machine Learning (pp.
42046-42057). PMLR.
[42] Remya Revi, K., Vidya, K. R., & Wilscy, M. (2021). Detection of
Deepfake Images Created Using Generative Adversarial Networks: A
Review. In Second International Conference on Networks and Advances
in Computational Technologies: NetACT 19 (pp. 25-35). Springer Inter-
national Publishing.
[43] Ganguly, S., Mohiuddin, S., Malakar, S., Cuevas, E., & Sarkar, R.
(2022). Visual attention-based deepfake video forgery detection. Pattern
Anal. Appl., 25(4), 981-992.
[44] Hooda, A., Mangaokar, N., Feng, R., Fawaz, K., Jha, S., & Prakash,
A. (2024). D4: Detection of Adversarial Diffusion Deepfakes Using
Disjoint Ensembles. In Proceedings of the IEEE/CVF Winter Conference
on Applications of Computer Vision (pp. 3812-3822).
[45] Ganiyusufoglu, I., Ngo, L. M., Savov, N., Karaoglu, S., & Gevers, T.
(2020). Spatio-temporal features for generalized detection of deepfake
videos. arXiv preprint arXiv:2010.11844.
[46] Walia, S., Kumar, K., & Kumar, M. (2023). Unveiling digital image
forgeries using Markov based quaternions in frequency domain and
fusion of machine learning algorithms. Multimedia Tools Appl., 82(3),
4517-4532.
[47] Guera, D., & Delp, E. J. (2018, November). Deepfake video detection
using recurrent neural networks. In 2018 15th IEEE International Con-
ference on Advanced Video and Signal Based Surveillance (AVSS) (pp.
1-6). IEEE.
[48] Han, X. Y., & Lewis, A. S. (2020). Disk matrices and the proximal
mapping for the numerical radius. arXiv preprint arXiv:2004.14542.
[49] Coccomini, D., Caldelli, R., Falchi, F., & Gennaro, C. (2023). On the
Generalization of Deep Learning Models in Video Deepfake Detection.
Journal of Imaging, undefined.
[50] Wang, Y., Wang, J., Liang, Y., Chen, X., & Guo, Y. (2023). Deepfake
Video Detection via Facial Action Dependencies Estimation.
[51] Prasad, S. S., et al. (2022). Human vs. Automatic Detection of Deepfake
Videos Over Noisy Channels. In 2022 IEEE International Conference
on Multimedia and Expo (ICME). IEEE.
[52] Lin, Y., et al. (2020). Exploring generalization capability for video
forgery and detection based on generative adversarial network. In 2020
International Conference on Computational Science and Computational
Intelligence (CSCI). IEEE.