Real-Time Anomaly Detection and Classification from Surveillance Cameras using

Deep Neural Network

The author of this paper presents a method for real-time anomaly detection and classification
from surveillance camera footage using a fine-tuned ResNet-50 model. The approach begins
with the creation of a dataset containing 10,483 real-world anomalous images, which
encompass 14 different types of anomalies, such as robbery and road accidents. The model
operates by preprocessing the images and converting them into 3D cubes, which are then fed
into the ResNet-50 architecture. This architecture is enhanced by incorporating an average
pooling layer, dropout layer, and dense layers, followed by a softmax activation function,
allowing the model to effectively learn and classify the anomalous patterns present in the
data. The results demonstrate that the model achieves a remarkable 100% accuracy in
detecting anomalies and an average classification accuracy of 79.69%, with a computational
cost of 61.45 milliseconds per frame. However, the authors acknowledge certain limitations,
including prediction flickering, where the predicted labels may vary across classes in the
output video. They suggest that future work could focus on improving model stability and
accuracy, as well as expanding the dataset to include a wider variety of anomalous events,
thereby enhancing the robustness of real-time anomaly detection systems [1].

Anomaly Detection in Videos Using Optical Flow and Convolutional Autoencoder

The author of this paper proposes a framework called OF-ConvAE-LSTM for detecting
anomalies in video surveillance systems. This model integrates Convolutional Autoencoder
(ConvAE) and Convolutional Long Short-Term Memory (ConvLSTM) networks to analyze
video data in an unsupervised manner. The framework begins with a feature extraction stage
that utilizes dense optical flow to capture the velocity and direction of foreground objects,
which is crucial for understanding motion patterns. The ConvAE is responsible for learning
spatial features, while the ConvLSTM captures temporal dependencies across video frames,
allowing the model to effectively learn the dynamics of normal activities and identify
deviations that signify anomalies .

The authors conducted experiments on three well-known public datasets: Avenue, UCSD
Ped1, and UCSD Peds2. The results demonstrated that the proposed framework could
accurately model the complex distribution of regular motion patterns, outperforming existing
state-of-the-art methods based on both unsupervised and semi-supervised deep learning
approaches. This indicates a significant advancement in the field of anomaly detection in
videos, particularly in environments where real-time analysis is critical .

However, the paper also acknowledges certain limitations. The performance of

trajectory-based approaches tends to degrade in crowded scenes, where detecting and
tracking multiple moving objects becomes challenging . Additionally, while the framework
shows promise, the authors suggest that future work could focus on enhancing the model's
robustness in complex environments and exploring the integration of additional features or
alternative learning strategies to further improve anomaly detection accuracy .[2]
Enhancement of Video Anomaly Detection Performance Using Transfer Learning and
Fine-Tuning

The author of this paper proposes an efficient frame-level video anomaly detection (VAD)
method that leverages transfer learning (TL) and fine-tuning (FT) techniques. The approach
utilizes 20 popular convolutional neural network (CNN)-based deep learning models,
including variants of VGG, Xception, MobileNet, Inception, EfficientNet, ResNet, DenseNet,
NASNet, and ConvNeXt. The models are trained using TL and FT to enhance their
performance in detecting anomalies in video streams. The methodology involves extracting
features from video frames and then applying the trained models to identify unusual events,
which is crucial for surveillance applications.
The experiments are conducted on three datasets: CUHK Avenue, UCSD Ped1, and UCSD
Ped2. The performance of the models is evaluated using various metrics, including area under
the curve (AUC), accuracy, precision, recall, and F1-score. The results indicate that the
proposed method achieves impressive AUC scores of 100%, 100%, and 98.41% for the
UCSD Ped1, UCSD Ped2, and CUHK Avenue datasets, respectively. These results
demonstrate that the suggested method offers state-of-the-art performance in VAD compared
to existing techniques in the literature.

However, the paper does not explicitly mention limitations or future work, which could
include exploring additional datasets, improving model robustness against different types of
anomalies, or integrating real-time processing capabilities. Overall, the study highlights the
effectiveness of using TL and FT in enhancing VAD performance, paving the way for further
advancements in this field .[3]

Deep Learning-Based Abnormal Behavior Detection for Elderly Healthcare Using

Consumer Network Cameras

The author of this paper presents a deep learning-based approach for detecting abnormal
behaviors in the elderly, utilizing consumer network cameras. The method is designed to
identify typical abnormal behaviors such as falls, aggression, and wandering, which are
critical for elderly healthcare. The model leverages a deep learning architecture that robustly
extracts skeleton joints and classifies abnormal behaviors while considering both spatial and
temporal contexts, addressing limitations found in conventional methods that struggle with
model generalization and coherence . The dataset consists of images captured by fixed
network cameras positioned in typical elderly activity areas, which are processed by a local
server equipped with powerful GPUs for real-time monitoring and alarming .

The results indicate that the proposed method achieves a competitive mean Average Precision
(mAP) greater than 85%, demonstrating its effectiveness in detecting and localizing various
anomalies . However, the system does face limitations, including some false detections and
challenges in distinguishing normal activities from abnormal patterns . Future work may
focus on refining the model to reduce false positives and enhance its ability to differentiate
between normal and abnormal behaviors more accurately, potentially incorporating more
diverse datasets and advanced techniques to improve overall performance .[4]
A New Unsupervised Video Anomaly Detection Using Multi-Scale Feature
Memorization and Multipath Temporal Information Prediction

The author of this paper presents a novel approach to unsupervised video anomaly detection
using a model called MsMp-net, which integrates multi-scale feature memorization and
multipath temporal information prediction. The model operates on a U-Net-like architecture
that employs a time-distributed 2D CNN-based encoder and decoder, allowing it to
effectively learn and reconstruct normal patterns in video data. During training, the model
utilizes a memory module to store relevant prototypical patterns of normal scenarios, which
aids in distinguishing anomalies during inference. The method leverages dilated convolutions
to extract contextual information across multiple scales, enhancing the model's ability to
recognize varied-size objects in the scene. The authors evaluated their approach on
benchmark datasets, specifically UCSD Ped1, UCSD Ped2, and CUHK Avenue,
demonstrating that their model outperforms many existing methods in terms of anomaly
detection accuracy. However, the study also identifies limitations, such as the model's
sensitivity to noise in the error maps, which can hinder performance, particularly when
detecting small anomalies. Future work is suggested to focus on improving the anomaly
scoring function by incorporating inter-frame dependencies and exploring adaptive
techniques for hyperparameter adjustments to enhance model robustness across different
datasets. [5]

Video Anomaly Detection with Sparse Coding Inspired Deep Neural Networks

The author of this paper presents a novel approach to video anomaly detection using a model
inspired by sparse coding and deep neural networks. They introduce a Temporally-coherent
Sparse Coding (TSC) framework, which incorporates a temporally-coherent term to maintain
similarity between similar frames, optimizing sparse coefficients through a Sequential
Iterative Soft-Thresholding Algorithm (SIATA). This optimization leads to a special stacked
Recurrent Neural Networks (sRNN) architecture, which is further enhanced by stacking an
additional layer to form a stacked Recurrent Neural Network Auto-Encoder (sRNN-AE) for
input reconstruction. The method aims to model the distribution of normal patterns using only
normal data during training, allowing the detection of anomalies based on reconstruction
errors. The authors conduct extensive experiments on both a toy dataset and real-world
datasets, including ShanghaiTech Campus, CUHK Avenue, and UCSD Ped2, demonstrating
that their method significantly outperforms existing techniques in terms of anomaly detection
accuracy. They also build a large-scale anomaly detection dataset that surpasses existing
datasets in volume and scene diversity. However, the authors acknowledge limitations in their
approach, such as the reliance on the quality of the training data and the challenge of
detecting rare anomalies. They suggest future work could focus on improving the model's
robustness to various types of anomalies and exploring more efficient training methods to
enhance performance in real-time applications. [6]

Vehicular Trajectory Classification and Traffic Anomaly Detection in Videos Using a

Hybrid CNN-VAE Architecture
The author of this paper proposes a novel approach for traffic anomaly detection using a
hybrid model that combines Convolutional Neural Networks (CNN) and Variational
Autoencoders (VAE). The methodology involves a semi-supervised labeling technique based
on a modified Dirichlet Process Mixture Model (mDPMM) for clustering trajectories, which
prepares training data for classifying vehicular paths and detecting anomalies. The model
works by encoding spatio-temporal information of vehicular trajectories into a color gradient
representation, allowing for effective classification and anomaly detection. The authors
utilize a dataset that includes trajectories captured from videos recorded by a static camera,
which helps in identifying various anomalies such as lane violations and sudden speed
changes. The results indicate that the hybrid CNN-VAE architecture performs well in
classifying trajectories and detecting unforeseen vehicular anomalies, with the CNN classifier
achieving higher accuracy compared to other methods. However, the authors acknowledge
certain limitations, such as the dependency on accurate tracking of vehicles and the necessity
for a large number of training samples to effectively learn the allowed paths at traffic
junctions. They suggest that future work could focus on improving tracking methods and
exploring additional datasets to enhance the model's robustness and applicability in
real-world scenarios. [7]

Deep learning based anomaly detection in real‑time video

The author of this paper proposes a system for automatic anomaly detection in real-time
video surveillance using a combination of Inflated 3D Convolution Network (I3D-ResNet50)
and deep Multiple Instance Learning (MIL). The model works by treating video snippets as
packets, where regular and unusual videos are classified as negative and positive packets,
respectively. Each video snippet is evaluated to generate an anomaly score through a fully
connected Neural Network (NN). The authors utilize the UCF-101 dataset, which contains
130 GB of videos featuring 13 abnormal events, such as fighting and stealing, alongside
normal events. Their experimental results demonstrate an Area Under Curve (AUC) score of
82.85% after only 10,000 iterations, indicating that their model is effective in identifying
anomalies in real-time videos. However, the authors acknowledge limitations in their
approach, particularly in the subjective nature of defining abnormal behavior, which can vary
significantly among individuals. They suggest that future work could focus on enhancing the
model's ability to generalize across different contexts and improving its robustness against
false anomaly warnings, which can occur when the model misclassifies normal behavior as
anomalous. [8]

Analysis of anomaly detection in surveillance video: recent trends and future vision
The author of this paper conducts a comprehensive analysis of anomaly detection (AD) in
surveillance video, focusing on recent trends and future directions. They utilize various
machine learning (ML) and deep learning (DL) techniques to enhance the detection of
anomalous activities in crowded environments. The methods discussed include Principal
Components Analysis for feature extraction, Gaussian Mixture Models, and U-Net centered
frameworks for improving AD performance. The U-Net model employs a bidirectional
prediction mechanism, where both forward and backward predictions are made to enhance
accuracy in detecting anomalies, particularly in complex scenarios with occlusions and
clutter.
The paper reviews multiple datasets and performance metrics used in existing studies,
highlighting the challenges posed by real-world applications, such as the ability to generalize
across different environments and the robustness to unexpected events . The results indicate
that while the proposed techniques show promise, they often perform less reliably on real
data compared to synthetic benchmarks. Limitations include the difficulty in accurately
identifying individual behaviors in large crowds and the need for methods to be tested in
more varied and uncontrolled environments.

Future work is suggested to address these limitations, focusing on improving the scalability
of the models and their adaptability to diverse scenarios. The author emphasizes the
importance of developing robust systems that can effectively operate in real-life conditions,
moving beyond controlled settings to ensure practical applicability in surveillance systems.

Robust Unsupervised Video Anomaly Detection by Multipath Frame Prediction

The author of this paper proposes a novel unsupervised video anomaly detection framework
called ROADMAP, designed to enhance performance across various scenarios. The model
incorporates multipath ConvGRUs, which effectively manage informative parts of different
scales and capture temporal relationships between frames while minimizing attention to static
and background elements. To address the interference from noisy pixels in video frames, a
noise tolerance loss is introduced, significantly improving the robustness and performance of
the anomaly detection predictions. The authors conduct experiments on three publicly
available video anomaly detection datasets, demonstrating the superior performance of
ROADMAP compared to several state-of-the-art baselines, achieving an AUROC score of
88.3% on the Avenue dataset and 76.6% on ShanghaiTech, with notable improvements over
previous methods.
The results indicate that the multipath structure and the ConvGRU components effectively
capture both spatial and temporal dependencies, focusing on informative parts of the video
while excluding irrelevant static objects . However, the authors acknowledge limitations in
their approach, particularly regarding the performance on grayscale and low-resolution
videos, where the noise tolerance loss was less effective . Future work may involve refining
the model to better handle such challenging video conditions and exploring additional
datasets to further validate the robustness of the proposed method. Overall, the findings
highlight the potential of ROADMAP in advancing video anomaly detection techniques,
paving the way for further research in this area.
TransAnomaly: Video Anomaly Detection Using Video Vision Transformer
The author of this paper proposes a novel approach for video anomaly detection called
TransAnomaly, which integrates the Video Vision Transformer (ViViT) with a U-Net
architecture. This model aims to enhance the prediction of future frames by capturing richer
temporal information and global contexts. The method works by modifying the ViViT to
make it suitable for video prediction, allowing it to effectively model the complex dynamics
of video data. The authors evaluate their model using benchmark datasets, specifically
focusing on the impact of different window sizes and strides in their calculations of regularity
scores, which help in identifying anomalies. The results demonstrate that TransAnomaly
outperforms existing state-of-the-art prediction-based methods in video anomaly detection,
showcasing its effectiveness in both anomaly detection and localization by tracking patches
with lower regularity scores.

However, the paper also acknowledges certain limitations, such as the model's dependency on
the choice of window size and stride, which can affect performance across different datasets.
Future work may involve exploring more adaptive methods for selecting these parameters or
enhancing the model's robustness against varying background complexities and object sizes
in different scenes . Overall, the findings indicate a promising direction for utilizing
transformer-based architectures in video analysis tasks, paving the way for further
advancements in anomaly detection methodologies. [11]

Anomaly Detection in Aerial Videos With Transformers

The author of this paper proposes a novel Transformer-based network for video anomaly
detection, marking a significant advancement in the field. The model operates by learning
feature representations of normality from normal data, which allows it to identify anomalies
based on large reconstruction errors during testing. This unsupervised method is particularly
effective as it does not require prior knowledge of specific anomalies, focusing instead on the
stable nature of normality . The authors create an annotated dataset consisting of 37 training
videos and 22 testing videos that encompass seven realistic scenes, providing a diverse range
of anomalous events for evaluation.

The results demonstrate that their approach outperforms existing methods, achieving the best
Area Under the Curve (AUC) results in various scenarios, including distinguishing irregular
behaviors of vehicles and pedestrians . However, the authors acknowledge limitations, such
as the lack of public ground-truth labels for anomalies in the AU-AIR-Anomaly dataset,
which necessitated manual labeling of certain events . Future work may involve expanding
the dataset further and refining the model to enhance its robustness and accuracy in detecting
a wider array of anomalies in different contexts. Overall, this paper contributes valuable
insights and methodologies to the ongoing research in video anomaly detection.

Enhancing Video Anomaly Detection Using a Transformer Spatiotemporal Attention

Unsupervised Framework for Large Datasets
The author of this paper introduces an unsupervised framework for video anomaly detection
that combines a vision transformer (ViT) with a convolutional spatiotemporal relationship
(STR) attention block. This model is designed to address the challenges of detecting
anomalies in video surveillance by effectively capturing both local and global relationships
within video frames, which traditional convolutional neural networks (CNNs) often struggle
to achieve. The proposed method utilizes a pre-trained ViT for feature extraction, which is
then enhanced by the STR attention block to better identify spatiotemporal relationships
among objects in videos. The model was evaluated on three benchmark datasets:
UCSD-Ped2, CHUCK Avenue, and the larger ShanghaiTech dataset, demonstrating its
capability to detect anomalies in large and heterogeneous datasets .
The results indicate that the model achieved impressive area under the receiver operating
characteristic curve (AUC ROC) values of 95.6, 86.8, and 82.1 for the respective datasets,
showcasing its superior performance compared to state-of-the-art methods. Additionally, the
model was tested on a subset of the Charlotte Anomaly Dataset (CHAD), achieving AUC
ROC values of 71.8 and 64.2 for different camera views, further validating its effectiveness in
handling large datasets .

However, the paper acknowledges some limitations, such as the model's reliance on large
datasets for optimal performance, which may not always be available in real-world scenarios.
Future work could focus on improving the model's performance on smaller datasets and
exploring the integration of additional features or techniques to enhance anomaly detection
accuracy in diverse environments .

[1] Rahman, M.M., Afrin, M.S., Atikuzzaman, M. and Rahaman, M.A., 2021, December.
Real-time anomaly detection and classification from surveillance cameras using Deep Neural
Network. In 2021 3rd International Conference on Sustainable Technologies for Industry 4.0
(STI) (pp. 1-6). IEEE.
[2] Duman, E. and Erdem, O.A., 2019. Anomaly detection in videos using optical flow and
convolutional autoencoder. IEEE Access, 7, pp.183914-183923.
[3] Dilek, E. and Dener, M., 2024. Enhancement of Video Anomaly Detection Performance
Using Transfer Learning and Fine-Tuning. IEEE Access.
[4] Zhang, Y., Liang, W., Yuan, X., Zhang, S., Yang, G. and Zeng, Z., 2023. Deep learning
based abnormal behavior detection for elderly healthcare using consumer network
cameras. IEEE Transactions on Consumer Electronics.
[5] Taghinezhad, N. and Yazdi, M., 2023. A new unsupervised video anomaly detection using
multi-scale feature memorization and multipath temporal information prediction. IEEE
Access, 11, pp.9295-9310.
[6] Luo, W., Liu, W., Lian, D., Tang, J., Duan, L., Peng, X. and Gao, S., 2019. Video anomaly
detection with sparse coding inspired deep neural networks. IEEE transactions on pattern
analysis and machine intelligence, 43(3), pp.1070-1084.
[7] Santhosh, K.K., Dogra, D.P., Roy, P.P. and Mitra, A., 2021. Vehicular trajectory
classification and traffic anomaly detection in videos using a hybrid CNN-VAE architecture.
IEEE Transactions on Intelligent Transportation Systems, 23(8), pp.11891-11902.
[8] Elmetwally, A., Eldeeb, R. and Elmougy, S., 2024. Deep learning based anomaly
detection in real-time video. Multimedia Tools and Applications, pp.1-17.
[9] Raja, R., Sharma, P.C., Mahmood, M.R. and Saini, D.K., 2023. Analysis of anomaly
detection in surveillance video: recent trends and future vision. Multimedia Tools and
Applications, 82(8), pp.12635-12651.
[10] Wang, X., Che, Z., Jiang, B., Xiao, N., Yang, K., Tang, J., Ye, J., Wang, J. and Qi, Q.,
2021. Robust unsupervised video anomaly detection by multipath frame prediction. IEEE
transactions on neural networks and learning systems, 33(6), pp.2301-2312.
[11] Yuan, H., Cai, Z., Zhou, H., Wang, Y. and Chen, X., 2021. Transanomaly: Video
anomaly detection using video vision transformer. IEEE Access, 9, pp.123977-123986.
[12] Jin, P., Mou, L., Xia, G.S. and Zhu, X.X., 2022. Anomaly detection in aerial videos with
transformers. IEEE Transactions on Geoscience and Remote Sensing, 60, pp.1-13.
[13] Habeb, M.H., Salama, M. and Elrefaei, L.A., 2024. Enhancing Video Anomaly
Detection Using a Transformer Spatiotemporal Attention Unsupervised Framework for Large
Datasets. Algorithms, 17(7), p.286.