International Conference on Communication and Signal Processing, July 28 - 30, 2020, India

A Survey on Human Activity Recognition and

Classification
Abhay Gupta, Kuldeep Gupta, Kshama Gupta and Kapil Gupta


Abstract—Activity Recognition and Classification is one of the This review paper surveys extensively the current progress
most significant issues in the computer vision field. Identifying made towards various activity recognition methods. Moreover,
and recognizing actions or activities that are performed by a we provide a comprehensive analysis of three techniques,
person is a primary key goal of intelligent video systems. Human
activity is used in a variety of application areas, from human-
namely wearable devices-based approach, pose based
computer interaction to surveillance, security, and health approach, and smartphone sensors. The first one uses sensing
monitoring systems. Despite ongoing efforts in the field, activity devices to be mounted on the subject to collect data from the
recognition is still a difficult task in an unrestricted environment sensors [2] whereas smartphone sensor based approach takes
and faces many challenges. In this paper, we are focusing on some input from the smartphone sensors such as gyroscope and
recent research papers on various methods of activity recognition. accelerometer [3] and in the last one activity is classified using
The work includes three popular methods of recognizing activity,
namely vision-based (using pose estimation), wearable devices, pose estimation which require the estimation of body key
and smartphone sensors. We will also discuss some pros and cons points through neural network [4]. HAR system can be
of the above technologies and take a view on a brief comparison subdivided into three levels, as follows in Fig. 1.
between their accuracy. The findings will also show how the The rest of the paper is structured as: Section II contains
vision-based approach is becoming a popular approach for HAR the related work of recent research papers in the field. Section
research these days.
III briefly describes the various methods and techniques used
in human activity recognition. A comparative study of
Index Terms—Deep learning, human activity recognition
(HAR), pose estimation, smartphone sensors, wearable sensors. different methods and their accuracy is given in Section IV;
Section V contains the discussion and our findings; Some
concluding remarks and future scope for further discussions
I. INTRODUCTION are given in Section VI.

A CTIVITY refers to the movement of the entire body or the

different positions of the limbs relative to time against
gravity. Human Activity Recognition (HAR) becomes a very
popular and active research area for researchers from the last
two decades. However, it still remains a complex task due to
some unresolvable issues such as sensor movement, sensor
placement, background clustering, and the inherent variability
of how different people perform activities. Determining
detailed activities is beneficial in many areas of human-centric
applications, such as home care support, postoperative trauma
rehabilitation, abnormal activities, gesture detection, exercise,
and fitness. Most of the person's daily tasks can be simplified
or automated if recognized by the HAR system. Usually, HAR
systems are based on either unsupervised or supervised
learning. A supervised system requires pre-training using
special datasets, but unsupervised systems have a set of rules
during development [1].

Abhay Gupta, Kuldeep Gupta, Kshama Gupta, Kapil Gupta are with Fig. 1. The overview of general HAR system. This is basic building blocks
Department of Computer Applications, National Institute of Technology, of almost every system.
Kurukshetra, India. (phone: +918979578203 e-mail:
[email protected] [email protected]
[email protected] [email protected]).

978-1-7281-4988-2/20/$31.00 ©2020 IEEE 0915

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 11:00:30 UTC from IEEE Xplore. Restrictions apply.
 II. RELATED WORK
In recent years, recognition and understanding of human
behaviour have received much attention. Different techniques
have been used to understand patterns of behaviour activity
and to understand the scene. In this work, we have reviewed
some selected papers from 2016-2019 on HAR. Our study
provides a brief analysis of HAR methods. Ghazal et al. [5]
proposed a pose based HAR, it uses openpose library and
forward feed CNN to predict a confidence map of 18 key
points and further uses the decision-making algorithm to
classify standing and sitting activity. Tsai et al. [6] presented a
system for detecting the activities using a Kinect sensor, and
11 kinds of activities are recognized through discrete Hidden
Markov models (HMMs). The system is developed for training
the robot. Gatt et al. [7] proposed an approach for detecting Fig. 2. Classification of HAR system based on their approaches.
abnormal behaviour such as fall detection. The work uses
PoseNet and Openpose pre-trained pose estimation model, multi-class classification problem that can also be modeled by
using neural networks. CNN accepts the body joints location
then Long Short-Term Memory (LSTM ) and CNN are used
as input and generates a number vector representing the
for activity classification.
probability of each activity labels accordingly. Some popular
Bulbul et al. [3] recognize human activity by using different
dataset is also available such as MPII, which contain more
classification and machine learning approaches such as than 20,000 labelled images of 410 specific subcategories
Bagging, k-NN, etc. For this, they have used two smartphone activities under 20 activity categories [10].
sensors, accelerometers, and gyroscope and recognize 6
different activities. Tran et al. [8] proposed a three smartphone B. Smartphone Sensor- Based
sensors-based approach for HAR. they used SVM for the Smartphones are the most useful tool in our everyday lives,
classification and identification of activity and optimized the and advanced technology is enabling us to meet the needs and
classification model to identify the activities. RoyChaudhury et expectations of customers every day. To make these devices
al. [9] used a single smartphone sensor for HAR. they use more functional and powerful, designers are adding new
different classifiers to test the proposed model like trees modules and devices to the hardware. Sensors enhance the
(Complex tree). They considered 12 activities, including static capabilities of smartphones and play a major role in
and dynamic activities, for his work. understanding the environment. As a result, most smartphones
have a variety of built-in sensors that can collect a wealth of
III. VARIOUS APPROACHES TO HAR useful data about the human’s daily life.
To achieve the goal of recognizing human activity, a HAR Sensors retrieve information from body gestures and then
system is required. The two most commonly used techniques recognized the activities. The most commonly used sensors are
for this purpose are sensor-based and vision-based activity accelerometer and gyroscope etc. Accelerometer sensor is used
recognition. We can classify them, as shown in Fig. 2. for measures the change in speed, and gyroscope is used to
measures the orientation of the body. Some techniques of
A. Pose Based Approach HAR through smartphones used in recent studies are SVM, k-
Poses are important for analyzing videos, which include NN, Bagging, Ada Boost [3].
humans, and there is strong evidence that body posture
concepts are very effective for various tasks such as activity C. Wearable Sensor Based
recognition, content extraction, etc. This approach classifies The wearable technique uses sensing devices to be mounted
human actions based on the coordinate information of the body on the subject to collect data from the sensors. As human
parts. Basically, HPE refers to the process of assessing the activity contains actions of different bodily positions, the
composition of a part of the human body (3D poses) or the research of human activity needs to capture information from
projection onto an image plane (2D HPE). It covers all issues more than one sensor installed on the different parts of the
related to the human body, from understanding the entire body of the person. Wearable devices must be designed with
human body to the detailed localization of body parts [4]. It is user accessibility in mind. Lightweight, modern, and
formulated as a regression problem that can be modelled with comfortable wearing devices with embedded sensors are used
a simple CNN. It takes the entire image as input and shows the for activity monitoring.
pixel coordinates of the body's key points. There are 15 body Activity monitoring sensors are used in multiple datasets.
joints: Neck, LKnee, LAnkle, RShoulder, RWrist, Relbow, The most commonly used sensors are an accelerometer,
LShoulder, RHip, LElbow, LWrist, Chest, and 14 joint gyroscope, magnetometer, and RFID tag [11]. After feature
connections. The classification problem can be formulated as a

0916

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 11:00:30 UTC from IEEE Xplore. Restrictions apply.
 TABLE I
COMPARATIVE ANALYSIS OF HUMAN ACTIVITY RECOGNITION ALGORITHM BASED ON PREVIOUS STUDIES

S.No. AUTHORS FEATURES REFINE DATASET ACTIVITIES MODEL USED ACCURACY (%)
AND YEAR FEATURES

1. SVM 99.4
Smartphone sensor Noise is filtered Dataset Recognize 6
(accelerometers and using median and collected from different activities k-NN 97.1
Bulbul et al., gyroscope) to 20Hz 9 individuals like walking,
2018 [3] recognize human Butterworth filter (2947 records sitting, climbing
activities and then 3Hz with 561 up the stairs, etc. AdaBoost 97.4
Butterworth filter features)
to filter the result Bagging 98.1
2.
Three smartphone Not available Data collected Recognize 6 Support Vector
Tran et al., 2016 sensors from 10 different activities Machine (SVM) 89.59
[8] (accelerometers, volunteers like walking,
gyroscope, and contains sitting, upstairs,
accelerometer sensor 10,939 samples etc.
linearity) used to
recognize human
activities
3. SVM 89.5
Smartphone sensor Median filter and They have Recognize 12
RowChowdhury (accelerometer) used Butterworth filter collected their different activities k-NN (k=1) 90.9
et al., 2018 [9] to recognize human to removes the own dataset like Sit on floor,
activities low-frequency Climbing upstairs, Ensemble 94.2
acceleration etc. (Bagged Trees)
(gravity) and
noise from 91.7
accelerometer Complex Tree
signal.
4. Linear Regression 53.92
Smartphone inertial Butterworth filter Recognize 6
Nandy et al., accelerometer and used for to filter Collected their different activities
Decision Tree 93.54
2019 0 heart rate sensor to noise own dataset like sitting,
recognize human walking, running,
activities. lying down, Multilayer 77.07
standing, climbing Perceptron
stairs, etc. Gaussian Nave 73.73
Bayes
5. J48 98.4
Khokhlov et al., Accelerometer and Not available Recognize three
2018 [2] gyroscope sensor Collected own activities, such as
Naïve Bayes 76.2
mounted on the dataset sitting, walking,
body standing, etc.
Random Forest 95.0

KNN (k=1) 93.6

6.
Ghazal et al., 18 body key points Openpose library Random Recognize Feedforward CNN 95.2
2018 [5] obtained used to extract images (146) activities such as and decision-
features downloaded Sitting on the making algorithm
from the ground or chair,
internet and standing.

7.
Tsai et al., 2017 Vector Quantization Not available Images for the 11 different kinds Vector 95.64
[6] is used to reduce the dataset from of activities for Quantization -
noise through Kinect sensor training robots. Hidden Markov
clustering. (275 samples) Model (VQ-
HMMs)

8.
Gatt et al., 2019 PoseNet and Not available COCO dataset Recognize Semi-supervised 93
[7] Openpose pre- abnormal activity LSTM and CNN
trained models used such as fall for classification
detection

0917

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 11:00:30 UTC from IEEE Xplore. Restrictions apply.
extraction and modelling, human activities can be recognized do not interfere with privacy. Despite having many benefits of
through statistics, and machine learning algorithm is applied. wearable sensors still, it has some drawbacks. HAR carrier
How to map low-level sensor data to higher-level abstractions systems typically require multiple sensors to be worn or
is the key to activity recognition. mounted on different parts of the body, which is troublesome,
disturbing, and uncomfortable for the subject [11].
IV. COMPARATIVE STUDY In terms of universal context, there is no specific indicator
In this section, a comparative analysis of various activity or measurement that indicates whether wearable sensors are
recognition system is taken out based on the literature study better than smartphone sensors or vice versa. Both sensors are
we have done so far. In Table I. we have shown accuracies of expected to have advantages and disadvantages depending on
different machine learning algorithms used in the developing the subject and the purpose of use. Therefore, researchers and
HAR system. developers need to figure out the subjects and their use before
using any HAR technology [12-14].
V. FINDINGS AND DISCUSSION
VI. CONCLUSION AND FUTURE SCOPE
Fig. 3. shows the graph of techniques used in HAR in recent
years. Our review results depict that both smartphone and A review has been completed on some selected research
wearable sensor technology are common in HAR research. On papers published in 2016-2019 on various HAR technologies.
the other hand, the pose-based approach is not so popular in We have categorized these technologies into three main
the early days, probably due to the limitation of scenes and categories, namely HAR using pose estimation (vision-based),
human movements in 3D space. In addition, detecting and smartphone sensors, and wearable sensors. From our study, it
extracting people from image sequences is another limitation is found that the emergence of wearable technology has
that requires sophisticated machining. Therefore, when large become a better solution for providing support services to
volumes of data are processed at once, real-time HAR system people. However, the system still has some limitations. Some
can achieve better results. The pose is estimated by cameras actions have low recognition rates. Further research is needed
so, another concern raised by these HAR systems is privacy to improve accuracy and increase the number of activities
issues. A person can be uncomfortable or forced to look at the detected by the system. Also, we noticed that vision-based
watch at all times. Though it is useful to recognize any approach is not much popular among the three in last two
activities with video cameras, most of all, they need decades in spite of having the better results due to its
infrastructure support as it needs to install cameras in limitations but in the upcoming years with the advancement in
surveillance places and is highly dependent on illumination the technology machines with high computational power is
[9]. made available easily, which are capable of processing large
Talking about other technologies, smartphones are easy to amount of data within less time vision-based approach will
carry and are used in everyday life, so it gives an edge to use become a great choice for HAR.
them for activity monitoring. Although there are some
REFERENCES
[1] O. C. Ann and L. B. Theng, "Human activity recognition: A review,"
2014 IEEE International Conference on Control System, Computing and
Engineering (ICCSCE 2014), Batu Ferringhi, 2014, pp. 389-393.
[2] I. Khokhlov, L. Reznik, J. Cappos and R. Bhaskar, "Design of activity
recognition systems with wearable sensors," 2018 IEEE Sensors
Applications Symposium (SAS), Seoul, 2018, pp. 1-6.
[3] E. Bulbul, A. Cetin and I. A. Dogru, "Human Activity Recognition
Using Smartphones," 2018 2nd International Symposium on
Multidisciplinary Studies and Innovative Technologies (ISMSIT),
Ankara, 2018, pp. 1-6
[4] Bearman, Amy L., Stanford and Catherine Dong. “Human Pose
Estimation and Activity Classification Using Convolutional Neural
Networks.” (2015).
[5] S. Ghazal and U. S. Khan, "Human posture classification using skeleton
information," 2018 International Conference on Computing,
Mathematics and Engineering Technologies (iCoMET), Sukkur, 2018,
pp. 1-4.
[6] A. Tsai, Y. Ou, C. Sun and J. Wang, "VQ-HMM classifier for human
activity recognition based on R-GBD sensor," 2017 International
Fig. 3. HAR papers published in recent years. Conference on Orange Technologies (ICOT), Singapore, 2017, pp. 201-
204.
practical limitations of working with real-time data from the [7] T. Gatt, D. Seychell and A. Dingli, "Detecting human abnormal
app, such as the type of smartphone used, sensor frequency, behaviour through a video generated model," 2019 11th International
smartphone location, etc. On the other side, wearable devices Symposium on Image and Signal Processing and Analysis (ISPA),
Dubrovnik, Croatia, 2019, pp. 264-270.
are becoming popular because they are easier to implement,
[8] D. N. Tran and D. D. Phan, "Human Activities Recognition in Android
available at low cost, operate in harsh environments, and also Smartphone Using Support Vector Machine," 2016 7th International

0918

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 11:00:30 UTC from IEEE Xplore. Restrictions apply.
 Conference on Intelligent Systems, Modelling and Simulation (ISMS), [12] S. N. Boualia and N. Essoukri Ben Amara, "Pose-based Human Activity
Bangkok, 2016, pp. 64-68 Recognition: a review," 2019 15th International Wireless
[9] I. RoyChowdhury, J. Saha and C. Chowdhury, "Detailed Activity Communications & Mobile Computing Conference (IWCMC), Tangier,
Recognition with Smartphones," 2018 Fifth International Conference on Morocco, 2019, pp. 1468-1475.
Emerging Applications of Information Technology (EAIT), Kolkata, [13] A. Das Antar, M. Ahmed and M. A. R. Ahad, "Challenges in Sensor-
2018, pp. 1-4 based Human Activity Recognition and a Comparative Analysis of
[10] A. Singh, S. Agarwal, P. Nagrath, A. Saxena and N. Thakur, "Human Benchmark Datasets: A Review," 2019 Joint 8th International
Pose Estimation Using Convolutional Neural Networks," 2019 Amity Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd
International Conference on Artificial Intelligence (AICAI), Dubai, International Conference on Imaging, Vision & Pattern Recognition
United Arab Emirates, 2019, pp. 946-952. (icIVPR), Spokane, WA, USA, 2019, pp. 134-139.
[11] R. Liu, T. Chen and L. Huang, "Research on human activity recognition [14] M. Ahmed, A. Das Antar and M. A. R. Ahad, "An Approach to Classify
based on active learning," 2010 International Conference on Machine Human Activities in Real-time from Smartphone Sensor Data," 2019
Learning and Cybernetics, Qingdao, 2010, pp. 285-290. Joint 8th International Conference on Informatics, Electronics & Vision
(ICIEV) and 2019 3rd International Conference on Imaging, Vision &
A. Nandy, J. Saha, C. Chowdhury and K. P. D. Singh, "Detailed Human Pattern Recognition (icIVPR), Spokane, WA, USA, 2019, pp. 140-145.
Activity Recognition using Wearable Sensor and Smartphones," 2019
International Conference on Opto-Electronics and Applied Optics
(Optronix), Kolkata, India, 2019, pp. 1-6.

0919

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 11:00:30 UTC from IEEE Xplore. Restrictions apply.