IMPROVED GAIT PATTERN ANALYSIS USING AUGMENTED LONG SHORT-

TERM MEMORY IN NEURAL NETWORK FOR CLASSIFICATION

Dr. Jose Reena K

Assistant Professor,
Department of Computer Science,
School of Computing Science,
Vels Institute of Science, Technology & Advanced Studies
Chennai, India
[email protected]

Abstract:

The study aims to classify the gait patterns from elderly people using machine learning
algorithm that determines the healthy and unhealthy states of individual. The analysis is
conducted based on the variables available in the dataset. The classification is conducted to
identify the individuals with gait and machine learning is performed to increase the accuracy
of pattern classification. In this paper, a Long Short-Term Memory is combined with Recurrent
Neural Network for the classification of healthy and unhealthy people among the patients with
Gait. The simulation is conducted to test the efficacy of the model with conventional machine
learning models on Parkinson disease datasets. The results of simulation show that the
proposed method obtains improved prediction accuracy than the other hybrid machine learning
models. The results of simulation show an improved accuracy and f-measure using the
proposed Augmented Long Short-Term Memory (ALSTM) Neural Network model.

Keywords:

Recurrent Neural Network, Long Short-Term Memory, Classification, Healthy People

1. Introduction

An increase in the number of devices like smartphones connected with newer networking
technology called Internet of Things (IoT) leads to increasing generation of data [1,2].
Smartphones play a key role among these devices as their functionality and user acceptance
are continuously expanding. Security must therefore be one of the primary issues in the creation
of those gadgets. Biometrics are part of the latest approaches used to prevent unwanted access
to mobile devices. This includes the analysis and assessment of several physical attributes or
behaviours of a person, to be recognised or identified.

Over time and with the development of technology, biometric identification is progressed.
However, there are other biometric variations which are less obtrusive. The recognition of
people based on their gait characteristics is an example of this. This identification is dependent
and its advantages are the possibility of periodic, automatic and non-intrusive identification,
because only the device needs to be transmitted or logged in to make a computer view
identification.

The study may point out as disadvantages that biometrics are less accurate than other
techniques, such as fingerprints. The continuous and periodic identification referred to above
can solve this problem. If the data sample used for the detection of the user forms a long enough
interval of users' walking, always trying to take as little time as possible to achieve a correct
identification, the effectiveness of Gait-based identification is boosted [3].

In this paper, a Long Short-Term Memory is combined with Recurrent Neural Network for the
classification of healthy and unhealthy people among the patients with Gait. This data is axially
recorded and processed to determine a vertical acceleration, as it simplifies the data and makes
it possible to identify the device irrespective of the position of the device. The database is of
open accessed one with 15 individuals, who carry out various activities to track and collect data
with the smartphones that are largely connected to their bodies. It is crucial to note that
Smartphone has been freely executed in various situations without any constraints.

2. Related Works

The database that is being used is open access [3]. It is a comprehensive database of fifteen
individuals who carry out various activities to track and collect data with the devices and
smartphones that are largely connected to their bodies. It is crucial to note that smartphones
have been freely executed in various situations without any constraints.

In past research studies, many mechanisms and approaches have been suggested and verified
to identify using gait characteristics. The mechanisms can be grouped into three main families
using gait surveillance and characterisation technologies: visual vision, floor, and wearable
methods on the basis of sensors [4]. Wearable sensors do not have any obstacles to computer
vision methods like occlusions, camera field of view/angle, or lighting deficiencies, and are
now commonly incorporated into mobile devices that provide scenarios for recognition [4].

The complete review of current person-identification techniques, based on the gait

characteristics in [5] has been carried out. Biometric characteristics are classified as soft and
hard by the authors. Hard ones usually permit adequate recognition of subjects, yet they suffer
from distinct character issues. Either soft features identify particular groups as a whole or lack
adequate permanency. While gait is regarded as a soft feature, in the past decade the accuracy
of algorithms has grown considerably.

In earlier research projects, numerous methods have been taken to challenge the identification
of individuals based on time series recorded from wearable sensor devices such as
accelerometers and gyroscopes. The study in [7] suggested a gastric algorithm of identification
based on a novel time-consuming similarity founding similarities in wearable sensor data. The
higher resemblance might be offered as the recognised user if the temporal dynamics of
collected sensor signals were compared to previous labelled data. For this purpose, several
metrics have been used.

A solution based on the similarities based on frequency domain, correlation and statistical
distribution of data obtained from the three-dimensional accelerometer on the subject waist
was proposed by authors in [8]. The study in [9] suggested a method of identifying with peak
distance, histogram, correlation, and moments of first order. The study in [10] used time
domain data for a dynamic time warp (DTW). For restricted sensor device execution, similarity
calculations were optimized [11] and specialised for specific need users, like elderly people
[12].

The machine learning algorithms uncovers specific patterns in data which are small in intra-
user data variability and large in user differences. Deep learning algorithms with wearable
sense data have also recently been employed for person identification to uncover complex
patterns. In [18], the authors study the use of a CNN to extract data features that are being fed
into a single-class support vector machine for user verification.

Deep CNN structure in [19] is used to attach detection probabilities to each of the users in the
dataset with a softmax function in fully connected layers. The research paper in [20] contains
a thorough examination of the construction and training of a repeating neural convolution
network with an authentic dataset based on the measurement of five body sensors. The five
body sensor time series is loaded into a convoluted neural network of 2 1D convolutional layers
and a gated recurrent units (GRU). Two layers are employed for sub-sampling features and a
final soft-max function is utilised to assess the possibility of capturing the grip of a particular
user for each data sample.

In [21], the authors use deep neural network, which is based on both CNN and LSTM. In
addition to the learning framework, the research analyses the data of free walking phases.
LSTM can model time-dependent patterns and has also been employed in [22] for optimum
gait-and user-recognition extraction. However, numerous manual tuning procedures have been
applied that limit the generalisation.

In the present article, we examine and optimise LSTM architectures for the identification of
gait for wild users. When compared to [21], which studies deep learning technologies for
uncontrolled walking segments, we are able to increase the accuracy of the results by about
4%.

3. Proposed Methods

This section discusses the acceleration data pre-processing procedure and provides details on
how and what architecture is utilised to identify the deeply recurrent neural network
optimization, which provides an insight into the specific patterns of each user features.

3.1.Data Pre-Processing

The database used consists of 30 individuals who carry out various tasks when monitoring and
data collection using the sensors and gadgets linked to themselves. The database is open access
[3]. In this part, the analysis of free walking data will be a priority. The smartphone used for
activity and activity capturing was conducted in various locations, without any limits. Samples
for various activities can be found and the sensor can be implanted in various regions of the
organism. When the sensor is put in the pocket/smell area, we select the samples that
correspond to the activity (walking) since it simulates a common case in daily life.

3.2.Vertical Acceleration Estimation

The initial step in the pre-processing of the data is to calculate the vertical acceleration
component. This approach involves estimating the gravitational force direction in a window of
samples. It means that the vector of seriousness is calculated at each instant, utilising the
preceding samples at the moment the estimate is calculated.

3.3.Sample Packaging
Packaging of samples means the acceleration samples are clustered for a specified user activity
time. If a single speed sample were used to identify users, the neural network would be virtually
impossible to identify users. A 150 samples was used for the general construction of this article
in order for each sample package to cover about 3 s of the user journey. We have been working
to keep the identification as brief as feasible and to minimise large capture times, which always
maintaining high accuracy as the key goal.

3.4.Data Labeling

Each user has a numerical identifier on each sample. This enables a user to know who belongs
to each vertical acceleration sample. The samples are labelled so it can be allocated to the user
when the samples have been packaged into windows. This is important if the neural model is
to be developed and validated. There was also a categorisation of the label. This approach
involves a one-hot encoding so that a vector with a value between 0 and 1 represents the label.

As many neurons as subjects may be recognised in the database will be the last layer of the
neural network. The last layer activation function will be softmax to allow the output to be a
vector with the same form and length as the categorisation for each sample. The estimate that
the LSTM makes on the samples may therefore be obtained and assigned to the appropriate
user after the inverse transformations have been carried out in the categorisation.

To finish, the data set must be divided into training and testing to complete a comprehensive
model training and evaluation. In order to do so, 70% of the data is randomly divided for
education and 30% considered for testing. The training data is utilised to create the neural
network and how each user is identified when introducing the samples. The test data will be
used to validate and test its efficiency that the system did not use during training and that are
consequently fresh to the neural network after the whole model has been built.

4. Proposed Architecture

The LSTM cells identification is carried out using a deep recurrent neural network (Figure 1).
RNNs networks are designed to learn time series patterns. In order to characterise the attitude
of different people, research in this publication has been centred on the analysis of architectures
to carry out user identification.

In summary, LSTM uses memory units to record values of the short and long past data that
respond to time patterns. Each LSTM cell consists of a status cell.
 • The status cell can be viewed over several instants as a continuous data flow. The study
chosen what information is retained, changed, and allowed to pass on at any time.

• Entrance Door: checks if the status cell is able to enter new information. It also
determines the amount of this new information in the status box.

• Exit Door: Checks when the memory information is being used for the output, and how
much information is being provided from the status cell at output.

• Forget door: Check if data is forgotten. This allows for important data to be created
and less relevant data to be removed. This door job is to determine whether or not the
information is stored or discarded in the status cell.

Input Output
i O
gate Gate

Cell

g Ct

Input
Modulation
Gate
f Forget gate

Figure 1: Single Cell

4.1. Proposed Architecture

A LSTM-layer connected with dense layers and a Softmax output layer are used in the
architecture proposed in order to associate the likelihood of each user input data window. In
order to maximise the accuracy of results, the different network settings are modified between
runs.

For each layer, the information is:

The first two layers of the packet input data are based on LSTM cells, made of LSTM cells. As
the samples are packaged in 150 sample windows, the input samples will be processed by 150
LSTM cells every time. The first shows ideal outcomes for 50 or 100 memory units, with
superior outcomes for 100 memory units on the second LSTM layer. The recurrent units reflect
the internal state and are able to learn time-based patterns based on prior state and current inputs
and forget them. Therefore, the ideal number of memory units depends on the time complexity
dependence of patterns.

Then two hidden layers, with 100 and 90 neurons each, are stacked and completely connected.
These layers are activated by ReLU, since it delivers rapid convergence that is especially
valuable for deep neural networks.

Finally, a layer comprises the number of neurons you want to separate from the number of total
users. As there are 15 users in the database, 15 neurons will exist. These neurons are activated
with softmax. Because this is a multi-class classification issue, a probability distribution across
probable K-subjects can be obtained with this activation function. Network design and
development have been done in Python 3.

For network training, the following parameters of interest were used:

• Categorical Cross Entropy-Loss Function: It helps in classifying the multi-class data.

• Adam Optimizer: It delivers rapid convergence based on the use of gradient estimates
of first and second minutes, adjusting the learning rate to each neural weight.

• Categorical Accuracy: Accuracy in multi-class classification issues for sample

identification.

LSTM

The LSTM (Figure 1b) network is a serial interconnection between separate portals, allowing
the output to be controlled by the inputs and repeated units. This part gives the specifics and
regularises the deep LSTM by means of a regulatory structure. This section sets the time steps
for the whole subscriptions and then defines the layers over n-dimensional states.

At a certain stage t, the state of a layer l is considered hidden. The LSTM works with an affine
function Tn,m :  N → M . Furthermore, the whole procedure htl  N is performed by an
element-wise multiplication ⊙ by means of a vector input ht0 at the appropriate time step k.

The LSTM model takes into account the activation function htL for output yt, with L in deep

LSTM as the total layers.

Definite transitions from previously hidden states to the present can be used to describe RNN
dynamics. A characteristic is defined as a deterministic change in the hidden state as follows:

RNN : htl −1 , htl−−11 → htl

The hidden state function on layer l is stated as follows for the traditional LSTM:

For conventional RNNs, this function is offered.

htl = f (Tn ,n htl −1 + Tn ,n htl−1 )

f { tanm, sigm}

The LSTM works with a sweeping dynamic which makes it easy for long-range information to
be specified. The memory is saved as a cell vector in LSTM. Even if several LSTM
architectures differ in connection and active structure, explicit memory cells are employed to
store long-term data for all LSTM architectures. The LSTM can determine if a memory cell is
to be overwritten, retrieved or maintained. In our experiments, the architecture of the LSTM is
as follows:

The LSTM contains complex dynamics that allow for a long time to read information easily.
A cell vector stores the memory long-term. While the activation structure and connectivity of
a number of LSTM designs vary, explicit memory cells are used to store data for all
architectures for a lengthy time. The LSTM advises that the existing values in the memory cell
be retrieved, overwritten, or maintained for the following step. The LSTM architecture used in
our studies is described as follows:

LSTM : htl −1 , htl−1 , ctl−1 → htl , ctl

 i   sigm 
     htl −1 
 f  =  sigm  T  
 o   sigm  2 n,4 n  htl −1 
   
 g   tanh 
 ctl = f  ctl−1 + i  g

htl = o  tanh ( ctl )

The purpose of the LSTM employed for the study is to reduce the overfitting in LSTMs. The
idea is the usage for repeating connections of the drop-out operator (D) rather than for other
connections, as in figure 3. More precisely, what is to be set with a zero argument is explained
in the following.

 i   sigm 
     D ( htl −1 ) 
 f  =  sigm  T  
 o   sigm  2 n,4 n  hl −1 
     t 
  
g tanh 

Operator D tends, in its intermediate evaluations, to corrupt the unit data and to make them
more robust. Not all the unit data for intermediate estimates is removed at the same time. In
particular, it is crucial to keep in mind what has happened numerous times in the past.

Yt-2 Yt-1 Yt Yt+1 Yt+2

Dropout
points

LSTM Data

Xt-2 Xt-1 Xt Xt+1 Xt+2

Figure 2a: Multilayer Regularized LSTM

Yt-2 Yt-1 Yt Yt+1 Yt+2

Xt-2 Xt-1 Xt Xt+1 Xt+2

 Figure 2b: Information flow in LSTM

Figure 2 a and Figure 2b shows how well the event data tends to occur in the hidden layer
process from step t−2 to t+2. It is shown that the data is L+1 times damaged by the Drop-off
operator, and that the total number of data permits is regarded independent. Drop-out operator
D interrupts the repetitive and recurrent connections to RNN such that LSTM finds it difficult
to hold data for longer periods of time. The LSTM takes advantage of controlling decreases
without compromising storage capacity by preventing frequent dropout connections.

5. Performance Evaluation

We have tried to optimise the hidden layers and memory cells using different setups and
settings.

5.1.Precision Test

Figure 3 shows some of the results and tests carried out with the changes in the proposed design,
with the loss (MSE) demonstrated with different configurations in a test set:

(a)
 (b)

(c)

Figure 3: Mean Square Loss (MSE)

The precision of all settings is quite good, above 96%. In addition, testing was carried out in
which acceleration are captured by a smart phone within the pocket and not on the breast. In
general, this sort of data includes high noise levels due to unintentional movements, and is also
less accurate than data from a user who walks with the smartphones.
 5.2.Confusion Matrix

This part contains the training graphs and the confusion matrix using ALSTM described in the
preceding section. The average accuracy of the model is assessed in each session. In this
scenario, throughout the training phase, you can witness the evolution and model convergence.
In this case, 5% of the training data were also chosen, and new validation data that had not
been used for training were prepared. It aims to see the real development of training accuracy
on data not utilised for training. The real estimated accuracy value is not equal to this.

For the training set and validation set, both the loss and accuracy at each time are provided.
Precision is believed to mean if the model sample predictions that helps in matching the actual
label of the sample. Loss is considered as a scalar value which is usually minimised, as a lower
number means that the value forecast by the model is accurate and matches real sample labels.
In this situation, it is used to calculate the loss value through the categorical loss function.

The faster convergence with 100 memory units of LSTM cells. Moreover, stability can be
achieved with high accuracy rates in both designs, but longer training periods have been
necessary to develop a model durable and capable of achieving precision levels such as in
earlier tables.

The confusion matrix is an instrument to measure the classifier performance. It is a table

showing the estimated values of the neural network and actual values. Figure 4 shows the
results of a Training and Testing Loss for various activities. The accuracy achieved in this
specific example is 97.4%.
(a)

(b)
 (c)

Figure 4: Training and Testing Loss for various activities

User identification has already been demonstrated for walking activities to work well. This is
a most obvious case for every user. In order to verify the efficiency of the model in other
situations, we have, however, conducted other trials in which activities differ from those that
capture the accelerometer patterns. The original database was used, and the data for the
activities: climbing, jumping and running are chosen. For consistency with the remaining trials
conducted in this project, the files that match the activity capture using a device inserted into
the thigh or pocket were chosen.

The predicted acceleration on the gravity axis will be calculated after processing axial
information in the files. A precision test as per the different preceding parts was carried out to
confirm that the provided models are good. A set of training data and a test set were chosen
that retained 70% of the data. The workout consisted of 75 batch intervals of 32 samples.

A precision above 85% is reached for the numerous scenarios indicated in the table to allow
users to be properly determined. 150 samples segments are used as in prior tests. Relatively
good results in terms of precision are obtained without modifying an architecture originally
developed for walking. This highlights the capacity of ALSTM and, in particular, architectures
built to adapt to the user various physical actions in multiple scenarios, without needing to
carry out a single activity depending on the user.
6. Conclusions

This paper shows that ALSTM can autonomously classify user gait parameters and specific
features with a precision of more than 0.98 when walking in the wild. The results outdo prior
characterization investigations in uncontrolled situations under hybrid deep neural architecture,
where the accuracy lesser than 0.94. Although the accuracy of the method provided in this
publication exceeds marginally other earlier studies under controlled conditions, it permits each
user to perform the walking activity freely.

The manuscript shows how to optimise some of the main architectural factors that produce
optimal accuracy on LSTM with 100-memory cells. The ALSTM architecture is shown as good
generalisation characteristics if fresh user information is included in a similar way.
Furthermore, the results have been slightly improved by combining both the acceleration
sensors and the gyroscope sensors, although when adding gyroscope data, the complexity and
time required for training has increased.

References

[1] Liao, R., Yu, S., An, W., & Huang, Y. (2020). A model-based gait recognition method
with body pose and human prior knowledge. Pattern Recognition, 98, 107069.

[2] Arshad, H., Khan, M. A., Sharif, M. I., Yasmin, M., Tavares, J. M. R., Zhang, Y. D., &
Satapathy, S. C. (2020). A multilevel paradigm for deep convolutional neural network
features selection with an application to human gait recognition. Expert Systems,
e12541.

[3] Jun, K., Lee, D. W., Lee, K., Lee, S., & Kim, M. S. (2020). Feature extraction using an
RNN autoencoder for skeleton-based abnormal gait recognition. IEEE Access, 8,
19196-19207.

[4] Xu, C., Makihara, Y., Li, X., Yagi, Y., & Lu, J. (2020). Cross-view gait recognition
using pairwise spatial transformer networks. IEEE Transactions on Circuits and
Systems for Video Technology, 31(1), 260-274.
[5] De Marsico, M., & Mecca, A. (2017). Gait recognition: The wearable solution. In
Human Recognition in Unconstrained Environments (pp. 177-195). Academic Press.

[6] Marsico, M. D., & Mecca, A. (2019). A survey on gait recognition via wearable sensors.
ACM Computing Surveys (CSUR), 52(4), 1-39.

[7] Prakash, C., Kumar, R., & Mittal, N. (2018). Recent developments in human gait
research: parameters, approaches, applications, machine learning techniques, datasets
and challenges. Artificial Intelligence Review, 49(1), 1-40.

[8] Deb, S., Yang, Y. O., Chua, M. C. H., & Tian, J. (2020). Gait identification using a new
time-warped similarity metric based on smartphone inertial signals. Journal of Ambient
Intelligence and Humanized Computing, 11(10), 4041-4053.

[9] Mantyjarvi, J., Lindholm, M., Vildjiounaite, E., Makela, S. M., & Ailisto, H. A. (2005,
March). Identifying users of portable devices from gait pattern with accelerometers. In
Proceedings.(ICASSP'05). IEEE International Conference on Acoustics, Speech, and
Signal Processing, 2005. (Vol. 2, pp. ii-973). IEEE.

[10] Gafurov, D., Snekkenes, E., & Bours, P. (2007, June). Gait authentication and
identification using wearable accelerometer sensor. In 2007 IEEE workshop on
automatic identification advanced technologies (pp. 220-225). IEEE.

[11] Thang, H. M., Viet, V. Q., Thuc, N. D., & Choi, D. (2012, November). Gait
identification using accelerometer on mobile phone. In 2012 International Conference
on Control, Automation and Information Sciences (ICCAIS) (pp. 344-348). IEEE.

[12] Cola, G., Avvenuti, M., & Vecchio, A. (2017). Real-time identification using gait
pattern analysis on a standalone wearable accelerometer. The Computer Journal, 60(8),
1173-1186.

[13] Sun, F., Zang, W., Gravina, R., Fortino, G., & Li, Y. (2020). Gait-based identification
for elderly users in wearable healthcare systems. Information fusion, 53, 134-144.

[14] Sprager, S., & Zazula, D. (2009). A cumulant-based method for gait identification using
accelerometer data with principal component analysis and support vector machine.
WSEAS Transactions on Signal Processing, 5(11), 369-378.
[15] Nickel, C., Brandt, H., & Busch, C. (2011). Classification of acceleration data for
biometric gait recognition on mobile devices. BIOSIG 2011–Proceedings of the
Biometrics Special Interest Group.

[16] Nickel, C., Busch, C., Rangarajan, S., & Möbius, M. (2011, March). Using hidden
markov models for accelerometer-based biometric gait recognition. In 2011 IEEE 7th
International Colloquium on Signal Processing and its Applications (pp. 58-63). IEEE.

[17] Nickel, C., Wirtl, T., & Busch, C. (2012, July). Authentication of smartphone users
based on the way they walk using k-nn algorithm. In 2012 Eighth international
conference on intelligent information hiding and multimedia signal processing (pp. 16-
20). IEEE.

[18] Huan, Z., Chen, X., Lv, S., & Geng, H. (2019). Gait recognition of acceleration sensor
for smart phone based on multiple classifier fusion. Mathematical Problems in
Engineering, 2019.

[19] Gadaleta, M., & Rossi, M. (2018). Idnet: Smartphone-based gait recognition with
convolutional neural networks. Pattern Recognition, 74, 25-37.

[20] Giorgi, G., Martinelli, F., Saracino, A., & Sheikhalishahi, M. (2017, September). Try
walking in my shoes, if you can: Accurate gait recognition through deep learning. In
International Conference on Computer Safety, Reliability, and Security (pp. 384-395).
Springer, Cham.

[21] Giorgi, G., Martinelli, F., Saracino, A., & Sheikhalishahi, M. (2018, September).
Walking through the deep: Gait analysis for user authentication through deep learning.
In IFIP International Conference on ICT Systems Security and Privacy Protection (pp.
62-76). Springer, Cham.

[22] Zou, Q., Wang, Y., Wang, Q., Zhao, Y., & Li, Q. (2020). Deep learning-based gait
recognition using smartphones in the wild. IEEE Transactions on Information Forensics
and Security, 15, 3197-3212.

[23] Fernandez-Lopez, P., Liu-Jimenez, J., Kiyokawa, K., Wu, Y., & Sanchez-Reillo, R.
(2019). Recurrent neural network for inertial gait user recognition in smartphones.
Sensors, 19(18), 4054.