A Data-Driven Model For Pedestrian Behavior Classification and Trajectory Prediction
A Data-Driven Model For Pedestrian Behavior Classification and Trajectory Prediction
ABSTRACT Pedestrian modeling remains a formidable challenge in transportation science due to the
complicated nature of pedestrian behavior and the irregular movement patterns. To this extent, accurate and
reliable positioning technologies and techniques play a significant role in the pedestrian simulation studies.
The objective of this research is to predict pedestrian movement in various perspectives utilizing historical
trajectory data. The study features considered in this research are pedestrian class, speed and position.
The ensemble of these features provides a thorough description of pedestrian movement prediction, whilst
contributes to the context of pedestrian modeling and Intelligent Transportation Systems. More specifically,
pedestrian movement is grouped into different classes considering gender, walking pace and distraction by
employing random forest algorithms. Then, position and speed prediction is computed employing suitable
data-driven methods, in particular, the locally weighted regression (LOESS method), taking into account
the individual pedestrian’s profile. An LSTM-based (Long Short-Term Memory) model is also applied for
comparison. The methodology is applied on pedestrian trajectory data that were collected in a controlled
experiment undertaken at the Campus of the National Technical University of Athens (NTUA), Greece.
Prediction of pedestrian’s movement is achieved, yielding satisfactory results.
INDEX TERMS Behavior classification, distraction, pedestrian speed prediction, pedestrian trajectory
prediction, random forests, GNSS, position fix.
neural networks are able to provide better estimation of prediction. The proposed methodology employs a classifi-
pedestrian speed at corridor and bottleneck experiments, cation and a regression algorithm in order to assign the
while outperforming the use of classical pedestrian models. obtained data into classes with similar characteristics, as
Kouskoulis et al. [23] also suggested superior performance well as to estimate pedestrian behavior state in real time
of a locally weighted regression algorithm compared to a and predict pedestrian speed and position in a short time
social force model for speed modeling using real data. span. In the proposed approach, the individual pedestrian
Pedestrian position prediction comprises yet another sig- profile is formed utilizing observations from previous time
nificant element in future cities, as accurate predictions instants. New observations of the following time instants are
could prevent a considerable number of traffic injuries then classified into the appropriate classes enabling trajectory
and improve pedestrian safety when employed in collision prediction based on this personalized pedestrian profile. The
warning systems or embedded in autonomous vehicle path proposed methodological framework is presented in Fig. 1.
planning [24]. Pedestrian trajectory is usually characterized
as series of generation actions [25]. Several researchers have B. PEDESTRIAN BEHAVIOR CLASSIFICATION
used data-driven methods for pedestrian position prediction, For the needs of this study, a database with pedestrian
including novel LSTM algorithms, Recurrent Neural observations including position, speed, acceleration, etc. as
Networks and Gaussian processes [24]–[31]. Pedestrian well as individual characteristics such as age, gender and
movement involves classes of behavioral status, speed and height, is utilized. The associations between these parame-
traveled distance values, as well as movement direction. Most ters are identified for every pedestrian. Initially, mean values
existing pedestrian movement prediction models tend to deal for the attributes of each pedestrian are calculated. The
with just one of these aspects. In this paper, we propose an classification step that follows divides the available obser-
integrated methodological framework to consider them all. vations into different pedestrian classes (i.e., gender, pace
and distraction conditions). For the implementation of data
III. METHODOLOGY classification, the random forests classification algorithm is
A. METHODOLOGICAL FRAMEWORK applied to the relevant data. Speeds, accelerations and devi-
Capturing pedestrian behavior heterogeneity has the poten- ations from the mean pedestrian speed are considered in this
tial to offer accurate input in the development of Intelligent process. The final result is a detailed historical database with
Transportation Systems. This research is based on [32] in classified data, while new data are assigned to the class with
order to expand the existing methodology considering also the highest resemblance.
pedestrian trajectory prediction (i.e., position and speed fea- The efficiency of classification methods is usually assessed
tures) further to pedestrian behavior classification. Thus, via the True Positive and False Negative rates [33].
the present research develops an integrated methodology for Evaluation of the classification algorithm performance is
pedestrian behavior classification and pedestrian movement achieved using a confusion matrix for the test data. Values
bag and the GNSS patch antenna was placed on the pedes-
trian’s cap (Fig. 2(c)). The Rover Station raw GNSS data
logging was handled by a smartphone device connected to
the receiver via USB and running the RTKGPS+ application.
Notwithstanding the selected test site is generally free of
obstacles, partial satellite obstruction and signal multipath
effect due to nearby buildings - mainly due to the high-
rise building located east of the area - pose a challenge
to GNSS positioning. Preliminary evaluation highlighted the
limitations of a single-point positioning (SPP) solution pro-
viding average accuracy at the order of ∼3 m while the
differential post-processing kinematic (PPK) solution pro-
vided accuracy levels of a few decimeters (Fig. 2(b)) leading
to the selection of this data processing approach. The extrac-
tion of pedestrians’ trajectories was implemented using the
RTKLIB software resulting in an average position trueness
of 0.2 m. The sampling rate of the data collection was set
at 5 Hz. During analysis observations were down-sampled
at 1 sec. Pedestrians’ coordinates were computed during
FIGURE 2. Experimental set-up. (a) Route followed by participants, (b) SPP (red) the experiment including computation of pedestrian speed
and PPK (blue) solutions trajectory comparison, (c) GNSS equipment employed for and acceleration. From this point onwards the data analysis
pedestrians.
and classification, as well as the development of data-
driven algorithms, have been performed in R programming
The walking path of the experiment is indicated with the language [51].
red line in Fig. 2a. In total, trajectories of 36 pedestrians
were recorded utilizing low cost GNSS technology. As this B. DATA PRESENTATION
is ongoing research, about 15.000 observations (17 pedes- This section presents the available data in order to draw
trians) have been processed and studied for the six walking a few preliminary conclusions. Regarding the walking pace
scenarios: base, talking on the phone and texting, all of them two trends can be identified in Fig. 3. The curves that exhibit
both at normal and fast pace. a sharp peak correspond to “fast” observations while those
The low cost GNSS receivers u-blox EVK-M8 /NEO- shown relatively flattened curves correspond to “normal”
M8T, u-blox C94-M8P, the high-quality geodetic type observations. The first set of curves associates with high
Pinwheel 702-GG satellite antenna of NovAtel, as well as speed values and deviations from pedestrian’s mean speed,
the RTKLIB software and the RTKGPS+ mobile applica- while the second one covers a wider area with lower speed
tion were used for the collection of the pedestrian data. values and deviations from pedestrian’s mean speed.
The Base Station GNSS data were collected utilizing the Nowadays, the use of smartphones has been inherent part
u-blox C94-M8P receiver connected with the geodetic grade of our life, and thus, constitutes a new distraction on pedes-
satellite antenna (NovAtel Pinwheel 702-GG) placed on a trians’ movement that takes various forms (e.g., talking or
geodetic pilar of known coordinates. At the same time, it texting). As shown in Fig. 4, a consequence of phone related
was ensured that the pedestrian localization sensor place- distraction while walking is slowing down the speed. From
ment was as non-intrusive as possible to minimize potential the same plot it is apparent that pedestrian speed decreases
effects on pedestrian motion. For this purpose, a u-blox more rapidly when texting. Table 2 summarizes the average
EVK-M8/NEO-M8T GNSS receiver was placed in a small speed observed for different pace scenarios.
(b)
FIGURE 5. Speed density plot for pedestrian groups with different age and gender
(base scenario and texting).
FIGURE 6. (a) Indicative routes (X and Y coordinates) under the three different
scenarios: base, talking on the phone, texting. (b) Indicative routes (X and Y
TABLE 2. Mean speed for all the experimental scenarios. coordinates) under the three different scenarios: base, talking on the phone, texting.
FIGURE 8. RMSN (%) of speed prediction for the pedestrians under three different
scenarios using LOESS method.
D. SPEED PREDICTION
In this case study, pedestrian speed is computed every
time instant (time step of 1 sec) considering the speed
and the acceleration values obtained at the previous time
FIGURE 7. Variable relationships for “fast” and “normal” walking pace. instant (vt+1 ∼ vt + at ). A LOESS algorithm is trained
separately for each pedestrian, so that personalized pro-
TABLE 5. Phone use recognition- confusion matrix. files are developed. For every pedestrian, the first half of
the dataset is used for training purposes and the remain-
ing set of data for testing. The optimal hyperparameters
have been chosen using the ‘train’ function, which estimates
algorithm performance through resampling [52]. The train-
ing step involves the selection of the optimal span (between
0.1 and 0.9) and degree (1 or 2). ’Span’ at a value of 0.75
and ‘degree’ at a value of 1 were chosen as the optimal
values of hyperparameters by this function.
Fig. 8 presents the results obtained using RMSN as a
Then, the trained algorithm is applied on the test data quality metric. The results are promising, as the algorithm
leading to 81% (Table 4) accuracy for pedestrian’s walking provides an error less than 1% of the complete dataset. The
pace prediction, either normal or fast. less accurate speed prediction seems to correspond to walk-
Fig. 7 plots two classes of pedestrians, those walking at ing while texting situations. This can be attributed to the
fast pace and those at normal pace. Evidently, the higher the fact that texting causes the highest reduction on pedestrian
walking speed the higher the acceleration and the deviation speed compared to the other scenarios, namely the base and
from pedestrian’s mean speed. In the same plot, the class the walking while talking on the phone scenario.
with high values of these features is depicted with grey
color and corresponds to the fast motion. When walking E. POSITION PREDICTION
at a fast pace, pedestrians tend to move at higher speeds The position prediction includes the computation of the dis-
and perform more abrupt maneuvers than they usually do, tance (ds) between sequential position fixes and the direction
resulting in higher values of accelerations and deviations angle (θ ). The distance dst+1 at the next time instant is esti-
from their mean speed. mated considering the speed vt and the acceleration at of the
Finally, the development of a random forest aiming at previous time instant (dst+1 ∼ vt + at ). The direction angle
investigating the effects of phone use (texting or talking) was θt+1 at the next time instant is estimated considering the
attempted. The trained random forest recognized the type of pedestrian’s transitions dXt and dYt and the direction angle
phone use with 73 % accuracy. The phone use recognition θt of the previous time instant (θt+1 ∼ θt + dXt + dYt ).
confusion matrix is presented in the Table 5. As this is an In order to estimate the predicted pedestrian coordinates
ongoing research, the last of the three random forests should at the next time instant based on the predicted distance and
be further improved, by exploring alternative explanatory direction angle (Fig. 9), the following formulas are used:
variables through increasing the number of previous time
instants and using a larger number of data series in general. Xt+1 = Xt + dst+1 ∗ cos θt+1 (12)
Deep learning could also be tested in this context. ϒt+1 = ϒt + dst+1 ∗ sin θt+1 (13)
TABLE 6. Final displacement error per pedestrian, per scenario and method.
FIGURE 10. Position prediction with high accuracy for pedestrian No. 34 and the
scenario walking while texting.
collection of high-quality datasets in a minimally-intrusive [19] R. Quintero, I. Parra, J. Lorenzo, D. Fernández-Llorca, and
manner still remains a widely open issue. M. A. Sotelo, “Pedestrian intention recognition by means of
a hidden Markov model and body language,” in Proc. IEEE
20th Int. Conf. Intell. Transp. Syst. (ITSC), 2017, pp. 1–7,
doi: 10.1109/ITSC.2017.8317766.
REFERENCES [20] M. Raza, M. Sharif, M. Yasmin, M. A. Khan, T. Saba, and
[1] V. Papathanasopoulou, C. Antoniou, and H. N. Koutsopoulos, “Data- S. L. Fernandes, “Appearance based pedestrians’ gender recognition
driven traffic simulation models: Mobility patterns using machine by employing stacked auto encoders in deep learning,” Future Gener.
learning techniques,” in Mobility Patterns, Big Data and Transport Comput. Syst., vol. 88, pp. 28–39, Nov. 2018.
Analytics: Tools and Applications for Modeling, C. Antoniou, [21] M. Fayyaz, M. Yasmin, M. Sharif, and M. Raza, “J-LDFR: Joint
L. Dimitriou, and P. F. Paperback, Eds. Amsterdam, The Netherlands: low-level and deep neural network feature representations for pedes-
Elsevier, 2019, pp. 263–295. trian gender classification,” Neural Comput. Appl., vol. 33, no. 1,
pp. 361–391, 2021.
[2] J. Guo, “Accuracy improvement of pedestrian trajectory prediction
by an extended Kalman filter and pedestrian behavior classification,” [22] A. Tordeux, M. Chraibi, A. Seyfried, and A. Schadschneider,
M.S. thesis, Dept. Electr. Comput. Eng., Rose-Hulman Institute of “Prediction of pedestrian speed with artificial neural networks,” in
Technology, 2018. Proc. Int. Conf. Traffic Granular Flow, Jul. 2017, pp. 327–335.
[23] G. Kouskoulis, I. Spyropoulou, and C. Antoniou, “Pedestrian simu-
[3] B. Völz, H. Mielenz, G. Agamennoni, and R. Siegwart, “Feature
lation: Theoretical models vs. data driven techniques,” Int. J. Transp.
relevance estimation for learning pedestrian behavior at crosswalks,”
Sci. Technol., vol. 7, no. 4, pp. 241–253, 2018.
in Proc. IEEE 18th Int. Conf. Intell. Transp. Syst., 2015, pp. 854–860,
[24] R. Quan, L. Zhu, Y. Wu, and Y. Yang, “Holistic LSTM for pedes-
doi: 10.1109/ITSC.2015.144.
trian trajectory prediction,” IEEE Trans. Image Process., vol. 30,
[4] V. Gikas, G. Retscher, and A. Kealy, “Collaborative positioning for
pp. 3229–3239, 2021, doi: 10.1109/TIP.2021.3058599.
urban intelligent transportation systems (ITS) and personal mobil-
[25] X. Li, Y. Liu, K. Wang, and F.-Y. Wang, “A recurrent atten-
ity (M): Challenges and perspectives,” in Mobility Patterns, Big
tion and interaction model for pedestrian trajectory prediction,”
Data and Transport Analytics: Tools and Applications for Modeling,
IEEE/CAA J. Autom. Sinica, vol. 7, no. 5, pp. 1361–1370, Sep. 2020,
C. Antoniou, L. Dimitriou, and Pereira F. Paperback, Eds. Amsterdam,
doi: 10.1109/JAS.2020.1003300.
The Netherlands: Elsevier, 2019, pp. 381–414.
[26] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and
[5] K. Okamoto et al., “Classification of pedestrian behavior in a shop-
S. Savarese, “Social LSTM: Human trajectory prediction in crowded
ping mall based on LRF and camera observations,” in Proc. MVA,
spaces,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jun. 2011, pp. 1–5.
2016, pp. 961–971, doi: 10.1109/CVPR.2016.110.
[6] A. Masiero et al., “A case study of pedestrian positioning with UWB [27] J. Bock, T. Beemelmanns, M. Klösges, and J. Kotte, “Self-learning
and UAV cameras,” Int. Archiv. Photogrammetry Remote Sens. Spat. trajectory prediction with recurrent neural networks at intelligent
Inf. Sci., vol. XLIII-B1-2021, 2021, pp. 111–116. intersections,” in Proc. VEHITS, Apr. 2017, pp. 346–351.
[7] G. Retscher, V. Gikas, H. Hofer, H. Perakis, and A. Kealy, “Range [28] A. Vemula, K. Muelling, and J. Oh, “Modeling cooperative navigation
validation of UWB and Wi-Fi for integrated indoor positioning,” Appl. in dense human crowds,” in Proc. ICRA, 2017, pp. 1685–1692.
Geomatics vol. 11, no. 2, pp. 187–195, 2019. [29] Y. Xu, Z. Piao, and S. Gao, “Encoding crowd interaction with
[8] G. Kouskoulis and C. Antoniou, “Systematic review of pedestrian deep neural network for pedestrian trajectory prediction,” in Proc.
simulation models with a focus on emergency situations,” Transp. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 5275–5284,
Res. Rec., vol. 2604, no. 1, pp. 111–119, 2017. doi: 10.1109/CVPR.2018.00553.
[9] New York Pedestrian Study, New York City Pedestrian Level of Service, [30] A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos, “PIE: A
Phase I, Department of City Planning, Transp. Division, New York, large-scale dataset and models for pedestrian intention estimation and
NY, USA, 2006. trajectory prediction,” in Proc. IEEE/CVF Int. Conf. Comput. Vis.,
[10] J. Hatfield and S. Murphy, “The effects of mobile phone use on pedes- 2019, pp. 6261–6270, doi: 10.1109/ICCV.2019.00636.
trian crossing behaviour at signalised and unsignalised intersections,” [31] X. Song et al., “Pedestrian trajectory prediction based on deep con-
Accid. Anal. Prevent., vol. 39, no. 1, pp. 197–205, 2007. volutional LSTM network,” IEEE Trans. Intell. Transp. Syst., vol. 22,
[11] J. Nasar, P. Hecht, and R. Wener, “Mobile telephones, distracted atten- no. 6, pp. 3285–3302, Jun. 2021, doi: 10.1109/TITS.2020.2981118.
tion, and pedestrian safety,” Accid. Anal. Prevent., vol. 40, no. 1, [32] V. Papathanasopoulou, I. Spyropoulou, H. Perakis, V. Gikas,
pp. 69–75, 2008. and E. Andrikopoulou, “Classification of pedestrian behavior
[12] E. Andrikopoulou, I. Spyropoulou, H. Perakis, and V. Gikas, using real trajectory data,” in Proc. 7th Int. Conf. Models
“Extracting pedestrian behavior using low cost GNSS receiver data,” Technol. Intell. Transp. Syst. (MT-ITS), Jun. 2021, pp. 1–6,
in Proc. 8th Transport Res. Arena TRA, Helsinki, Finland, Apr. 2020, doi: 10.1109/MT-ITS49943.2021.9529266.
pp. 1–6. [33] D. Ridel, E. Rehder, M. Lauer, C. Stiller, and D. Wolf, “A literature
[13] R. Alsaleh, T. Sayed, and M. H. Zaki, “Assessing the effect of pedes- review on the prediction of pedestrian behavior in urban scenarios,” in
trians’ use of cell phones on their walking behavior: A study based on Proc. 21st Int. Conf. Intell. Transp. Syst. (ITSC), 2018, pp. 3105–3112,
automated video analysis,” Transp. Res. Rec. J. Transp. Res. Board, doi: 10.1109/ITSC.2018.8569415.
vol. 2672, no. 35, pp. 46–57, 2018. [34] P. Kothari, S. Kreiss, and A. Alahi, “Human trajectory forecasting
[14] P. D. Loeb and W. A. Clarke, “The cell phone effect on pedes- in crowds: A deep learning perspective,” IEEE Trans. Intell. Transp.
trian fatalities,” Transp. Res. E, Logist. Transp. Rev., vol. 45, no. 1, Syst., early access, Apr. 19, 2021, doi: 10.1109/TITS.2021.3069362.
pp. 284–290, 2009. [35] T. Lin, Z. Jin, S. Choi, and H. Yeo, “A framework for pedestrian
[15] S.-W. Lee and K. Mase, “Recognition of walking behaviors for pedes- sub-classification and arrival time prediction at signalized intersection
trian navigation,” in Proc. IEEE Int. Conf. Control Appl. (CCA), using preprocessed Lidar data,” 2022, arXiv:2201.05877.
Sep. 2001, pp. 1152–1155, doi: 10.1109/CCA.2001.974027. [36] C. Amisse, M. E. Jijón-Palma, and J. A. S. Centeno, “Fine-tuning
[16] T. Jan, M. Piccardi, and T. B. Hintz, “Detection of suspicious pedes- deep learning models for pedestrian detection,” Boletim de Ciências
trian behaviour using modified probabilistic neural networks,” in Geodésicas, vol. 27, no. 3, 2021, Art. no. e2021017. [Online].
Proc. Image Vis. Comput. New Zealand Image Vis. Comput. Conf. Available: https://doi.org/10.1590/s1982-21702021000300017
Image Vision Comput. New Zealand, Auckland, New Zealand, 2002, [37] V. Papathanasopoulou and C. Antoniou, “Towards data-driven car-
pp. 237–241. following models,” Transp. Res. C, Emerg. Technol., vol. 55,
[17] Z. Chen, D. C. Ngai, and N. H. C. Yung, “Pedestrian behavior pp. 496–509, Jun. 2015.
prediction based on motion patterns for vehicle-to-pedestrian colli- [38] L. Zhu, H. Fan, Y. Luo, M. Xu, and Y. Yang, “Temporal
sion avoidance,” in Proc. 11th Int. IEEE Conf. Intell. Transp. Syst., cross-layer correlation mining for action recognition,” IEEE
Oct. 2008, pp. 316–321, doi: 10.1109/ITSC.2008.4732644. Trans. Multimedia, vol. 24, no. 51, pp. 668–676, Feb. 2021,
[18] C. G. Keller and D. M. Gavrila, “Will the pedestrian cross? A study on doi: 10.1109/TMM.2021.3057503.
pedestrian path prediction,” IEEE Trans. Intell. Transp. Syst., vol. 15, [39] D. Helbing, “A mathematical model for the behavior of pedestrians,”
no. 2, pp. 494–506, Apr. 2014, doi: 10.1109/TITS.2013.2280766. Behav. Sci., vol. 36, no. 4, pp. 298–310, 1991.