Received February 17, 2021, accepted March 7, 2021, date of publication March 10, 2021, date of current version

March 19, 2021.

Digital Object Identifier 10.1109/ACCESS.2021.3065125

Social Density Monitoring Toward Selective

Cleaning by Human Support Robot With 3D
Based Perception System
ANH VU LE 1,2 , BALAKRISHNAN RAMALINGAM 1 , BRAULIO FÉLIX GÓMEZ 1 ,
RAJESH ELARA MOHAN 1 , TRAN HOANG QUANG MINH 2 , AND VINU SIVANANTHAM 1
1 ROAR Laboratory, Engineering Product Development, Singapore University of Technology and Design, Singapore 487372
2 Optoelectronics Research Group, Faculty of Electrical and Electronics Engineering, Ton Duc Thang University, Ho Chi Minh City 700000, Vietnam

Corresponding author: Tran Hoang Quang Minh ([email protected])

This work was supported in part by the National Robotics Programme through Robotics Enabling Capabilities and Technologies under
Project 192 25 00051, in part by the National Robotics Programme through Robot Domain Specific under Project 192 22 00058, and in
part by the Agency for Science, Technology and Research.

ABSTRACT Monitoring the safe social distancing then conducting efficient sterilization in potentially
crowded public places are necessary but challenging especially during the COVID-19 pandemic. This work
presents the 3D human space-based surveillance system enabling selective cleaning framework. To this
end, the proposed AI-assisted perception techniques is deployed on Toyota Human Support Robot (HSR)
equipped with autonomous navigation, Lidar, and RGBD vision sensor. The human density mapping repre-
sented as heatmap was constructed to identify areas with the level being likely the risks for interactions. The
surveillance framework adopts the 3D human joints tracking technique and the accumulated asymmetrical
Gaussian distribution scheme modeling the human location, size, and direction to quantify human density.
The HSR generates the human density map as a grid-based heatmap to perform the safe human distance
monitoring task while navigating autonomously inside the pre-built map. Then, the cleaning robot uses the
levels of the generated heatmap to sterilize by the selective scheme. The experiment was tested in public
places, including food court and wet market. The proposed framework performance analyzed with standard
performance metrics in various map sizes spares about 19 % of the disinfection time and 15 % of the
disinfection liquid usage, respectively.

INDEX TERMS COVID-19, human space, social distance, cleaning robotics, human support robot.

I. INTRODUCTION reducing the risk of the local spread of COVID-19. Tobías

The recent outbreak of COVID-19 has caused a pandemic and Saez et al. [2] statistics reports indicate that COVID 19
alert around the world. It has now globally affected almost spread has considerably reduced in Spain and Italy after
all the continents, infecting more than 82 million people and implementing safe social distancing measures.
1,79 death reports (30 December 2020). According to World In a safe distancing measure, the physical distancing of at
Health Organisation (2020), physical distancing and routine least 1m is essential in crowded zone [1]. Globally, several
sterilizing are the effective ways to slow down the spread of practices are implemented to ensure safe social distancing
the virus because when people maintain safe social distancing and avoid crowd gathering in busy places. For example, sev-
and avoid physical contact, the chances of transmitting the eral countries are deploying Safe Distancing Officers (SDO)
virus from one person to another reduces significantly [1]. in public places to monitor the people, safe distance sign
The absence of an approved vaccine for this disease urges mark, limiting the number of people in workplaces, and
the need to minimize the spread of the contagious and lethal restricting to large gathering [1], [3]. However, monitoring
virus. Social distancing measures proved the efficient in and tracking safe social distancing in potentially crowded
places such as public transit stations and industrial settings
The associate editor coordinating the review of this manuscript and such as factory workplaces, dormitories, schools, and shop-
approving it for publication was Ehsan Asadi . ping malls are quite challenging. Moreover, monitoring safe

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 9, 2021 41407
 A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

distance measures is a manual process that can bring the overhead perception system. The system integrated trans-
surveillance staff in close proximity with people affected by fer learning of the YOLOv3 an open source deep learning
COVID-19. object detection framework with an overhead human data set
One significant aspect that has been studied widely in in video sequences. The proposed system simply uses the
social safety for spatial interaction is the idea of personal Euclidean distance of detected bounding box centroids to
space, or proxemics [4]–[8]. According to [4], based on estimate the distances of pairwise people. Then to estimate
different types of interaction and relationships between peo- social distance violations between people, an approximation
ple, people maintain different culturally defined interpersonal of physical distance to pixel are set as fixed threshold. How-
distances. Hall differentiated four different zones between ever, the quantification in term of 3d interaction between
the interaction distance as follows: Public interaction: Public human space and the utilization of human interaction are not
speeches in-crowd, more than 4 m away. Social interac- considered in the mentioned references.
tion: business meetings, 1–4 m. Personal interaction: friendly In this context, service robots are a viable candidate for
interaction distance gap of arm’s length, 0.5–1 m. Intimate monitoring safe distance measure. Robotics and several other
interaction: about 0.5 m apart. autonomous technologies in AGV vehicles have made great
In this paper, we specifically focus on the zone of personal strides in fighting the COVID-19 pandemic [22], [23]. The
space as it is a culturally defined zone of ‘‘spatial insula- service robot uses the AI-assisted technology to deliver the
tion’’ that people maintain around themselves and others [9]. medicine to covid 19 patients, safe entry check-in body tem-
Research work in [10], has described personal space for the perature measurement, sanitize the infected area, frequent
case of both people approaching each other and standing in cleaning of high touchpoints like hospital walls, floor, and
line is asymmetric [11]. Also [12] discusses personal space is door handle [16], [24], [25]. Hence, by considering the advan-
not a constant as it is dependent on individual attributes such tage of service robots and AI-assisted technology, as well
as volume, age, gender, and direction of interaction. as the flexible navigation in the vast and complex envi-
Automate the monitoring of social distancing measures is ronments such as shopping mall, food court, wet market,
a viable solution. Some effort worldwide that were imple- resents robot system gradually replace human in the tedious
mented on an ad-hoc framework to enforce social distanc- jobs. In the monitoring, surveillance tasks and cooperate
ing rules. Some industries recently use lightweight wearable conveniently through the comment operation system with
devices that employ ultra-wideband (UWB) technology to another service robot such as cleaning robot are the recent
measure people’s distance automatically. It alerts them imme- trends.
diately if they come closer than the required distance [13]. In this article, based on the literature survey on the human
Some countries have adopted ubiquitous technologies, such space and multi human interaction, we propose a human
as Wi-fi, cellular, GNSS positioning (localization) systems to safe distance monitoring technique using Toyota Human
monitor and alert the social distance in public and crowded Support Robot (HSR) and AI-assisted 3D computer vision
areas [1], [14]. Recently, many countries worldwide have framework. The computer vision framework was built with
used the drones, IoT, and AI-assisted techniques to monitor a modified Openpose 3D human tracking algorithm, depth
the human density, predict and alert the safe distance breach image fusion technique, Gaussian heat map scheme, and uses
in crowded areas in indoor and outdoor [3]. However, these the RGBD vision sensor data. The entire framework built
techniques have numerous limitations and have poor per- on top of Robot Operating System (ROS) [26] and tested in
formance in dynamic and complicated indoor environments. real-time with HSR Toyota robot [27] deployed in crowded
With the advanced high speed and accuacy devices, the cap- public areas of Singapore, including food court and wet
tured 3D object [15] have been applied in suveliance appli- market. The service robot navigates to clear the waypoints
caitons for quality assessmen. Moreover, the exiting CCTV around the mapped indoor area and performs the SDO tasks
system [16]–[18] consisting of monocular RGB cameras is that include detecting people’s clusters, space between the
not cover the whole the workspace and CCTV is hard to detect humans, human interaction pose, safe distance measure, and
the 3D human attributes. In the works of [19], the authors raising warning alerts commuters when violating the safe
have addressed the distance-time encounter patterns in a distance rule.
crowd that allows the fixed surveillance system to identify The main contributions of this paper are threefold. (1) the
social groups, such as families, by imposing adaptive thresh- design and development of safe social distance surveillance
olds on the distance-time contact patterns. On other hand, with collaborative multi-robot cleaning system, (2) to develop
in the works of [20], an artificial intelligence based social and implement a vision-based AI perception algorithm for the
distancing surveillance system is present to detect distances robot to closely generate a heatmap based on the 3D human
between human and warning them can slow down the spread interaction model, (3) to develop and test an adaptive velocity
of the deadly disease. The work presented the four essential behaviour model for the multi-robot cleaning systems to
ethical factors of surveillance system of: keep the privacy, not clean the environment efficiently based on the generated heat
target the particular detected human, no human supervisor, map. The efficiency of the proposed selective cleaning system
and open-source. The work of [21] has proposed a deep was assessed with standard performance metrics, and results
learning technique to track the social distance by an fixed are reported.

41408 VOLUME 9, 2021

A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

FIGURE 1. The framework for human density mapping based on human attributes.

The paper is organized as follows. The context of the appli- state that human interaction tends toward the human direc-
cation is introduced in Section 2. The methodology of the tion, and space occupied. This is fundamental to motivate us
proposed robot is detailed in Section 3. In Section 4, the HSR to quaintly the human by human space of 3D location, volume
platform is presented. The optimal human attributes estima- and facing direction. The perception unit of HSR outputs the
tion methods and Social Distancing and Density heat map human attributes includes the 3D positioning of human joints
are validated in Section 5 and Section 6, respectively. The in the map, the volume of space occupied by the detected
conclusion, together with potential future works, is explored person, and the person’s movement direction.
in the last Section 7. To quantify human-to-human interactions, we propose a
distribution kernel with direction and magnitude proportional
II. CONTEXT OF APPLICATION to the detected and tracked a person’s identity and plotted
Figure 1 depicts the workflow used in the proposed frame- on the map concerning the person’s position. To this end,
work to map the human density based on detected human we extract the color image data from the RGB-depth camera,
attributes by service robot. The output heatmap generated is and human joints are detected and marked by the AI-based
a distribution that highlights the level of interaction between Openpose algorithm [28]. The 3D depth information from the
humans. Based on this heatmap distribution, the system iden- camera frame is used to estimate the person’s joint positions,
tifies locations with the level of human interactions in a and the position is then converted to the map frame to be
pre-built map of the environment. The user can set a threshold tracked while it is in the field of view of HSR camera.
to determine the level of interaction and issue a warning We divide the pre-built map into grid cells and plot 3D
alert whenever safe distancing measures are violated. More- directional distributions of tracked human positions over the
over, the system helps deploy area sterilization by cleaning grid cells.
robot systems that activate adaptive cleaning based on the The asymmetrical Gaussian distribution has its peak value
area heatmap levels. The interactive monitoring system has set at the location where humans are detected and spread
been deployed for trials evaluations at testbed wet-market gradually along the human volume and moving direction of
in Singapore. each person. After plotting all the human positions, each cell
value in the map is updated by accumulating the human distri-
III. METHODOLOGY bution values during detection. Figure 2 shows the proposed
This software framework is developed on a Toyota HSR robot framework models in terms of quantity the interaction of three
equipped with an onboard AI-embedded perception system. persons at the same distance but own different attributes then
Unlike traditional approaches that only monitor safe physical interaction between them differently.
distancing based on a person’s position from a fixed camera
system, this study defines the degree of human-to-human IV. HSR ROBOT PLATFORM
interaction by quantifying social interaction using an asym- A. OVERVIEW OF HSR SYSTEM ARCHITECTURE
metrical Gaussian distribution with the shape derived from The Human Support Robot (HSR), as shown in Figure 3,
human attributes. Note that the theoretical background for is the research platform developed by Toyota Ltd. Co.
social interaction is based on the survey work of [4], which HSR has been implemented in multiple applications such as

VOLUME 9, 2021 41409

 A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

up in a public wet-market and food-court in a neighborhood

community area in Tampeniss Town, Singapore. This location
is chosen as it is a highly busy environment with more crowd
gathering. So it is crucial to monitor safe social distancing
and regularly sterilize to avoid the spreading of the virus.
Hector Simultaneous Localization and Mapping (SLAM)
[39] is implemented in HSR to map continuously from
FIGURE 2. Proof of concept where 3 humans stay same distances and
with difference attributes so that interaction differently represented by
the environment using a Hokuyo laser to locate its correct
the level of color of areas in constructed heatmap. position. Within the generated market map, the HSR can
reach to its destination efficiently and quickly. The generated
map is shown in Figure 4(left) and the market is shown in
Figure 4(right).

FIGURE 3. Human Support Robot (HSR).

FIGURE 4. Generated market map & Market environment.

cleaning and inspection [29]. The HSR platform is equipped

with sensors like 2D Lidar, IMU, ASUS Xtion RGBD cam- To maximize area coverage, a set of coordinates in the
era, stereo vision camera, and embed main processing unit map environment are generated in a zigzag pattern. These
with dedicated GPU necessary to support the autonomy generated coordinates are given to the robot one by one as
AI-based perception of the robot platform. This research uses sub-goals for the robot to manoeuvre. The waypoints are
the data from the ASUS Xtion RGBD camera that extracts assigned manually to HSR move based after the map is built.
RGB color and depth information around the robot. The HSR generates a trajectory connecting the coordinates using
RGBD camera has a resolution of 1280 × 1024 and runs a local trajectory planner algorithm and follows the trajec-
at 30 frames per second. Hokuyo URG-04LX 2D LiDAR tory to generate the heatmap by simultaneously inspecting
installed in the HSR base enables the simultaneous localiza- the human density in the market environment. Based on the
tion and mapping to build a map of the environment. Besides, feedback from laser data and wheel odometry information,
pneumatic bumpers are installed at the robot base to provide HSR will move to the designated coordinates one by one.
an emergency stop to avoid any possible collision. A dual sys- During the movement, HSR can make proper decisions from
tem including Intel Core i7 CPU (16GB RAM, 256GB SSD) collected data for obstacle detection and avoidance.
and NVIDIA Jetson TK1 embedded GPU board are the cen-
tral processing units used alongside the robot. Robot control V. HUMAN ATTRIBUTES ESTIMATION METHODS
and motion planning require proper communication between A. HUMAN JOINT DETECTION IN 2D COLOR IMAGE
software algorithms and hardware modules. This communi- HSR uses an inbuilt Asus Xtion Camera that provides the
cation framework is enabled through the ROS environment RGB and depth data. This data is used as input for Open-
by ROS nodes and topics in Linux Ubuntu-based system. Pose [28] that gives us the skeleton of the detected humans
in every frame, as shown in Figure 5. This dependency gives
B. HSR COVERS AREA BY DEFINED WAYPOINTS us every joint of the human body in a two-dimensional array
NAVIGATION of 25 rows and 3 columns where every row is a skeleton’s
The service robots are the active research and gradually to joint. Note that we consider the (x,y) plane is the 2d navi-
be the commercial applications such as building mainte- gation, (x,y) plane is the camera world coordinate, y is the
nance [30]–[32], road maintenance [33], [34], and path track- distance and z direction models the object height. The first
ing [35], [36] and service robot [37], [38]. The HSR robot two columns are the X and Z pixel position of the joint in RGB
the research service platform developed by Toyota [29] is set image, the third column shows the percentage of accuracy.

41410 VOLUME 9, 2021

A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

2) 3D JOINT TRACKING
The DeepSORT is an improved version of the Simple Online
and Real-time Tracking (SORT) algorithm. The DeepSORT
tracking framework was build using the hypothesis tracking
technique with Kalman filtering algorithm and DL based
association metric approach (Deep SORT). Further, the Hun-
garian algorithm was utilized to resolve the uncertainty
between the estimated Kalman state and the newly received
measuring value. The tracking algorithm uses the appearance
data to improve the performance of Deepsort [40], [41].
In this work, the 3D joint coordinates of human detection
with the aligned RGB and depth frames are fed into the
modified DeepSORT network for tracking the human move-
ments. Then we retrained by transferred learning technique
the original DeepSORT object tracking algorithm which is
FIGURE 5. Human skeleton by Openpose. initially designed for 2D object tracking to track the detected
3D joints. According to bounding box coordinates and object
By taking the average of accuracy from every joint, we filter appearance, deep sort assigns an id for each human detection
the false detections. and performs the tracking in 3D camera frame coordinate.

1) 3D LOCATION ESTIMATION B. TRANSFORMATION TO MAP FRAME AND HUMAN

After estimating the 2D pixel coordinates of joints from the VOLUME
RGB camera frame using the object detection algorithm, When HSR navigates, the SLAM algorithm maintains a rela-
the system will derive the world frame corresponding to the tive position from the map frame to the robot base link frame.
3D position obtained from depth data. Since the RealSense Sine color and depth image is aligned by Realsense camera,
D435 of the perception system is a stereo camera with both we crop the area of 10 × 10 at center of the neck’s position of
an RGB image sensor and depth image sensor, they are of every detected human in the depth images. By appling the
the same resolution. Since each pixel in the RGB image mean filter for the cropped area, we get the distance with
corresponds to the pixel in the Depth image, this enables us noise is filtered out from the camera to the human’s neck, thus
even to map noisy objects and localize their pixel coordinates avoiding the probability of taking a None value in-depth data
concerning depth value. When the depth images are sub- array. The frame transformation operations are done in ROS
scribed, the image will be filtered via an adaptive directional to estimate an object’s position in the real world relative to
filter [33] to remove noise. After filtering the depth image, the sensor component used on the robot. The transformation
the depth value of the noisy human objects, yo , can be derived is then translated from the sensor frame to the Base Link,
by using the Equation 1. the robot’s centre. The base link frame is the base point of
P
yi reference on the robot to locate the humans and link their
yo =  (1) position to the Map transform frame that is the origin of

the World Space where we handle on ROS. We estimate the
where  represents the sum of pixels within a 10×10 window, distance between the shoulders, the height of the human, and
and yi represents the value of the depth in the pixel. The the hips’ distance from the OpenPose output data. Then we
human objects in real-world coordinates will be estimated apply Equation 3 for the calculations.
by converting the X and Z pixel coordinate system to the x
and z world coordinate system. This is done through camera Hs = DS × Hh × Dh (3)
calibration techniques to calculate the intrinsic and extrinsic
matrix values of the camera. Camera calibration method esti- where Hs is human size, Ds is the Distance Shoulders and Hh
mates the focal length, fx , and fz , of the RealSense RGB-D is human height and Dh is the Distance Hip.
sensor and the optical center, cx , and cz , of the RealSense Base on [4], to tun the parameters the values σh ,σs and σr
RGB-D sensor in its respective x and y coordinate to get of asymmetrical Gaussian distribution adaptively, we take the
the intrinsic camera matrix,I . The RealSense RGB-D sensor referenced values as the adult human dimensions: 150cm for
in-built functions also provide us with the translation vector, the height, 40 cm as the distance between the two shoulders,
K , and a Q rotation vector. The extrinsic matrix, S, can be and 30cm as the distance between hips. The values will be
derived from S = [QK ]. We can convert from the pixel coor- used as a quotient with the actual measures of the detected
dinate system to the world coordinate system as Equation 2. human. The results will be multiplied to σh ,σs and σr as a
scalar value. Derived parameters give us a variation of size in
p = I ∗ [QK ] ∗ P = I ∗ S ∗ P. (2)
the distribution proportional to the current size of the detected
where P is the world coordinates, p is the pixel coordinates. humans.

VOLUME 9, 2021 41411

 A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

distribution of the detected humans. We quantify the shape of

space human occupying by adjusting the sigma parameters
of this function. Equation 6 shows a 1-dimensional Gaussian
distribution with σ :
2
− (x−µ)
2
f (x) = exp 2σ (6)
The two dimension Gaussian distribution as shown in 7
is the center at (x0 ; y0 ), and the variance are represented by
σx and σy :
(x−x0 )2 (y−y0 )2
−( + )
2σx2 2σy2
f (x, y) = exp (7)

FIGURE 6. In the left image a person facing directly the camera with the
Typically, a 2-dimensional Gaussian distribution is sym-
nose tracked by OpenPose, in the right a human facing contrary the metric along the x and y axes. However, to formulate a
camera without nose joint. distribution function for personal space between people, a
2-dimensional asymmetric Gaussian function is necessary;
C. HUMAN DIRECTION AND FACING this can be done with a shared σx and differing σy values.
The human direction is the vector with the θ direction is set ALGORITHM 1: Asymmetric Gaussian Based-
to orthogonal with the vector linking left and right shoulder Heatmap Generation
joints. Once we estimate the human left and right shoul- − atan2(y − yc , x − xc ) − θ + π/2
1 : find :α ←
der joints’ position, we will find the angle of the vectors 2: Normalize(α)
formed by those two joints. To deal with the situations of 3: a←− (cosθ)2 /(2σ 2 ) + (sinθ)2 /(2σs2 )
human facing upward and backward the camera, specifically, 4: b←− sin(2θ)/(4σ 2 ) − sin(2θ)2 /(4σs2 )
we classify the detected human direction into front and rear 5: c←− (sinθ)2 /(2σ 2 ) + (cosθ)2 /(2σs2 )
facing cases. Then the formula as in Equation 4 is used to 6 : return(exp(−(a(x − xc )2 + 2b(x − xc )(y − yc ) + c(y − yc )2 ))
find the direction for the case of existing the detected human
nose joint and in Equation 5 for the case of non-existing the Algorithm 1 explains the computation of an arbitrarily
detected human nose joint. given the left and right shoudlers rotated asymmetric Gaussian function at a detected human
have the tracked codinates xls , yls , zls , xrs , yrs , zrs , the orienta- location (x; y). The following notations are used: θ is the
tion value gives the angle of the line in a clockwise direction. direction of the distribution taken from the human’s estimated
Nevertheless, this angle can be wrong sometimes because the orientation. σh variance in θ direction. σs variance to the sides
shoulders can only be detected in an interval of 180 degrees. (θ +− π/2 direction). σr variance to the back (-θ direction).
To enhance the accuracy, with the known camera angle value Those parameters σh , σs , σr are set to be proportional with
concerning the tracked 3D human joints, we can estimate the detected human volume. Lines 1, 2, and 3 calculate the
whether this human is facing the camera. normalized angle of the human facing σs direction. This
OpenPose tries to detect every joint in the human body. If it means α points along the side of the function, and 0 < α < π .
is impossible to detect the joint by the probability mean lower The two 2D Gaussian functions in Line 3 will be used for the
than the threshold, it will fill with an empty value such as null point of interest, (x; y). In the case of α = 0, the point of
value for every column of the respective joint. This behavior interest is located directly to the side of the function center
is essential because if the human face a contrary direction to and depends only on σs . Figure 7 displays some views of
the camera, there will not be any joint-related with the face an Asymmetric Gaussian cost function as shown in Equa-
on the joint’s array. Taking that as a reference, we can know tion 8. This function has a rotation of θ = π/6, is centered
if the human is facing toward or backward, so we modify at (0; 0) and has as variances σh = 2:0, σs = 4=3, and
the formula as follows, taking into account that 0 degrees are σr = 1. The maximum cost of this function is 1 in the center
facing contrary to the camera: of the distribution.
If Joint 0 (Nose) exist: 2 +2b(x−x )(y−y )+c(y−y )2 )
f (x, y) = exp−(a(x−xc ) c c c (8)
θ = (atan2(|yls − yrs |, |xls − xrs |) × 180/π) + 180 (4)
The Figure 8 presents the example the human detected at
else: the origin (0,0) facing upward by θ = π rads, σs = 13.33,
θ = atan2(|yls − yrs |, |xls − xrs |) × 180/π (5) σr = 10.00, σh = 20.00

VI. SOCIAL DISTANCING AND DENSITY HEAT MAP B. HEAT MAP GENERATION
A. DISTRIBUTION KERNEL The heat map generation consists of following steps:
An asymmetric Gaussian integral function using for social 1) OpenPose that detect the humans and give the joints of
robot interaction in [42] was deployed to calculate the the skeleton.

41412 VOLUME 9, 2021

A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

FIGURE 7. Asymmetric Gaussian Distribution views located at (0, 0), FIGURE 9. Two persons facing opposite directions & Two persons facing
oriented by θ = π/6 rad, and with variances σh = 2, σs = 4, and σr = 1. each other.

FIGURE 10. Four persons detected by system.

FIGURE 8. Output of the Aasymmetric Gaussian integral function for one

detected human at origin and facing upward.
FIGURE 11. tracking group of humans and remove the outliner. (a) three
detected humans and one outliner, (b) tracking the humans’ location on
2) The Detection node that will take OpenPose output and ROS Rviz, (c) heatmap of three humans.
calculate location, volume, and orientation
3) The Distribution Kernel and plotter Node that calculates will not be re-plotted. the new distribution is plotted if the
the intensity of distribution using the asymmetric Gaussian new id is detected or the tracked id moves to a different cell.
formula and displays the results as a 2D heatmap.
Specifically, the map is divided into a grid base workspace VII. EXPERIMENTAL RESULTS IN REAL ROBOT
with each cell equals 0.1 × 0.1 m. Once the distribution’s To evaluate the proposed software algorithm’s performance
raw data is calculated, we take it and plot it in a color mesh to monitor social safe distancing measures in public places,
plot. For this process, we need to set up an interval as a we implemented the software packages on HSR through the
reference, so the intensity of distribution is represented as ROS framework. Principally, HSR runs through the ROS
a variation of color. The interval is chosen between the min interface, and ROS architecture is flexible to implement dif-
value of intensity currently registered and the max value by ferent tools or dependencies in the form of packages and
default. This scale is updated every frame, so if there is an model utilities. OpenPose uses the RGB-D input to detect
increment in the intensity, it will update the scale in real- every human; this happens in the Detection node. Once we
time. An interval can be set up as an argument of the Heatmap detect a human, we publish all the joints of every detected
generator node in ROS. human, the location and orientation, and volume. To this
When two points overlap in the calculation of the two end, we use a costumed Message on ROS that we publish
detected persons, the algorithm adds the intensity values of under the topic. In parallel with the generated distribution
all asymmetric Gaussian distributions for to form the level of kernels during robot flows the trajectory, the plotter Node will
human interaction in the considered cell of the grid map. The accumulate all values belonged to each cell of the grid map
detected human’s id with the grid cell within the map frame to generate the final heatmap.
where the human presents is tracked by HSR. During the In the first experimental section, we experimented with
robot navigation, if the detected person has the same id and the proposed framework by considering several different
staying in the same grid cell, the distribution of this human interaction scenarios. Figure 9 demonstrates the interaction

VOLUME 9, 2021 41413

 A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

FIGURE 12. Robot navigation with real-time density mapping.

FIGURE 14. The sterilization robot in tested foodcourt.

TABLE 1. Numerical spent time and liquid of regular and selective

FIGURE 13. Heat map for Testbed Market environment. cleaning schemes.

scenarios between two persons maintaining a distance

of 1.5 meters. Figure 9 shows a scenario where two persons
face in different directions. Figure 9 b shows the scenario
where two standing persons face opposite directions. One
can observe from the distribution plot, two persons facing
opposite directions generate the least risk area. On the other
hand, the second scenario demonstrates a high risk in the
distribution plot when two persons face each other.
The third scenarios consider a group of persons with dif- distribution result is used as input to a disinfection robot to
ferent attributes and locations. The output results are shown decide disinfecting areas having level distributions on the
in Figure 10 and Figure 11, where we can observe that the map need to be cleaned.
pedestrians maintaining a safe distance between each other The selective disinfection algorithm estimates the time to
have their distribution spread in the green region. On the other sterilize the environment in proportion to the heatmap distri-
hand, when people come closer and face each other, their bution values. Specifically, the higher the heatmap’s intensity,
distribution changes gradually from blue to red spectrum. the longer the time will be taken to disinfect the region. In the
Meaning that they are interacting with other people and are case of a conventional disinfection robot, the main focus
not maintaining a safe distance. One can observe from the would only be on maximizing the area coverage. So dur-
results that four persons with different attributes and loca- ing the cleaning process, the conventional disinfection robot
tions. The persons who stay far away have the distribution spends equal time in all the map regions. Thus, the disin-
with greener color. On the other hand, the persons who stay fection amount savings between the proposed method and
in the middle of two big volume persons and face toward them conventional robotic disinfection will become a criterion for
have the redder color; this indicates that these persons have a evaluation.
greater chance of interacting with other persons. To conduct the comparison evaluation between regular
Based on the distribution’s value, we can select the prede- uniform cleaning and selective cleaning, one conventional
fined threshold value to identify the area in the map with the sterilization robot as shown in Figure 14 is first deployed
higher risk of human interaction, as shown in Figure 12 and to pray the cleaning liquid while following a defined zigzag
Fugue 13. Depending on this input threshold value, the HSR trajectory, and the times it takes to cover the sub-maps of the
robot can trigger the alarm warning when it patrols around wet market as of Figure 4(left) are recorded for all trials. This
the working environments. robot with a liquid sterilization system is deployed to cover
In the second experimental section, we validated the effi- the 100, 150, and 200 sqm areas with humans’ presenting.
ciency and performance of proposed heatmap in terms of Then, the total time to cover each testbed submap is divided
selective disinfection efficiency by deploying the robot sys- by its size to find the time interval the robot has stayed at
tem at a public food court in Tampines, Singapore. Firstly, each grid cell. The found time is set to the time that the
we mapped the environment using occupancy grid-based sterilization robot with selective cleaning methods remains
mapping. Depending on the human interaction activity in the at the highest heat cell (the cell with the reddest color inside
environment, a heat map is constructed on the grid cells by the build heatmap). The retention time of cells will decrease
the proposed framework, as shown in section 4. The heatmap gradually with the degrees of the cell heat maps.

41414 VOLUME 9, 2021

A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

Travel time and disinfection liquid spent were recorded [10] J. R. Aiello and D. E. Thompson, ‘‘Personal space, crowding, and spatial
during the 5 trials for each tesbed map. Table 1 describes the behavior in a cultural context,’’ in Environment and Culture. Boston, MA,
USA: Springer, 1980, pp. 107–178.
experiment’s comparison averaged results on testbed layouts [11] Y. Nakauchi and R. Simmons, ‘‘A social robot that stands in line,’’ Auton.
during the trials. The average per-time disinfectant solution Robots, vol. 12, no. 3, pp. 313–324, 2002.
offered by the selective method with heatmap can save about [12] J. W. Burgess, ‘‘Interpersonal spacing behavior between surrounding near-
est neighbors reflects both familiarity and environmental density,’’ Ethol.
19 %, 20 %, and 16 % the spent time and 15%, 19%, and 12% Sociobiol., vol. 4, no. 1, pp. 11–17, Jan. 1983.
of disinfection liquid, respectively. [13] M. M. Islam, S. Mahmud, L. J. Muhammad, M. R. Islam, S. Nooruddin,
and S. I. Ayon, ‘‘Wearable technology to assist the patients infected with
novel coronavirus (COVID-19),’’ Social Netw. Comput. Sci., vol. 1, no. 6,
VIII. CONCLUSION p. 320, Nov. 2020.
COVID-19 is the third pandemic of the 21st century. [14] A. A. R. Alsaeedy and E. K. P. Chong, ‘‘Detecting regions at risk for
spreading COVID-19 using existing cellular wireless network functionali-
COVID-19 pandemic is easily spread by people in close prox- ties,’’ IEEE Open J. Eng. Med. Biol., vol. 1, pp. 187–189, 2020.
imity, especially in crowds with mobile individuals (e.g., food [15] U. Stenz, J. Hartmann, J.-A. Paffenholz, and I. Neumann, ‘‘High-precision
3D object capturing with static and kinematic terrestrial laser scanning
court, wet market). The proposed social distant monitoring
in industrial applications—Approaches of quality assessment,’’ Remote
and selective sterilization strategy have validated efficiency in Sens., vol. 12, no. 2, p. 290, Jan. 2020.
the real public environment. The proposed system is the initial [16] A. V. Le, R. Parween, P. T. Kyaw, R. E. Mohan, T. H. Q. Minh, and
C. S. C. S. Borusu, ‘‘Reinforcement learning-based energy-aware area
works to deploy an adaptive multi-robot cleaning strategy coverage for reconfigurable hRombo tiling robot,’’ IEEE Access, vol. 8,
based on coverage path planning that works in synergy with pp. 209750–209761, 2020.
the human interaction heat map generated by safe social [17] A. V. Le, P. Veerajagadheswar, P. T. Kyaw, M. A. V. J. Muthugala,
M. R. Elara, M. Kuma, and N. H. K. Nhan, ‘‘Towards optimal hydro-
distance monitoring systems blasting in reconfigurable climbing system for corroded ship hull
Our future works will focus on: redesigning the long-term cleaning and maintenance,’’ Expert Syst. Appl., vol. 170, May 2021,
autonomy framework intensity, implement the autonomous Art. no. 114519.
[18] S. K. Udgata and N. K. Suryadevara, COVID-19, Sensors, and Internet of
path generation to re-clean the part of the surface concerning Medical Things (IoMT). Singapore: Springer, 2021, pp. 39–53.
the generated heatmap, working on the optimization algo- [19] C. A. S. Pouw, F. Toschi, F. van Schadewijk, and A. Corbetta, ‘‘Monitoring
rithm to control the generated heatmap to reduce the running physical distancing for crowd management: Real-time trajectory and group
analysis,’’ PLoS ONE, vol. 15, no. 10, Oct. 2020, Art. no. e0240963.
time usage.Since the deploying the system at a public food [20] D. Yang, E. Yurtsever, V. Renganathan, K. A. Redmill, and Ü. Özgüner,
court in Tampines, Singapore requires the particular setups ‘‘A vision-based social distancing and critical density detection sys-
so that the comparisons between the surveillance systems will tem for COVID-19,’’ 2020, arXiv:2007.03578. [Online]. Available:
http://arxiv.org/abs/2007.03578
be also considered as the future works. [21] I. Ahmed, M. Ahmad, J. J. P. C. Rodrigues, G. Jeon, and S. Din, ‘‘A deep
learning-based social distance monitoring framework for COVID-19,’’
Sustain. Cities Soc., vol. 65, Feb. 2021, Art. no. 102571.
ACKNOWLEDGMENT [22] M. Antony, M. Parameswaran, N. Mathew, S. V. S. J. Joseph, and
The authors would like to thank Jia Yin for insightful com- C. M. Jacob, ‘‘Design and implementation of automatic guided vehicle
ments and discussion. for hospital application,’’ in Proc. 5th Int. Conf. Commun. Electron. Syst.
(ICCES), Jun. 2020, pp. 1031–1036.
[23] Z. H. Khan, A. Siddique, and C. W. Lee, ‘‘Robotics utilization for health-
REFERENCES care digitization in global COVID-19 management,’’ Int. J. Environ. Res.
Public Health, vol. 17, no. 11, p. 3819, May 2020. [Online]. Available:
[1] V. Chamola, V. Hassija, V. Gupta, and M. Guizani, ‘‘A comprehen- https://www.mdpi.com/1660-4601/17/11/3819
sive review of the COVID-19 pandemic and the role of IoT, drones, [24] T. W. Teng, P. Veerajagadheswar, B. Ramalingam, J. Yin, R. E. Mohan,
AI, blockchain, and 5G in managing its impact,’’ IEEE Access, vol. 8, and B. F. Gómez, ‘‘Vision based wall following framework: A case
pp. 90225–90265, 2020. study with HSR robot for cleaning application,’’ Sensors, vol. 20, no. 11,
[2] M. Saez, A. Tobias, D. Varga, and M. A. Barceló, ‘‘Effectiveness of the p. 3298, Jun. 2020. [Online]. Available: https://www.mdpi.com/1424-
measures to flatten the epidemic curve of COVID-19. The case of Spain,’’ 8220/20/11/3298
Sci. Total Environ., vol. 727, Jul. 2020, Art. no. 138761. [25] K. P. Cheng, R. E. Mohan, N. H. K. Nhan, and A. V. Le, ‘‘Multi-objective
[3] M. Gupta, M. Abdelsalam, and S. Mittal, ‘‘Enabling and enforcing genetic algorithm-based autonomous path planning for Hinged-Tetro
social distancing measures using smart city and its infrastructures: reconfigurable tiling robot,’’ IEEE Access, vol. 8, pp. 121267–121284,
A COVID-19 use case,’’ 2020, arXiv:2004.09246. [Online]. Available: 2020.
https://arxiv.org/abs/2004.09246 [26] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler,
[4] E. T. Hall, The Hidden Dimension, vol. 609. Garden City, NY, USA: and A. Y. Ng, ‘‘ROS: An open-source robot operating system,’’ in Proc.
Doubleday, 1966. ICRA Workshop Open Source Softw., Kobe, Japan, 2009, vol. 3, nos. 3–2,
[5] S. Weitz, Nonverbal Communication: Readings With Commentary. p. 5.
New York, NY, USA: Oxford Univ. Press, 1979. [27] U. Yamaguchi, F. Saito, K. Ikeda, and T. Yamamoto, ‘‘HSR, human support
robot as research and development platform,’’ in Proc. Abstr. Int. Conf.
[6] P. Mishra, ‘‘Proxemics: Theory and research,’’ Perspect. Psychol. Res.,
Adv. Mechatronics, Toward Evol. Fusion IT Mechatronics (ICAM), 2015,
vol. 6, pp. 10–15, 1983.
pp. 39–40.
[7] P. Veerajagadheswar, K. Ping-Cheng, M. R. Elara, A. V. Le, and M. Iwase, [28] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, ‘‘OpenPose:
‘‘Motion planner for a tetris-inspired reconfigurable floor cleaning robot,’’ Realtime multi-person 2D pose estimation using part affinity fields,’’ 2018,
Int. J. Adv. Robot. Syst., vol. 17, no. 2, 2020, Art. no. 1729881420914441. arXiv:1812.08008. [Online]. Available: http://arxiv.org/abs/1812.08008
[8] A. V. Le, P. T. Kyaw, R. E. Mohan, S. H. M. Swe, A. Rajendran, [29] J. Yin, K. G. S. Apuroop, Y. K. Tamilselvam, R. E. Mohan, B. Ramalingam,
K. Boopathi, and N. H. K. Nhan, ‘‘Autonomous floor and staircase clean- and A. V. Le, ‘‘Table cleaning task by human support robot using deep
ing framework by reconfigurable sTetro robot with perception sensors,’’ learning technique,’’ Sensors, vol. 20, no. 6, p. 1698, Mar. 2020.
J. Intell. Robot. Syst., vol. 101, no. 1, pp. 1–19, Jan. 2021. [30] A. Le, P.-C. Ku, T. T. Tun, N. H. K. Nhan, Y. Shi, and R. Mohan,
[9] J. K. Burgoon, D. B. Buller, and W. G. Woodall, Nonverbal Communica- ‘‘Realization energy optimization of complete path planning in differential
tion: The Unspoken Dialogue. New York, NY, USA: Harpercollins College drive based self-reconfigurable floor cleaning robot,’’ Energies, vol. 12,
Division, 1989. no. 6, p. 1136, Mar. 2019.

VOLUME 9, 2021 41415

 A. V. Le et al.: Social Density Monitoring Toward Selective Cleaning by HSR With 3D-Based Perception System

[31] A. V. Le, N. H. K. Nhan, and R. E. Mohan, ‘‘Evolutionary algorithm-based BRAULIO FÉLIX GÓMEZ received the bachelor’s degree in mechatronic
complete coverage path planning for tetriamond tiling robots,’’ Sensors, and bachelor’s degree in informatics engineering from the Technological
vol. 20, no. 2, p. 445, Jan. 2020. Institute of Los Mochis, Mexico, in 2018 and 2019, respectively. He is a
[32] A. Le, V. Prabakaran, V. Sivanantham, and R. Mohan, ‘‘Modified a-star Research Assistant with ROAR Laboratory, Singapore University of Tech-
algorithm for efficient coverage path planning in Tetris inspired self- nology and Design (SUTD). His research interests include robotics vision,
reconfigurable robot with integrated laser sensor,’’ Sensors, vol. 18, no. 8, robotics control, and AI based robotics.
p. 2585, Aug. 2018.
[33] A. V. Le, A. A. Hayat, M. R. Elara, N. H. K. Nhan, and K. Prathap, ‘‘Recon-
figurable pavement sweeping robot and pedestrian cohabitant framework
by vision techniques,’’ IEEE Access, vol. 7, pp. 159402–159414, 2019.
[34] L. Yi, A. V. Le, A. A. Hayat, C. S. C. S. Borusu, R. E. Mohan,
N. H. K. Nhan, and P. Kandasamy, ‘‘Reconfiguration during locomotion
by pavement sweeping robot with feedback control from vision system,’’
IEEE Access, vol. 8, pp. 113355–113370, 2020.
[35] Y. Shi, M. R. Elara, A. V. Le, V. Prabakaran, and K. L. Wood, ‘‘Path
tracking control of self-reconfigurable robot hTetro with four differential RAJESH ELARA MOHAN received the B.E.
drive units,’’ IEEE Robot. Autom. Lett., vol. 5, no. 3, pp. 3998–4005, degree from Bharathiar University, India, in 2003,
Jul. 2020. and the M.Sc. and Ph.D. degrees from Nanyang
[36] M. A. V. J. Muthugala, A. V. Le, E. S. Cruz, M. R. Elara, Technological University, in 2005 and 2012,
P. Veerajagadheswar, and M. Kumar, ‘‘A self-organizing fuzzy logic classi- respectively. He is currently an Assistant Professor
fier for benchmarking robot-aided blasting of ship hulls,’’ Sensors, vol. 20, with the Engineering Product Development Pillar,
no. 11, p. 3215, Jun. 2020. Singapore University of Technology and Design.
[37] A. K. Lakshmanan, R. E. Mohan, B. Ramalingam, A. V. Le,
He is also a Visiting Faculty Member with the
P. Veerajagadeshwar, K. Tiwari, and M. Ilyas, ‘‘Complete coverage
path planning using reinforcement learning for tetromino based cleaning International Design Institute, Zhejiang Univer-
and maintenance robot,’’ Autom. Construct., vol. 112, Apr. 2020, sity, China. He has published over 80 papers in
Art. no. 103078. leading journals, books, and conferences. His research interests include
[38] A. V. Le, R. Parween, R. E. Mohan, N. H. K. Nhan, and R. E. Abdulkader, robotics with an emphasis on self-reconfigurable platforms and research
‘‘Optimization complete area coverage by reconfigurable hTrihex tiling problems related to robot ergonomics and autonomous systems. He was a
robot,’’ Sensors, vol. 20, no. 11, p. 3170, Jun. 2020. recipient of the SG Mark Design Award in 2016 and 2017, the ASEE Best of
[39] S. Kohlbrecher, O. von Stryk, J. Meyer, and U. Klingauf, ‘‘A flexible and Design in Engineering Award in 2012, and the Tan Kah Kee Young Inventors’
scalable SLAM system with full 3D motion estimation,’’ in Proc. IEEE Int. Award in 2010.
Symp. Saf., Secur., Rescue Robot., Nov. 2011, pp. 155–160.
[40] N. Wojke, A. Bewley, and D. Paulus, ‘‘Simple online and realtime tracking
with a deep association metric,’’ in Proc. IEEE Int. Conf. Image Process.
(ICIP), Sep. 2017, pp. 3645–3649.
[41] N. Wojke and A. Bewley, ‘‘Deep cosine metric learning for person re-
identification,’’ in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV),
Mar. 2018, pp. 748–756.
[42] R. Kirby, ‘‘Social robot navigation,’’ Ph.D. dissertation, Robot. Inst.,
Carnegie Mellon Univ., Pittsburgh, PA, USA, May 2010.
TRAN HOANG QUANG MINH received the
B.S. degree in electronics and telecommunications
from the Hanoi University of Technology, Viet-
nam, in 2007, and the Ph.D. degree in electron-
ANH VU LE received the B.S. degree in elec- ics and electrical from Dongguk University, South
tronics and telecommunications from the Hanoi Korea, in 2015. He is currently with the Opto-
University of Technology, Vietnam, in 2007, and electronics Research Group, Faculty of Electrical
the Ph.D. degree in electronics and electrical and Electronics Engineering, Ton Duc Thang Uni-
from Dongguk University, South Korea, in 2015. versity, Ho Chi Minh City, Vietnam. He is also a
He is currently with the Optoelectronics Research Postdoctoral Research Fellow with ROAR Labora-
Group, Faculty of Electrical and Electronics Engi- tory, Singapore University of Technology and Design. His current research
neering, Ton Duc Thang University, Ho Chi Minh interests include robotics vision, robot navigation, human detection, action
City, Vietnam. He is also a Postdoctoral Research recognition, feature matching, and 3D video processing.
Fellow with ROAR Laboratory, Singapore Univer-
sity of Technology and Design. His current research interests include robotics
vision, robot navigation, human detection, action recognition, feature match-
ing, and 3D video processing.

BALAKRISHNAN RAMALINGAM received the

B.Sc. and M.Sc. degrees in electronics from VINU SIVANANTHAM received the B.Tech.
Bharathidasan University, Tiruchirapalli, India, degree in electronics and communication engi-
in 2006 and 2009, respectively, and the Ph.D. neering from Amrita University, in 2017. From
degree in VLSI design from SASTRA University, 2015 to 2017, he was a Junior Research Assis-
Thanjavur, India. He is a Postdoctoral Research tant with HuT Laboratory, where he was working
Fellow with ROAR Laboratory, Singapore Uni- on mechatronics and software development. He is
versity of Technology and Design, Singapore. His currently a Visiting Fellow with the Singapore
research interests include artificial intelligence, University of Technology and Design, where he is
computer vision and embedded system design, working on the mechatronics and software devel-
reconfigurable hardware, information security, and multiprocessor system opment of reconfigurable robots.
on chips.

41416 VOLUME 9, 2021