Received September 3, 2018, accepted October 22, 2018, date of publication October 29, 2018,

date of current version November 19, 2018.

Digital Object Identifier 10.1109/ACCESS.2018.2878423

Multi-Sensor Obstacle Detection System Via

Model-Based State-Feedback Control in Smart
Cane Design for the Visually Challenged
NUR SYAZREEN AHMAD , (Member, IEEE), NG LAI BOON, AND PATRICK GOH
School of Electrical and Electronic Engineering, Engineering Campus, Universiti Sains Malaysia, Nibong Tebal 14300, Malaysia.
Corresponding author: Nur Syazreen Ahmad ([email protected])
This work was supported by the Ministry of Education of Malaysia through the FRGS Scheme under Grant 203/PELECT/6071347.

ABSTRACT Smart canes are usually developed to alert visually challenged users of any obstacles beyond
the canes’ physical lengths. The accuracy of the sensors and their actuators’ positions are equally crucial
to estimate the locations of the obstacles with respect to the users so as to ensure only correct signals
are sent through the associated audio or tactile feedbacks. For implementations with low-cost sensors,
however, the users are very likely to get false alerts due to the effects from noise and their erratic readings,
and the performance degradation will be more noticeable when the positional fluctuations of the actuators
get amplified. In this paper, a multi-sensor obstacle detection system for a smart cane is proposed via a
model-based state-feedback control strategy to regulate the detection angle of the sensors and minimize the
false alerts to the user. In this approach, the overall system is first restructured into a suitable state-space
model, and a linear quadratic regulator (LQR)-based controller is then synthesized to further optimize
the actuator’s control actions while ensuring its position tracking. We also integrate dynamic feedback
compensators into the design to increase the accuracy of the user alerts. The performance of the resulting
feedback system was evaluated via a series of real-time experiments, and we showed that the proposed
method provides significant improvements over conventional methods in terms of error reductions.

INDEX TERMS Multi-sensor, visually challenged, model-based control, state-feedback, obstacle detections.

I. INTRODUCTION is a type of ETA which is typically designed to fit on top of

White canes are universally recognized as symbols of blind the white cane for obstacle detections above-knee levels. This
people, and they have been used since the 1920s as mobility device is intended to help the visually challenged to engage in
aids to guide users while walking or navigating particularly a safe and efficient independent travel by increasing the user
in unfamiliar places [1]. Other than their basic function which access to certain categories of environmental information.
is to give the users tactile information about the environment The most common technologies that are used for distance
such as obstacles on the ground, holes and uneven surfaces, measurements from obstacles include infrared (IR) sensors
most of them are designed in such a way that they are light and which transmit IR lights towards the objects, sonar sensors
retractable or foldable, which can increase the travel conve- which use high-frequency sound waves in place of IR lights,
nience of the users. Nevertheless, these traditional travel aids and laser rangefinders which produce laser waves for the
only have short sensing ranges which limit the obstacle detec- same purpose. The IR sensor technology works by measuring
tions below the knee levels and require the canes to physically the signal strength to estimate the distance. The advantages of
bump into the objects to alert the users. Due to these limi- IR sensors over other sensor types are faster response, narrow
tations, electronic travel aids (ETAs) were introduced in the range and high resolution which makes them more suited
1970s [1]–[3], not just to extend the sensing range, but also for small distance measurement [4], [5]. The accuracy of this
to promote a safer and more confident independent walking type of sensor however is affected by the reflectivity and color
experience. of the objects due to its dependence on signal strength. As it
Sensor technology is one of the most important factors that is also very sensitive to the sunlight, its reading can vary if
can enhance the performance of the ETAs. Smart cane device the luminance in the environment changes [6]. This drawback

2169-3536 2018 IEEE. Translations and content mining are permitted for academic research only.
64182 Personal use is also permitted, but republication/redistribution requires IEEE permission. VOLUME 6, 2018
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

however is not an issue to the sonar and laser range finders detection range can be achieved, the size and power consump-
which share a similar working principle commonly known as tion of the device will also increase which directly affect the
Time-of-Flight (ToF). This principle leads to a simple mathe- price, portability and mobility of the device. For some cases,
matical calculation for distance measurement from the sensor a lot of variations in the user alerts may be confusing and
to the object, that is, by generating a beam of energy waves less intuitive, and users usually prefer to receive simplified
directed towards an object, the time it takes for the beam to signals without having to process a lot of raw data from
make its journey back towards its source after being reflected the feedbacks. These issues have sparked a growing interest
can be used to estimate the distance traveled. As laser beam among researchers to study on the design guidelines and
offers a distinct advantage of being able to travel many times improvements that can be introduced to increase the usability
faster than that provided by sonar sensors, it is often used to and marketibility of the devices [10], [11], [28].
detect both static and moving objects [7]. In [8], a fusion of a Inspired by a number of recent smart cane configurations
laser sensor and a camera for an electronic virtual white cane [6], [14], [15], a simple design with integration of ultrasonic
implementation was proposed where the distance calculation and IR sensors for obstacles detection, and vibrotactile and
was based on the laser’s position and the image captured. audio techniques for the feedbacks has been implemented
Nevertheless, for simple design goals, some preferred to for the prototype smart cane in this work. The sensors are
avoid using this kind of technology as the laser light is positioned in such a way that the sensing range of the ultra-
known to be harmful to humans, and extra precautions are sonic sensors includes the left and right front of the user, and
required when using them [9]. The sonar technology, on the from the ground level to the head level, while the IR sensor is
other hand, despite its low resolution and slower response as used for uneven ground surface detection such as holes and
compared to IR and laser, is still being preferred by many descending stairs. Since the device involves low-cost sensors,
developers and has become among the most common sensing the signals sent to the user are prone to noise and erratic
technique for mobility aids. This is mainly due to its low-cost readings which may lead to false alarms, and the performance
and broad beam-width which allows for a wide detection will become worse when the sensors’ positions oscillate with
range [1], [10]. the user’s hand movements. As suggested in [28], the hor-
As the sensor technology is rapidly evolving in paral- izontal orientation of the cane can be fixed by including a
lel with the emerging trends in Internet-of-Things (IoT) mark or indicator on the handlebar to ensure the sensors are
and embedded systems, many refinements of the early always facing forward. While this can be controlled by the
ETAs with new innovative technologies have been developed user, the sensors’ vertical detection angle is bound to fluctuate
in modern assistive devices [11]. Integration of multiples since the user usually has to tilt the cane back and forth
sensors on a single platform to overcome the limitations while walking. Motivated by these issues, the focus of this
of individual sensors has become increasingly popular in work is on improvement of the obstacle detection system
recent years [4], [12]–[17]. Combinations of multiple ultra- by means of model-based state feedback technique. In this
sonic and other sensors have been reported in a series of approach, the overall system is first restructured into a suit-
papers [13], [15], [17], [18] to accommodate a wider range of able state-space model which also includes an accelerometer
obstacles and sensing area. There is also a growing number to sense the tilt angle. A motorized actuator is used to control
of recent studies on microcontroller-based assistive devices the vertical detection angle of the ultrasonic sensors, and a
which allow faster user alerts via various types of actu- linear quadratic regulator (LQR)-based controller is synthe-
ators and wireless feedbacks. Vibrators, for instance, are sized to further optimize the actuator’s control actions while
extensively used to provide haptic or vibrotactile feedbacks ensuring its position tracking. We also integrate dynamic
with different intensities [14], [19], [20]. Another interest- feedback compensators into the design which additionally act
ing approach by [21] and [22] where steering actions of a as noise filters to increase the accuracy of the user alerts. The
mini wheeled mobile robot attached to a white cane was performance of the resulting feedback system was evaluated
introduced to provide a vibrotactile feedback with a sense via a series of real-time experiments, and we showed that the
of direction. Apart from that, audio voice/texts or acoustic proposed method provides significant improvements over the
feedbacks have also been considered by many researchers conventional methods in terms of error reductions.
which can alert the users wirelessly through smartphones
and/or headsets [16], [18], [20], [21], [23]–[26]. II. METHODOLOGY AND MAIN RESULTS
Despite the technological revolution, the assistive devices The overall view of the smart cane system configuration is
have not been successfully adopted and used by a large depicted as in Figure 1 where five sensors (one accelerome-
number of people with visual impairments, and many still ter, Sa , three ultrasonic sensors Sh , Smr , Sml and one infrared
prefer to use the white canes [27], [28]. Several research sensor, Sg ) serve as the interface for input signals to an
findings have shown issues related to the limited use of ATmega328p microcontroller, a servo motor to control the
these smart devices, which include high prices, safety, sensors’ positions, a vibration motor for the vibrotactile alert,
orientations, speed, mobility, portability and optimizations and a bluetooth module for wireless audio feedback. The
of techniques [28], [29]. Combinations of many different focus of this work is on improving the performance of the
sensors, for instance, although faster feedback and wider user alerts which rely on the accuracy of the sensors’ positions

VOLUME 6, 2018 64183

 N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

TABLE 1. Notations used in this work.

FIGURE 1. Overview of the smart cane system architecture. The scope of

this work is highlighted by the solid blue and red arrows which represent
the signals to/from actuators/sensors and control algorithms respectively.

and readings. These are highlighted by the blue arrows signals

as in the figure, and the red arrows which also indicate the
associated control algorithms.
The sensors Sa , Sh , Smr , Sml are attached to a smart
cane (SC) board, and are positioned in such a way that any
obstacles from the head level to the ground level in front of
the user can be detected. The sketch of the detection range
is illustrated in Figure 2 together with Sg for the drop-off
detection at the ground level. Appropriate signals can then
be transmitted to the user via vibrotactile and wireless audio
feedbacks to an Android device and headset/speaker via blue-
tooth for obstacle detection alerts.
FIGURE 3. System modelling for the signals around the SC board.

angle with respect to xs -axis respectively. The distances of

Sml , Smr and Sh from obstacles as illustrated in Figure 4 are
written as x2 ∈ R3 with x2 = [x21 x22 x23 ]T , and the distance
of Sg from the drop-off location as x3 ∈ R. The state-vector
can then be represented by xo = [x1 x2 x3 ]T . The angles θh ,
θm and θg are fixed and can be adjusted according to the user’s
height or convenience. As for the outputs, let y1 , y2 , y3 and yd
represent the servo’s angle w.r.t the x-axis, the input to serial
communication via bluetooth for voice alert, the input to the
vibration motor (tactile feedback), and the smart cane’s angle
with respect to xs -axis respectively. The inputs to the system
FIGURE 2. Sketch of obstacle and drop-off detection range with will be u1 (input to the servo), u02 = [u21 u22 u23 ]T (input
three-leveled sensors (Sa , Sg , Sml , Smr and Sh ). signals to Sml , Smr and Sh respectively), u03 (input signal
to Sg ) and ud (user’s movement).
In what follows, we present an equivalent open-loop model The open-loop system can then be written as
of the SC system in state space domain which is restruc-
x˙o = Axo + Buo (1)
tured to pave the way for model-based state-feedback control
design. The notations used throughout this paper are listed yo = Cxo + Duo (2)
in Table 1.
where
   
A. OPEN-LOOP SYSTEM MODELLING A1 0 0 B1 0 0 Bd
With reference to Figure 3, define x1 = [x11 x12 ]T where A=0 0 0 , B=0 0 0 0 ,
x11 and x12 represent the servo’s angle and accelerometer’s 0 0 0 0 0 0 0

64184 VOLUME 6, 2018

N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

FIGURE 5. Open-loop response of the system Ga without any

state-feedback for three different scenario; Movement 1 (top),
Movement 2 (middle) and Movement 3 (bottom). The orange line
represents the duty cycle of u1 , and the light blue line represents y1 .

FIGURE 4. Close-up on the side view (top) and top view (bottom) of the
smart cane system with the sketch of its detection range. subsystem Ga , i.e.

  Ga ∼ (A1 , B1 , C1 , 0);
   
C1 0 0 A 0 B
0 C2 0 A1 = 11 , B1 = 11 , C1 = [1 − 1] (4)
C = , D = 04×4 0 0 0
0 0 C3 
Cd 0 0 Without any state feedback, the first state cannot be controlled
at all. Assuming x11 = α where α ∈ R is a constant,
yo = [y1 y2 y3 yd ] ,
T
uo = [u1 u02 u03 T
ud ] ;
the uncontrolled output y1 then reduces to y1 = α − x12 .
(3) If α = 0 for instance, the output will be the inverse of the
accelerometer’s angle from the xs -axis. In order to estimate
with A1 ∈ R2×2 , B1 ∈ R2×1 , C1 , Cd ∈ R1×2 , Bd , C2 ∈ R1×3 the state-space model of the main actuator, three different
and C3 ∈ R. The matrices A1 , B1 , C1 , C2 and C3 depend control input profiles which represent the responses from the
on the model of the main actuator and the sensors, while servo motor based on three different user movements were fed
Bd and Cd rely on the user’s hand movement. It is also into the system, and the outputs were then compared with the
straightforward that Cd = [0 γ ], γ ∈ R, as the orientation inputs as shown in Figure 5. Via the open-loop model, the user
of Sa is parallel with the SC-board’s. can also be alerted of the obstacle’s position and drop-off
Although servo motors generally provide a perfect through x2 and x3 . For instance, setting C3 = 1, a warning
steady-state tracking particularly for step responses, with the to the user for the drop-off can be delivered via y3 which then
ultrasonic sensors and other load attached, their dynamic activates the vibrator. As for the obstacle positions, the alerts
will be slightly affected. Moreover, as the nature of the via y2 will be sent to the user via wireless serial transmit,
inputs is always uncertain and highly depends on the hence discretized signals for the audio feedbacks are more
user’s movement, it is therefore useful to take into account suitable. This can be acheived by assigning
the actuator’s dynamic in order to ensure the tracking 
behaviour stays within the desired specifications. From the 1 when u2i 6= 0
x2i = (5)
system’s architecture, we can write the main actuator as a 0 when u2i = 0

VOLUME 6, 2018 64185

 N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

 
for i = 1, 2, 3 and C2 = [1 2 4]. The relationship between x2   0 0 0 0
C1 01×2 0 D2 0 0
and y2 along with the user alert is summarized in Table 2. C̃ = , D̃ =   (6)
03×2 03×2 0 0 D3 0
0 0 0 D4
TABLE 2. User alert of the obstacle’s position via open-loop control.

with A1 ∈ R2×2 , B1 ∈ R2×1 , C1 , ∈ R1×2 , D2 , D3 ∈ R and

D4 ∈ R3 . The main compensators are represented by
 
F1 0 0
Kr ∈ R3×3 ; F =  0 F2 0  and
0 0 F3
 
H1 0 0
0 82 0 .
H = (7)
0 0 83 
Although the abovesaid methods can be easily imple- 0 84 0
mented, the users are very likely to get false alerts due to
with F1 , H1 ∈ R, F2 , F3 ∈ RH3×3 ∞ , and 82 , 83 , 84 :
fluctuations of detection areas, along with noise and erratic
R3 → R. Apart from that, the limitation of the actuator,
readings from the sensors. In the next subsection, we intro-
L = diag(φL , 1, 1, 1), is also included to accurately model
duce a model-based state-feedback control design to enhance
the system. This gives ũ = [φL (ub1 ) ub2 ub3 ub4 ]T where φL :
the performance of the smart cane system by minimizing the
R → R represents the constraint of the main actuator, i.e.
false alerts to the users.
for ub1 < 0

0
u1 = φL (ub1 ) = ub1 for 0 ≤ ub1 ≤ 180 (8)
180 for ub1 > 180.


1) DESIGN SPECIFICATIONS AND CONTROLLER SYNTHESIS

Let r̃ = [r1 r2 r3 ]T , Kr = diag(K1 , K2 , K3 ) and define the
output of controller F as β̃ = [β1 β2 β3 ]T . It is desired that
y1 tracks the reference at r1 when the SC-board’s orientation
fluctuates between 0◦ and −90◦ from the x-axis. In order to
FIGURE 6. Proposed model with state-feedback control strategy. achieve this, the following method is proposed:
Result 1: Consider the proposed closed-loop model as
depicted in Figure 6, let H1 = 1, R1 (s) = L {r1 (t)} = v/s
where v ∈ R− is the angle in degree, and G1 ∼ (A11 , B11 ,
B. MODEL-BASED STATE FEEDBACK CONTROL DESIGN
C11 , 0) with C11 = 1. Define the quadratic cost function as
In this work, it is desired that the angle of the main actuator Z ∞
stays at an optimal point (i.e. slightly below the x-axis) to T
J= (x11 Qx11 + uTb1 Rub1 ) dt (9)
ensure accurate obstacle position detection, and the outputs of 0
the microcontroller which are fed to the vibration motor and
bluetooth module are able to alert the user on any obstacles where Q, R ∈ R+ .
The angle of the main actuator, y1 will
and drop-off ahead. To this end, a state-feedback approach track its reference at r1 with controllers F1 and K1 which can
is introduced as shown in Figure 6. It is worth noting that, be designed with
from the open-loop model in (1)-(3), only y1 , y2 and y3 F1 = R−1 BT11 P and K1 = Gc1 (0)−1 (10)
can be regulated by the microcontroller, and the exogenous
signal ux = [u02 u03 ud ]T depends on the user and inputs where P ∈ R+ satisfies
to the sensors. In this state-feedback design, another new
output, y4 , is augmented to send a signal when the obstacle AT11 P + PA11 − PB11 R−1 BT11 P + Q = 0 (11)
is within a predefined distance (the detail is in requirement and
(A2) below) from the user. As this entails a new control
scheme to detect the obstacle’s position, a new control input Gc1 (s) = C11 (sI − (A11 − B11 F1 ))−1 B11 (12)
vector ũ is introduced. The new output and control input
vectors can then be formed as ỹ = [y1 y2 y3 y4 ]T and Proof: The transfer function G1 (s) corresponds to the
ũ = [u1 u2 u3 u4 ]T respectively, which results in the following subsystem of the feedback loop for the state x11 . With K1 = 0
state space matrices: and H1 = 1, the system reduces to a standard state-feedback
    control framework where the control law F1 can be designed
A1 02×2 B1 02×3 by selecting appropriate values of Q and R and minimizing
à = , B̃ = ,
02×2 02×2 02×1 02×3 the cost function (9). The latter can also be simplified by

64186 VOLUME 6, 2018

N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

solving (11) [30]. The perfect reference tracking can then be TABLE 3. Accuracy (%) of the estimated model based on three different
movements. M1, M2 and M3 denote Movements 1,2 and 3 respectively.
achieved by including K1 ∈ R+ where

K1−1 = lim sGc1 (s)R(s), (13)

s→0

which is also equivalent to the second equation in (10). 

Other than ensuring the reference tracking for the main actu-
ator to minimize its positional fluctuations, it is also desired
that III. NUMERICAL AND EXPERIMENTAL RESULTS
A. SYSTEM MODELLING FOR THE MAIN ACTUATOR
(A1) the user is notified of the obstacles’ positions via y2 and
Three types of movements as explained in Section II-A
any drop-offs via y3 ;
have been considered in this work. The corresponding
(A2) the user is alerted when the obstacle is within 50cm to
responses as shown in Figure 5 were compared via MATLAB
80cm via y4 to provide a comfortable stopping distance
System Identification Toolbox to estimate the parameters
at a normal walking speed;
A11 and B11 . For each movement, estimated values of A11 and
(A3) the effects of x2 on y2 from any fast moving object that
B11 were generated, and the accuracy of the response for each
is not approaching the user must be suppressed;
(A11 , B11 ) pair was compared for each case. The results were
(A4) the number of false alerts is smaller than that from the
summarized in Table 3. From the table, the estimated model
open-loop approach.
with the highest accuracy on average is given by (A11 , B11 ) =
In this regard, we propose the following control algorithm. (−78.67, 77.17). The responses of y1 in open-loop via this
Result 2: Let D2 = D3 = D4 = 1, F2 ∼ (Af 2 , Bf 2 , estimated model (simulation) and experiment for the three
Cf 2 , Df 2 ) and F3 ∼ (Af 3 , Bf 3 , Cf 3 , Df 3 ) be designed such types of movements are shown in Figure 7.
that eig (Afi ) ∈ R− ,|Afi | ≥ 0.5, Fi (0) = 1 for i = 1, 2,
β3 = F3 x3 and β2 = F2 x2 with β2 = [β21 β22 β23 ]T . Also,
write r2 = [0 0 0]T and r3 = 0 so that

ub2 = −82 (β2 ), ub3 = −83 (β3 ) and ub4 = −84 (β2 )
(14)

If 82 , 83 and 84 satisfy the following constraints

0 if β2i ≤  ∀i = 1, 2, 3


7 if − β2i ≤ − ∀i = 1, 2, 3




if (β21 , −β22 , −β23 ) ≤ (, −, −)

6



5 if (−β21 , β22 , −β23 ) ≤ (−, , −)


−82 (β2 ) = (15)
 4 if (β21 , β22 , −β23 ) ≤ (, , −)
if (−β21 , −β22 , β23 ) ≤ (−, −, )

3



(β21 , −β22 , β23 ) ≤ (, −, )

2 if



1 if (−β21 , β22 , β23 ) ≤ (−2 , 1 , 1 )



1 if β3 > 0

−83 (β3 ) = (16)
0 otherwise.
1 if 50 − l < β2i < 80 − u for any i

−84 (β2 ) = (17)
0 otherwise.
where  ∈ [25, 40] and l , u ∈ [10, 30], the outputs y2 , y3
and y4 can be controlled to meet the design requirements as
described in (A1)-(A4).
Proof: From the parameters of D̃ and algorithms for
ub2 , ub3 and ub4 , the requirements (A1) and (A2) can be
clearly met. In order to satisfy (A3) and (A4), F2 and F3 are
designed such that the output maintains its stability (via eig FIGURE 7. Comparison of the output y1 in open-loop via simulation and
(Afi ) ∈ R− and Fi (0) = 1), and the false alerts due to the experiment.

noise/erratic readings from the sensors can be minimized by

delaying the output response with |Afi | ≥ 0.5. Also, due to the B. CONTROL SYNTHESIS AND PERFORMANCE
dynamic properties of F2 and F3 , the threshold values of , l EVALUATIONS
and u are included which can be selected after the system Applying Result 1 with Q = 1.1 and R = 0.2 to the best
calibrations.  estimated model from Table 3, we obtained an optimized

VOLUME 6, 2018 64187

 N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

FIGURE 8. Output y1 from the experiment with corresponding duty cycles FIGURE 10. Output y1 from the experiment with corresponding duty
for Movement 1. Both methods show a large overshoot due to unstable cycles for Movement 3. Slightly larger overshoots are seen from the
movement of the user at the beginning (at t ≤ 3s). Slightly larger response of FKn at t ∈ (0, 1)s, t ∈ (7, 8)s and t ∈ (13, 14)s.
overshoots are seen from the response of FKn at t ≈ 7.5s, t ≈ 15.5s
and t ≈ 18s.

errors (IAEs), i.e.

Z ∞
IAE = |e(t)|dt, e(t) = r1 (t) − y1 (t) (18)
0
are summarized in Table 4, and it can also be concluded
that the method proposed in Result 1 provides a significant
reduction in terms of the position errors.

TABLE 4. Performance evaluation for the response of y1 via optimized

and non-optimized compensators.

FIGURE 9. Output y1 from the experiment with corresponding duty cycles

for Movement 2. Larger overshoots are seen from the response of FKn
due to the duty cycles which go beyond the limits (at t ≈ 2s, t ∈ (8, 11)s
and t ∈ (19, 21)s). With regard to the obstacle avoidance, the controllers
F2 and F3 have been designed via Result 2 with
Af 3 = −1.5, Bf 3 = 1.5, Cf 3 = 1; Df 3 = 0; (19)
compensator with F1 = 1.5378 and K1 = 2.5572. In order to 
−0.5 0 0

test the position tracking performance of the main actuator, Af 2 =  0 −0.5 0  (20)
a suitable angle of v = −10◦ was selected. The responses 0 0 −0.55
of y1 with respect to Movements 1,2 and 3 together with the  
corresponding duty cycles (which map (0, 1) to (0◦ , 180◦ )) 0.5
Bf 2 =  0.5  , Cf 2 = 1 1 1
 
are illustrated in Figures 8, 9 and 10 respectively. The yellow Df 2 = 03×3 (21)
line (i.e. ref ) represents r1 = v whereas FKopt and FKn 0.55
denote the y1 responses via the optimized and non-optimized For the implementation on the microcontroller, F2 and F3
(i.e. K1 = 1, and F1 = 1) compensators respectively. From were discretized via bilinear trasformation method with a
the figures, it is observed that both methods are generally sampling time of 0.2s. The signal x3 was configured to output
able regulate the output at r1 . However, relatively larger ‘‘1’’ when there is a drop-off, and ‘‘0’’ otherwise. The thresh-
positional fluctuations can be seen from the response via FKn olds of  = 30, l = u = 20 have also been selected after
for each movement. This is mainly due to the non-optimized the system calibrations.
duty cycles (or control signals) which go beyond the φL In order to evaluate the performance via y2 , y3 and y4 , six
limits for a certain period of time during the movements. The different experiments were structured as depicted in Figure 11
responses via FKopt on the other hand show a significant where Experiments 1,2,3,4,5 and 6 were represented by sub-
improvement as the compensator’s parameters have been figures (a), (b), (c), (d), (e) and (f) respectively. The smart
optimized to ensure the duty cycles stay within the constraints cane user is indicated by the blue circle whereas the static
for all movements. The corresponding integral of absoulte obstacle on the ground is denoted by the grey rectangle.

64188 VOLUME 6, 2018

N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

FIGURE 12. Side views for experiments with obstacle on the ground (top)
and drop-off (bottom).

TABLE 5. Performance evaluations from y2 , y3 and y4 responses via

open-loop (OL) and Result 2 (R2.2).

FIGURE 11. Experiments for the obstacle avoidance performance

evaluations; the user is represented by the blue circle and the static
obstacle on the ground is represented by the grey rectangle. Expected
drop-off areas are indicated by the small dark-red circles. The dashed red
lines show the user’s predefined walking path and direction (with speed
approximately at 20cm/s). The specified distances/length are a = 50,
b1 = 120, b2 = 150, b3 = 130 and c = 45. From left to right: (a) Obstacle
on the left; (b) Obstacle on the right; (c) Obstacle at the center; (d) A
hanging obstacle at the center; (e) A non-moving human as a high
obstacle at the center; (f) A human walking from left to right at around
60cm/s as a moving obstacle. Let yir (i = 2, 3, 4) be the expected output signals of yi
where y2r and y3r are the desired user alerts for obstacle posi-
The dashed red lines show the user’s predefined walking tion and drop-off as in requirement (A1), and y4r is the desired
path and direction (with speed approximately at 20cm/s). alert to satisfy (A2). Also, let yib denote yi when Result 2
Figure 12 illustrates the side views for the experiments with is applied, and yix be yi produced via open-loop mode. The
obstacle on the ground and drop-off. The smart cane was responses for yi were recorded in Figures 13, 14, and 15
attached to a 1.16m walking stick for a user with a height which also showed the corresponding xi and βi . From the
of 155cm. The angles θh , θm and θg (as depicted in Figure 4) figures, it was observed that the proposed method is able
were respectively fixed to 50◦ , 40◦ and 45◦ . For all the exper- to significantly reduce the number of false alerts as can be
iments, the user walked from the same starting point, and the seen from the responses of yib and yix . This was mainly
obstacles were initially bj (j = 1, 2, 3) cm away from the user. due to the compensators Fi that produced filtered output βi
Experiments 1, 2 and 3 were designed to analyse the obstacle from the raw sensor signals xi which usually suffered from
detection performance when a static obstacle with a height of the sudden drop-to-zero issues. With regard to Experiment 6
20cm on the ground was placed on the left, right and at the where another person was moving fastly from left to right
center with respect to the user, whereas for a hanging obstacle at b3 cm in front of the user, the proposed method was also
(approximately at the user’s head-level), static and moving capable to reduce the number of alerts as can be observed
humans, the performance were evaluated via Experiments from the last column in Figure 14. This in turn satisfied the
4,5 and 6 respectively. To test the controller’s performance for requirement in (A3).
Sg , the drop-off areas are included in Experiments 4 and 6 as The total error (TE) for quatitative performance evaluation
indicated by the small dark-red circles. of y2 was calculated similar to the IAE as in (18), with

VOLUME 6, 2018 64189

 N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

FIGURE 13. The responses of x2 , β2 and y2 for Experiments 1,2 and 3 are represented by the left, middle and right subfigures respectively.

FIGURE 14. The responses of x2 , β2 and y2 for Experiments 4,5 and 6 are represented by the left, middle and right subfigures respectively.

y1 and r1 replaced by y2 and y2r . For y3 and y4 , the TEs were the nature of ub3 and ub4 and suitability of the alerts to the
evaluated slightly different than that for y2 to accommodate user. To this end, let Ne be the the number of false readings,

64190 VOLUME 6, 2018

N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

FIGURE 15. The responses of y3 and y4 for performance evaluations of requirement (A2) (via Experiments 1,2,3 and 5) and
drop-off detection (via Experiments 4 and 6).

state-feedback control strategy to regulate the detection angle

of the sensors and minimize the false alerts to the user. The
sensors’ positions were further optimized via an LQR-based
controller while the sensor signals sent to the users were
filtered out via dynamic feedback compensators to minimize
the false alerts. The effectiveness of the approach has been
verified via the designed experiments, and the numerical
results have shown that the proposed method can provide
significant improvements over the conventional methods in
terms of error reductions.
For future work, the techniques proposed can be adopted
and modified to suit other ETAs with different designs such as
the wearable assitive devices for the blinds. A survey among
FIGURE 16. Prototype smart cane.
the visually challenged on the convenience of the prototype
via several experiments may also be useful to enhance the
and τd be the delay between y3 and y3r in seconds. The usability of the device.
corresponding TEs read
Z ∞
TEy3 = (W1 Ne + W2 τd )dt, (22) REFERENCES
Z0 ∞ [1] Electronic Travel Aids: New Directions for Research, Committee Vis.
[2] D. Dakopoulos and N. G. Bourbakis, ‘‘Wearable obstacle avoidance elec-
TEy4 = W1 Ne dt, (23) tronic travel aids for blind: A survey,’’ IEEE Trans. Syst., Man, Cybern. C,
0 Appl. Rev., vol. 40, no. 1, pp. 25–35, Jan. 2010.
where W1 = 1 and W2 = 0.5 were the preferred weights. [3] W. Elmannai and K. Elleithy, ‘‘Sensor-based assistive devices for
From the recorded results in Table 5, it is clearly seen that visually-impaired people: Current status, challenges, and future direc-
tions,’’ Sensors, vol. 17, no. 3, p. 565, 2017.
all the TEs via applications of Result 2 are significantly [4] A. S. Al-Fahoum, H. B. Al-Hmoud, and A. A. Al-Fraihat, ‘‘A smart
smaller than those via the open-loop method. Thus, the infrared microcontroller-based blind guidance system,’’ Active Passive
design requirement in (A4) was satisfied. The prototype smart Electron. Compon., vol. 2013, Jun. 2013, Art. no. 726480.
[5] B. Mustapha, A. Zayegh, and R. K. Begg, ‘‘Ultrasonic and infrared sensors
cane that was tested throughout the experiments is shown performance in a wireless obstacle detection system,’’ in Proc. 1st Int.
in Figure 16 where the HC-SR04 ultrasonic sensors were Conf. Artif. Intell., Modelling Simulation, Dec. 2013, pp. 487–492.
used for Sh , Smr and Sml , the Sharp GP2Y0A21YK0F infrared [6] R. Pyun, Y. Kim, P. Wespe, R. Gassert, and S. Schneller, ‘‘Advanced
sensor for Sg and MMA7361 accelerometer for Sa . augmented white cane with obstacle height and distance feedback,’’
in Proc. IEEE 13th Int. Conf. Rehabil. Robot. (ICORR), Jun. 2013,
pp. 1–6.
IV. DISCUSSIONS AND CONCLUSIONS
[7] M. Dekan, D. František, B. Andrej, R. Jozef, R. Dávid, and M. Josip,
In this work, we have proposed a multi-sensor obsta- ‘‘Moving obstacles detection based on laser range finder measurements,’’
cle detection system for a smart cane via model-based Int. J. Adv. Robot. Syst., vol. 15, no. 1, pp. 1–18, 2018.

VOLUME 6, 2018 64191

 N. S. Ahmad et al.: Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback Control

[8] S. V. F. Barreto R. E. Sant’Anna, and M. A. F. Feitosa, ‘‘A method for image [28] S. Y. Kim and K. Cho, ‘‘Usability and design guidelines of smart canes
processing and distance measuring based on laser distance triangulation,’’ for users with visual impairments,’’ Int. J. Des., vol. 7, no. 1, pp. 99–110,
in Proc. IEEE 20th Int. Conf. Electron., Circuits, Syst. (ICECS), Dec. 2013, 2013.
pp. 695–698. [29] A. G. Abdel-Wahab and A. A. A. El-Masry, Mobile Information Com-
[9] R. Henderson and K. Schulmeister, Laser Safety. New York, NY, USA: munication Technologies Adoption in Developing Countries: Effects and
Taylor & Francis, 2004. Implications. Hershey, PA, USA: IGI Global, 2011.
[10] P. Chanana, R. Paul, M. Balakrishnan, and P. Rao, ‘‘Assistive technology [30] K. Ogata, Modern Control Engineering, 4th ed. Upper Saddle River, NJ,
solutions for aiding travel of pedestrians with visual impairment,’’ J. Reha- USA: Prentice-Hall, 2001.
bil. Assistive Technol. Eng., vol. 4, pp. 1–16, Aug. 2017.
[11] E. M. Ball, Electronic Travel Aids: An Assessment. London, U.K.:
Springer, 2008, pp. 289–321.
[12] J. Liu, J. Liu, L. Xu, and W. Jin, ‘‘Electronic travel aids for the blind
based on sensory substitution,’’ in Proc. 5th Int. Conf. Comput. Sci. NUR SYAZREEN AHMAD (M’14) was born in
Edu. (ICCSE), Aug. 2010, pp. 1328–1331. Kuala Lumpur, Malaysia. She received the B.Eng.
[13] S. Kumpakeaw, ‘‘Twin low-cost infrared range finders for detecting obsta- degree (Hons.) in electrical and electronic engi-
cles using in mobile platforms,’’ in Proc. IEEE Int. Conf. Robot. Biomimet- neering and the Ph.D. degree in control sys-
ics (ROBIO), Dec. 2012, pp. 1996–1999. tems from The University of Manchester, U.K.,
[14] G.-Y. Jeong and K.-H. Yu, ‘‘Multi-section sensing and vibrotactile percep- in 2009 and 2012, respectively. She became a
tion for walking guide of visually impaired person,’’ Sensors, vol. 16, no. 7, member of the IEEE Control Systems Society
p. 1070, 2016. in 2014.
[15] F. Prattico, C. Cera, and F. Petroni, ‘‘A new hybrid infrared-ultrasonic Since 2013, she has been with the School of
electronic travel aids for blind people,’’ Sens. Actuators A, Phys., vol. 201, Electrical and Electronic Engineering, Universiti
pp. 363–370, Oct. 2013. Sains Malaysia. Her research work centers around motion control, robust
[16] D. N. Hung, V. Minh-Thanh, N. Minh-Triet, Q. L. Huy, and V. T. Cuong, stability and performance analysis of constrained and nonlinear systems,
‘‘Design and implementation of smart cane for visually impaired people,’’
and optimization-based controller synthesis with linear matrix inequality
in Proc. 6th Int. Conf. Develop. Biomed. Eng. Vietnam (BME6), T. V. Van,
searches. Her current research interest includes phase-locked loops, embed-
T. A. N. Le, and T. N. Duc, Eds. Singapore: Springer, 2018, pp. 249–254.
[17] S. Chaurasia and K. V. N. Kavitha, ‘‘An electronic walking stick for ded control systems, and robust control analysis, and design of autonomous
blinds,’’ in Proc. Int. Conf. Inf. Commun. Embedded Syst. (ICICES), mobile systems.
Feb. 2014, pp. 1–5.
[18] R. K. Megalingam, A. Nambissan, A. Thambi, A. Gopinath, and
M. Nandakumar, ‘‘Sound and touch based smart cane: Better walking expe-
rience for visually challenged,’’ in Proc. IEEE Canada Int. Humanitarian NG LAI BOON was born in Butterworth,
Technol. Conf. (IHTC), Jun. 2014, pp. 1–4. Malaysia, in 1992. He received the B.Eng. degree
[19] D. Kim, K. Kim, and S. Lee, ‘‘Stereo camera based virtual cane system (Hons.) in electronic engineering from Universiti
with identifiable distance tactile feedback for the blind,’’ Sensors, vol. 14, Sains Malaysia in 2017. He is currently with Intel,
no. 6, pp. 10412–10431, 2014. Penang, and pursuing a part-time M.Sc. degree
[20] D. R. Chebat, S. Maidenbaum, and A. Amedi, ‘‘Navigation using sensory in electrical engineering (computer and micro-
substitution in real and virtual mazes,’’ PLoS ONE, vol. 10, no. 6, pp. 1–18, electronic systems) with Universiti Teknologi
Jun. 2015. Malaysia. He is currently with Intel, Penang,
[21] S. Shoval, J. Borenstein, and Y. Koren, ‘‘Mobile robot obstacle avoidance Malaysia.
in a computerized travel aid for the blind,’’ in Proc. IEEE Int. Conf. Robot.
Autom., vol. 3, May 1994, pp. 2023–2028.
[22] I. Ulrich and J. Borenstein, ‘‘The GuideCane-applying mobile robot
technologies to assist the visually impaired,’’ IEEE Trans. Syst., Man,
Cybern. A, Syst., Humans, vol. 31, no. 2, pp. 131–136, Mar. 2001.
[23] A. Rodrìguez, J. J. Yebes, P. F. Alcantarilla, L. M. Bergasa, PATRICK GOH received the B.S., M.S., and Ph.D.
J. Almazán, and A. Cela, ‘‘Assisting the visually impaired: Obstacle detec- degrees in electrical engineering from the Uni-
tion and warning system by acoustic feedback,’’ Sensors, vol. 12, no. 12, versity of Illinois at Urbana–Champaign, Cham-
pp. 17476–17496, 2012. paign, IL, USA in 2007, 2009, and 2012 respec-
[24] A. S. Martinez-Sala, F. Losilla, J. C. Sánchez-Aarnoutse, and tively.
J. Garcia-Haro, ‘‘Design, implementation and evaluation of an indoor Since 2012, he has been with the School of
navigation system for visually impaired people,’’ Sensors, vol. 15, no. 12, Electrical and Electronic Engineering, Universiti
pp. 32168–32187, 2015. Sains Malaysia, where he currently specializes in
[25] D. Nakamura, H. Takizawa, M. Aoyagi, N. Ezaki, and S. Mizuno,
the study of signal integrity for high-speed digital
‘‘Smartphone-based escalator recognition for the visually impaired,’’ Sen-
sors, vol. 17, no. 5, p. 1057, 2017.
designs. His research interest includes the devel-
[26] B.-S. Lin, C.-C. Lee, and P.-Y. Chiang, ‘‘Simple smartphone-based guiding opment of circuit simulation algorithms for computer-aided design tools.
system for visually impaired people,’’ Sensors, vol. 17, no. 6, p. 1371, 2017. He was a recipient of the Raj Mittra Award in 2012 and the Harold L. Olesen
[27] U. R. Roentgen, G. J. Gelderblom, M. Soede, and L. P. de Witte, ‘‘Inventory Award in 2010, and has served on the technical program committee and
of electronic mobility aids for persons with visual impairments: A literature international program committee in various IEEE and non-IEEE conferences
review,’’ J. Vis. Impairment Blindness, vol. 102, no. 11, pp. 702–724, around the world.
2008.

64192 VOLUME 6, 2018