Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

Visual Object Tracking Based on Dynamic Weights and Gaze Density

HUI-HUI CHEN1, CHIAO-WEN KAO2*, YUN-TING HUANG1,
BOR-JIUNN HWANG1, KUO-CHIN FAN2
1
Department of Computer Communication and Engineering, Ming Chuan University, Taoyuan, Taiwan
2
Department of Computer Science and Information Engineering, National Central University, Taoyuan,
Taiwan
[email protected]

ABSTRACT
This study proposes an adaptive algorithm for
tracking objects based on the dynamic weights
scheme and gaze density. The proposed adaptive
algorithm is to amend the deficiency on tracking
moving objects that are not driven by inertia. Using
the dynamic weights scheme and the information of
the gaze point density, possible areas where the
objects may appear are constantly researched to
target and trace the aimed objects. It is proven to be
effective on tracking non-inertia driven moving
objects.

KEYWORDS
gaze tracking, object tracking, adaptive, gaze
density, dynamic weights

1 INTRODUCTION
Object tracking is a popular research
subject and applied widely in the domain of
computer vision for applications, such as
automatic video surveillance, e-healthcare,
vehicle detection, human-computer interaction,
and railway cross obstructions. There are
numerous algorithms proposed for object
tracking. Whereas, there are some general
obstacles when the tracked objects are partially
obscured, the observable perspectives are
converted, the observable appearance is altered,
or there are multiple objects to be tracked.
Mostly as seen in prior research studies, the
ascertainment of the moving model of the
objects is highlighted in terms of machine
learning, fixed-rate moving model, appearance-

ISBN: 978-1-941968-16-1 2015 SDIWC

based moving model. The prior probability of

the motion is calculated based on the foregoing
movements of the object. In other words, most
tracking systems focus on the characteristics
and the motion characteristics of the object to
predict the consecutive moving model. Some
well-known and common-seen prediction
modules are as the background-subtraction
method, optical flow method, temporal-spatial
method, Mean shift, Camshift (Continuously
Adaptive Mean Shift Algorithm), and Particle
filter method [1].
Camshift first applies Mean shift and
infers the algorithm to the image sequence. It
utilizes the location of the prior image as the
outset to calculate the surrounding differences
of the current image. It gradually shifts toward
the direction of the decremented vales and
updates the size of the searching window. The
new scaled searching window will stop being
generated and the searching process will stop
when the decremented value is smaller than the
threshold. Camshift is remarkable due to its
performance on solving the transformed
appearance and occlusion of the tracked object.
It consumes less computing power and is apt
for tracking objects in the surveillance system
of monochromatic background [1]. However, in
merely consideration of color histogram
without taking account of the spatial
distribution of the target object, it is easily
leading to misjudgment on tracing and losing
the object when the background context is more
complicated and the color composition of target
object is similar to the background [2].

152

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

To those videos with changing scenes,

such as movies, advertisements, and
multimedia learning materials, their objects in
the content are mostly not driven by inertia but
by post-production editing. Tracking this kind
of objects requires overcoming the problem on
the spatial and locational inconsistency in the
continuous time. The effects caused by
changing scenes make the regularities of the
presentations on objects difficult to find, as
shown in Figure 1(a) with the inconsistent
objects. The effects caused by changing scenes
make the same objects moving too fast and not
driven by the inertia, as shown in Figure 1(b)
with the non-inertia driven moving objects.
Hence, the accuracy rate of predicting the
changing locations of the objects will decrease
if relying on the consideration of the object
movements only.
Moreover, the purpose of tracking objects
is to find out the interested objects, such as P.
Hidayatullah [3] using multiple color-tones of
objects to solve the misjudgment problem
caused by the color composition of the

interested objects similar to the background

colors. Z. Zhou [4] uses Particle Filter to find
out the closest moving path to solve the
problem of the obscured and the loss of the
tracked objects. Though above research studies
can accurately define or track the interested
objects, with the consideration of adding the
viewers visual fixations while watching or
viewing the movies and pictures, it may be
more profound to target the objects from the
interested objects to the level of the meaningful
objects. From the psychological explanation,
people focus more on what they are interested
in or on what they care about when watching
movies or viewing pictures, the subjectively
meaningful parts of the scene composition [5].
Therefore, this paper defines the
meaningful objects to be the part of the
interested objects. It adds in the viewers gaze
density to target the cognitive objects to the
viewers. In other words, it is to clarify the
objects that attract the viewers visual attention.
The research result can apply to the analysis of
advertisement effects, the display orientation

Frame 314

Frame 315

Frame 316

Frame 317

Frame 318

Frame 319

Frame 320

Frame 321
(a) The Inconsistent Objects

Frame 322

Frame 323

Frame 251

Frame 252

Frame 253

Frame 254

Frame 255

Frame 256

Frame 257
Frame 258
Frame 259
(b) Non-Inertia Driven Moving Objects

Frame 260

sourcehttps://www.youtube.com/watch?v=ikpdmeQ1Dq8

Figure 1. Changing scenes

ISBN: 978-1-941968-16-1 2015 SDIWC

153

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

of the TV shopping channel, and the implant

commercial effects in the movies. It may also
apply to the distance learning in the environment
of multiple teaching methods and self-learning in
the way that the teacher can analyze and
understand the learning and viewing processes to
find out the interested objects through the
recording of the students visual fixations while
viewing the instructional materials. Furthermore,
it can concentrate the original instructional
videos into the abstract videos with meaningful
objects for the particular student. Comparing the
difference between the original instructional
videos with the particular students abstract
videos, it may help the teacher understand the
learning condition of the particular student.
Moreover, it is hoped to track and locate the
objects through the viewers visual fixations as
one of the accordant information. Combing the
adaptive object tracking algorithm with the
viewers visual fixations can assist to relocate the
center and the range of the traced objects and can
increase the accuracy rate of tracking while the
tracked objects are obscured, the appearance of
the objects are transformed, or the objects are
disappeared during the changing scenes.
As stated in the research study of Osberger
[6], peoples visual are attracted easily by some
factors as divided into the high level factors and
the low level of factors. The low level factors
include color difference, size, shape, color,
movement, or intensity. The high level factors
generally include feedbacks from memory and
comparisons between models, such as the
relatively locations of objects in the scenes,
foregrounds and backgrounds, human-shape
objects, and controlled experimental instructions.
In order to precisely extract the meaningful
objects, it must respectively take these two levels
of factors into account. Many have been
considered on the extracting the characteristics of
objects and on the object tracking process, as
found in S.W. Chou [7], using the human-shape
feature to outline the target objects and then
make the found outline as one of the factors to
track objects. In order to increase the accuracy,
S.M. Singh [8] adapts multiple factors of colors

ISBN: 978-1-941968-16-1 2015 SDIWC

and textures and assigns weight values to

calculate the similarity between video images.
Furthermore, the motivation of watching or
viewing some particular objects is stressed for
the high level of factors. These points of views
provide the important tracking basis for this
paper on generalizing the meaningful objects and
on identifying the transformed objects.
For quantization on the high level factors,
the visual fixations can be the most direct and
reliable index. Various types of visual tracking
equipment are developed. To decrease the
disturbance to the viewer, recent research studies
focus on using webcams to acquire the
appearance of eyes and transform the spatial
coordinates [9]-[14] in replacement of the
wearable or intrusive devices. H.C. Lu [15] uses
the texture comparison to get gaze coordinates.
C.W. Kao [16] uses webcam with pattern voting
scheme to acquire gaze points and navigation
paths. Y. Sugano [12] finds that images on the
visual saliency area are easily to be noticed. W.T.
Li [17] departs the foreground and background
based on the visual and motion saliency to find
the interested objects.
This paper intends to use gaze density to
increase the effects on tracking objects, to
overcome the discontinuity problem on the
spatial locations in the continuous time frames,
and to identify the meaningful objects.
Consequently, this paper proposes the adaptive
tracking algorithm based on the visual behaviors
with the consideration of the high level of factors
on adjusting the dynamic distribution of weights
on colors, shapes, and textures. Moreover, while
there are conditions of the obscured and loss of
images occurred, the algorithm will calculate the
density of the viewers visual fixations
automatically. The detailed method and process
will be described in the following part.

154

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

2 PROPOSED METHOD
The proposed adaptive object tracking
algorithm is derived from Camshift in increasing
the tracking accuracy rate in terms of utilizing
the comparisons between the characteristics of
the video images as well as incorporating the
gaze density to adjust the prediction on the
central positions of the objects. The designated
flow of the adaptive object tracking algorithm is
shown as in Figure 2 and the corresponding
parameter specifications are listed in Table 1.
Parameters
n
WC
WS
WT
PC
PS
PT
FV
TF
Th
WA
N
(XAE,YAE)
dij
MGD
TGD
R

Table 1. Parameter specification

Description
nth - frame
Weight of color feature
Weight of shape feature
Weight of texture feature
Similarity of color feature
Similarity of shape feature
Similarity of texture feature
Similarity of features
Similarity threshold of tracked object
Threshold of adjusting weight
Dynamic weight
(A denotes color, shape or texture)
The number of frames for evaluating
Average of gaze point coordinate
The distance between gaze point i and j
Density of gaze points
Threshold of MGD
Magnification of searching window

The selected object initially is composed of

the RGB color space. Due to the easy
interference to the RGB color by light or other
factors, the color of the selected object is
transformed to HSV color space. For the
deduction on calculation, this paper adapts the
Hue color of the object image as the color
characteristic feature.
After transforming the RGB color space to
the HSV color space, the three characteristic
features are extracted simultaneously. They are:
1. Color histogram: the respective color
histogram is generated corresponding to the
selected and color-transformed object.

ISBN: 978-1-941968-16-1 2015 SDIWC

2. Shape: Canny edge detector is used to

find the shape of the object.
3. Texture: Local Binary Pattern (LBP) is
utilized to get the texture of the object.

Figure 2. Adaptive Object Tracking

The region of interest and the coordination

are acquired through the determination when the
similarity of features (FV) is greater than the
threshold value (TF). The total similarity value
(FV) is the sum of the similarity (PC, PS, PT),
where the PC, PS, and PT are features of color,
shape, and texture multiplied by weights (WC, WS,
WT) respectively. Then, the size and location of
the search box are recalculated and adjusted to
track the selected object according to the
comparison of characteristic features between the
back-to-back frames and the histograms on the
back projection.
In some situations, the proposed dynamic
weights as demonstrated in Figure 2 and Figure 3,
when the variation tolerance of FV is greater than
Th, the weights will change according to the
variation found on the PC, PS, and PT. If the
variation on PC of the color feature is greater
than the variations on PS and PT of the shape and
texture features, the weight of the color feature
will be increased and the weights of the shape
and texture features will be decreased. Vice versa,
the weights of the shape and texture features will

155

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

be increased when the variations are the most.

The adjustment on weights is to increase the
accuracy rate on tracking objects in different
conditions dynamically.

Figure 3. Dynamic Weights Scheme

Other than solving the problem on

misjudgment with adding the comparisons on the
characteristic features, this paper intends to have
some discussion on the issue of losing track
during the tracking process. If the object is
obscured or lost due to the changing scenes or
editing, it cannot return a quality result with mere
the traditional path prediction method. Hence,
this paper combines the visual tracking system to
rationally predict the possible appearing range of
the object through the users visual fixations, the
gaze density, as shown in Figure 2 and Figure 4.

During
the
tracking
process,
the
coordination of users fixations are estimated and
the distances between gaze points (dij, distance
between gaze point i and j) are calculated, as
shown in Equation (1). When FV is smaller than
TF (TF, similarity threshold of tracked object) a
loss problem will be identified. Calculating the
average coordination (XAE,YAE) of the visual
fixations in the close N frames, as shown in
Equation (2) and Equation (3), as well as
calculating the density of gaze points as in
Equation (4), when the MGD (MGD, density of
gaze points) is smaller than TGD (TGD, threshold
of MGD ), the average coordination (XAE,YAE) of
the visual fixations will serve as the central
position to predict the location of the object. The
edges (a and b) of the search box, as in Equation
(5), will be calculated to fulfill the propose on
adjusting and predicting the central position of
the tracked object.
d ij

X AE

X1 X 2 X 3 X N
N

(2)

Y1 Y2 Y3 YN
N

(3)

YAE

M GD

dij
N

X j Yi Y j
2

X j Yi Y j
2

(1)

(4)

a b R M GD

(5)

3 EXPERIMENTAL RESULTS
The accuracy rate of this proposed system is
shown as in Figure 5. The terms, true positive
and true negative, refer to the correct notation of
the detected object (the green rectangle in slash
pattern) and the ground truth object (the blue
rectangle). The terms, false positive and false
negative, refer to the incorrect indication of the
detected object and the ground truth object. The
Figure 4. Adjust Prediction Center

ISBN: 978-1-941968-16-1 2015 SDIWC

156

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

Precision ( Ppr ) is derived from Equation (6). The

Recall ( Pre ) is derived from Equation (7). The
Accuracy ( Pac ) of the system is calculated as in
Equation (8).
Ppr TP TP FP 100%

(6)

Pre TP TP FN 100%

(7)

Pac Ppr Pre 2

(8)

Figure 5. Quantitative evaluation

The performance of the proposed algorithm

is compared with the Camshift in two different
characteristics of experimental videos.
Case study 1: the object in the continuously
moving state
The object is moving back and forth and is
obscured by some other objects in similar colors.
A misjudgment will generate when the object is
tracked. There are 321 frames in the resolution of
1280x720 pixels.
Case study 2: the object in the animation
video
The non-inertia driven moving object
appears on the discontinuous spatial locations in
the continuous time. A misjudgment will
generate when the object is tracked. There are
158 frames in the resolution of 1280x720 pixels.
This paper uses the animation movie, Frozen, as
the experimental material.

ISBN: 978-1-941968-16-1 2015 SDIWC

Applying the general Camshift onto the case

study 1, the accuracy rate is 60.23%; comparing
to applying the proposed adaptive algorithm, the
accuracy rate is 90.28%. The comparisons are
shown as in Table 2.
Table 2. Frame detection accuracy of the proposed
algorithm and the Camshift method in case study 1
Accuracy
Failure*
Camshift
60.23%
39.77%
the Adaptive Tracking
90.28%
9.72%
Algorithm
* Failure =100%-Accuracy

Furthermore, other than discussing the

accuracy rate shown in Table 2, more intensive
accuracy rate can be considered via TN (TN, truenegative). There are two conditions caused by TN,
as shown in Figure 6(a) no appearance of the
object but still with the appearance of the search
box sketching the false object; Figure 6(b) the
object appears but the search box totally fails to
appear. As shown in Table 3, applying Camshift
returns a higher recall rate without TN due to
Camshift operates the color feature as the basis
of comparison. Camshift tends to be simpler in
calculation than the adaptive tracking algorithm
to be. Nevertheless, overall, the adaptive tracking
algorithm performs better than the general
Camshift.

(a) No appearance of the object

(b) Appearance of the object with false detection

Figure 6. TN (true-negative) conditions

157

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

Table 3. Frame detection accuracy of with and without TN

situation
Adaptive Tracking
Camshift
Algorithm(proposed)
Precision
44.61%
67.96%
With TN
Recall
59.28%
85.29%
Accuracy
51.95%
76.63%
Precision
74.06%
75.25%
Without
Recall
94.48%
98.42%
TN
Accuracy
86.24%
84.87%

More tracking results applying Camshift are

presented in Figure 7 for the case study 1
condition. Camshift majorly makes the judgment
based on the color features of the object. If there
appears another object with the similar color (as
in Figure 7, Frame 28), there will be a
misjudgment (as in Figure 7, Frame 31) and a
followed false tracking (as in Figure 7, Frame
112).
More tracking results applying the proposed
adaptive tracking algorithm are presented in
Figure 8 for the case study 1 condition. With the
add-in comparisons between the shape, texture,
and color features, objects are determined
whether they are the same one object based on
the three features. If the comparison result is
determined as the different objects, the search
box (in red) will stay put on the same spot (as in
Figure 8, Frame 31), and the weights of features
will adjust (as in Figure 8, Frame 112) and vary
(as shown in Table 4).

Table 4. The adjustment of feature weights in Case study1

Frame
FV
WC
WS
WT
12
0.93
0.5
0.25
0.25
63
0.76
0.45
0.2
0.35
131
0.96
0.55
0.15
0.3
207
0.81
0.5
0.1
0.4
285
0.97
0.6
0.05
0.35

The experimental results applying Camshift

for the case study 2 are presented in Figure 9 and
Figure 10. When the tracking object is in noninertia driven moving state, there will be a loss
problem (as in Figure 9, Frame 92) even though
the tracked object stays in the same screen
window. Camshift cannot identify the tracked
object and mark its current position (as in Figure
9, Frame 93). Camshift will continue the false
tracking (as in Figure 9, Frame 103).
When the object moves in non-inertia driven
and causes the loss problem (as in Figure 10,
Frame 92), the visual fixations (as in Figure 11)
will be included to calculate the density of gaze
points and coordinate (as in Table 5). The central
position of the predicted object and the size of
the search box will be readjusted. When the
object appears again, it can be tracked constantly
(as in Figure 10, Frame 103).
Table 5. Density of gaze points and coordinates
Frame
(XAE, YAE)
MGD
102
(750, 324)
69.11
103
(760, 318)
71.25
104
(768, 311)
73.36
105
(779, 306)
66.39
106
(790, 302)
66.45

Frame 21

Frame 28
Frame 31
Figure 7. Object Tracking with Camshift

Frame 112

Frame 21

Frame 28
Frame 31
Figure 8. Object Tracking Dynamic Weights Scheme

Frame 112

ISBN: 978-1-941968-16-1 2015 SDIWC

158

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

Frame 91

Frame 92
Frame 93
Figure 9. Object Tracking with Camshift

Frame 103

Frame 91

Frame 92
Frame 93
Figure 10. Object Tracking with Adaptive Algorithm

Frame 103

Frame 93

Frame 94

Frame 95

Frame 96

Frame 97

Frame 98

Frame 99

Frame 100

Frame 101

Frame 102

Frame 103 - The Cumulated Visual Fixations (N=10)

(sourcehttps://www.youtube.com/watch?v=ikpdmeQ1Dq8)

Figure 11. The Collected Visual Fixations

4 CONCLUSIONS
From the above experimental results, it is
learned that using the color, shape, and texture
features of the video images to enhance the
comparisons between the characteristic features
and to adjust the weights dynamically can return
better performance on tracking objects. In
addition, adding in the coordinate of users visual
fixations, the adaptive tracking algorithm can
solve the tracking problem on the loss. The
proposed adaptive tracking algorithm can
complement the insufficiency of Camshift for
Camshift using of color feature to make the
judgment on tracking the object. Especially when

ISBN: 978-1-941968-16-1 2015 SDIWC

there are objects with similar colors or

backgrounds, there will easily be the interference
on tracking and problem on misjudgment.
Moreover, there will be the loss problem when
objects are in the continuous time but in the
discontinuous spatial locations caused by the
editing of the video or changing scenes. This
paper uses the proposed adaptive tracking
algorithm to extract the characteristic features of
color, shape, and texture and combines the
density of the gaze points to adjust the central
position of the predicted object. It solves the
problems of misjudgment and loss on tracking
objects.

159

Proceedings of The Fourth International Conference on Informatics & Applications, Takamatsu, Japan, 2015

ACKNOWLEDGEMENTS
This paper is supported by Ministry of
Science and Technology (MOST), Taiwan, under
Contract of MOST 103-2221-E-130 -011 -.
REFERENCES
[1] H.T. Chen, T.L. Liu, and C.S. Fuh. Learning
Effective Image Metrics from Few Pairwise
Examples, 10th IEEE International Conference on
Computer Vision (ICCV), vol. 2, pp. 1371-1378, 2005.
[2] E. Emami, and M. Fathy. Object tracking using
improved CamShift algorithm combined with motion
segmentation, Machine Vision and Image Processing
(MVIP), 7th Iranian. IEEE, pp. 1-4, 2011..
[3] P. Hidayatullah and H. Konik, CAMSHIFT
Improvement on Multi-Hue and Multi-Object
Tracking, IEEE International Conference on
Electrical Engineering and Informatics (ICEEI), pp.16, 2011.
[4] Z. Zhou, X. Peng, D. Wu, Z. Zhu, C. Wu and J. Wu,
Object Tracking Based on Particle Filter with
Improved Camshift, Journal of Applied Sciences,
Asian Network for Scientific Information, vol. 14, no.
2, pp. 121-128, Oct. 21-23, 2014
[5] J. M. Henderson, and A. Hollingworth, High-Level
Scene Perception, Annual Review of Psychology, vol.
50, pp. 243-271, 1999.
[6] W. Osberger, and A. J. Maeder, Automatic
Identification of Perceptually Important Regions in an
Image, IEEE Proceedings, Fourteenth International
Conference on Pattern Recognition, vol. 1, pp. 701704, 1998.
[7] S.W. Chou, C.H. Hsieh, B.J. Hwang and H.W. Chen,
A Modified Mean Shift Algorithm for Visual Object
Tracking, Signal and Information Processing
Association Annual Summit and Conference
(APSIPA), pp. 1-5, 2013.
[8] S.M. Singh and K. Hemachandran, Content-Based
Image Retrieval using Color Moment and Gabor
Texture Feature, International Journal of Computer
Science Issues (IJCSI), vol. 9, no. 1, pp. 299-309,
2012.

ISBN: 978-1-941968-16-1 2015 SDIWC

[9] C.W. Kao, Y.W. Chen, C.W. Yang, K.C. Fan, B.J.
Hwang and C.P. Huang, Eye Gaze Tracking based on
Pattern Voting Scheme for Mobile Device, First
International
Conference
on
Instrumentation,
Measurement, Computer, Communication and Control
(IMCCC), pp. 337-340, 2011.
[10] C.W. Kao, Y.J. Huang, K.C. Fan, H.H. Chen, W.C.
Chung, B.J. Hwan and C.H. Hsieh, The integrated
gaze, web and object tracking techniques for the webbased e-learning platform, IEEE International
Conference on Teaching, Assessment and Learning for
Engineering (TALE), pp. 720-724, Bali Dynasty
Resort, Kuta, Indonesia, 2013.
[11] C.W. Kao, B.J. Hwang, C.W. Yang, K.C. Fan and C.P.
Huang, A Novel with Low Complexity Gaze Point
Estimation Algorithm, First International Conference
on
Instrumentation,
Measurement,
Computer,
Communication and Control (IMECS 2012), pp. 204208, March 14-16, Hong Kong, 2012.
[12] Y. Sugano, Y. Matsushita and Y. Sato, AppearanceBased Gaze Estimation Using Visual Saliency, IEEE
Transactions on Pattern Analysis and Machine
Intelligence, vol. 35, no. 2, pp. 329-341, 2013.
[13] K.H. Tan, D. Kriegman and N. Ahuja, Appearancebased eye gaze estimation, In Applications of
Computer Vision (WACV), Proceedings. Sixth IEEE
Workshop, pp. 191-195, 2002.
[14] D.W. Hansen and Q. Ji, In the eye of the beholder: A
survey of models for eyes and gaze, Pattern Analysis
and Machine Intelligence, IEEE Transactions on, vol.
32, no. 3, pp. 478-500, 2010.
[15] H.C. Lu, G.L. Fang, C. Wang and Y.W. Chen, A
novel method for gaze tracking by local pattern model
and support vector regressor, Signal Processing, pp.
1290-1299, 2010.
[16] C.W. Kao, B.J. Hwang, C.W. Yang, K.C. Fan and C.P.
Huang, A Novel with Low Complexity Gaze Point
Estimation Algorithm, Proceedings of the
International MultiConference of Engineers and
Computer Scientists, vol. 1, 2012.
[17] W.T. Li, H.S. Chang, K.C. Lien, H.T. Chang and Y.F.
Wang, Exploring Visual and Motion Saliency for
Automatic Video Object Extraction, IEEE
Transactions on Image Processing, vol. 22, no. 7, pp.
2600-2610, 2013.

160