2004 IEEE InternationalConference on Systems, Man and Cybernetics

Colour Based Human Motion Tracking for Home-based

Rehabilitation*
Yaqin Tao Huosheng Hu
Dept. of Computer Science, University of Essex Dept. of Computer Science, University of Essex
Wivenhoe Park, Colchester CO4 3SQ, U.K. Wivenhoe Park, Colchester CO4 3SQ, U.K.
Email: [email protected] Email: [email protected]

-
Abstract In this paper, we propose a colour based including depth ambiguities, occlusion and kinematics
tracking algorithm for capturing the motion of human body singnlarities, etc. In order to simplify the human motion
parts for home-based rehabilitation. Dzyerent colour belts tracking problem, most human motion tracking algorithms
are attached to the body joints of interest and tracked in the employ a shape model of the subject to support the tracking,
video sequence. The perjCormance of the colour-based others use multi-cameras to improve the robustness. The
tracking algorithm is analysed by comparing the tracking shape model of a subject varies from the simplest skeleton
results with the results >om a commercial marker-based model to 2D patches models [13], and to the sophisticated
tracking system, Qualisys. The experiment results show the 3D volumetric models [6] [SI [4] [lo].
eficiency of our method in tracking simple target reaching Most human motion tracking methods can be formulated
motion of human arms. into a two-step framework [9], namely feature extraction
and feature correspondence. The fmt step is to extract the
Keywords: Human motion tracking, target reaching,
image features from each image frame. The most widely
rehabilitation.
used image features are edge, optical flow, silhouette,
contour, and template. Others like feature points, colour [5],
1. Introduction blobs are also used in motion tracking. And then the
Traditionally patients who sustain a stroke take correspondence is built up by matching the 2D image
physiotherapy with the help of physiotherapists or well- features to the model data. The correspondence analysis is
often supported by prediction algorithms such as Kalman
trained carers to diagnose if they are doing the rehabilitation
properly. However, because of the staffing shortages at the Filter and Condensation [I 11 (Particle Filter). Hogg used
edge information to track a waking person [6]. The body
Nation Health Service and the need of a prolonged period
of time for the rehabilitation exercise, patients are not segments of the subject were represented by cylinders and
receiving enough treatment. We propose to develop a visual connected at joints. Bregler et al. used the twist and
tracking system to support the rehabilitation program for exponential map to track whole body motions withiin an
the patients at home environments so that the burden of optical flow framework [3]. Sidenbladh et al. used the
optical flow as the tracking primitive and a generative
hospitals and the physiotherapists could be relieved.
model of image appearance to track human motion in 3D by
The standard methods for clinical human motion analysis
are marker-based motion tracking method. There are a lot using a stochastic method [SI. hard et al. used contour to
track different moving objects such as the hand, the upper
of commercial marker-based human motion capture
body of a human and a leaf [ll].
systems that can be employed for tracking the patients’
motion, such as the Qualisys and CODA. However, apart The home-based rehabilitation requires a visual-tracking
from the difficulties of calibrating both cameras and system that should be cheap, accurate and nm in real time.
markers, these systems are too expensive for a daily Both the marker-based tracking method and the video-
deployment by stroke patients at home, and too complicated tracking (marker-free) method can only fulfil parts of these
for physiotherapists to interpret the tracking results of requirements. In other words, the marker-tracking system
patients’ motion. can provide the required tracking accuracy, but the system
Recently, a lot of attempts have been made to design a is too expensive for home-based rehabilitation applications.
marker-free tracking system for the human motion capture On the other hand, a video-tracking system is relatively
cheaper but its robustness and accuracy need improvement.
[7] [9] [14]. Marker-free tracking systems are very
attractive because only conventional cameras are needed Therefore, it becomes necessary to build a tracking system
instead of special cameras and intrusive markers. However, that has the advantages from both tracking systems. In this
designing a video system for tracking human motion is a paper, we propose to develop a visual tracking algorithm
non-trivial task. There are a number of difficulties [4], for home-based rehabilitation in order to achieve real-time
performance and sufficient accuracy.

*0-7803-8566-7/04/$20.00 0 2004 IEEE.

773
 The rest of the paper is organised as follows. Section 2 min(R, G ,B )
S=1-3
introduces the proposed colour-based human motion R+G+B
tracking method. The implementation of the tracking 1
system is described in Section 3. Our experimental results V = -(R + G + B )
3
on the accuracy test of the colour tracking method are
presented in Section 4. Finally, conclusions &d future work 2.3. ~~l~~~ T ~ ~ ~ H ~ ~
are presented in Section 5.
Colour tracking is essentially a two-steu tracking
procedure: (I) Se-&” target objects from each image
2. Colour Based Motion Tracking frame. (2)
. , Build UD the corresuondence of the same obiect
In this section, we introduce our current work on between consecutive image frames.
developing the colour-based motion tracking method. In each image fiame, a search window Wfor each colour
belt is used to save computational time. The colour models
2.1. Method Mare actually used as the decision rules io classify image
pixels to one of the objects or background. A point P in a
Video camem are employed in our tracking system to
capture the subject’s motion. In order to track the human
-
search window belones to the colour belt k if its colour
components f (PH,Ps,Pv,) meet the colour model Mk,
motion accurately, different colour belts are attached to
different body joints of the subject (see Figure I). The task
of human motion tracking is now simplified to track the (4)
colour belts in the input image sequence and the estimated
centres of these colour belts are regarded as the joint where S, is the set of all the pixels belonging to the same
positions of the corresponding human body parts. The idea object k .
of using colour belts attached to the joint positions is The mean image location of all the pixels is calculated
inspired by the marker-based tracking system, which uses and considered as the position of the corresponding body
the markers to highlight the joint where the colour belt is attached. And it is also used as
region of interest. We assume the centre of the search window in the next frame. If it fails
that the subject doesn’t wear the to find the colour belts in the original window, the search
clothes that have similar colour window is increased and the search method is repeated. If
with the colour belts, and the algorithm fails to fmd the colour belts in the whole
background is less cluttered. image (the colour belt may be occluded by other body
Similar work has been done by parts), the joint position in the previous frame is used as the
Zhuang et a1.[15]. The major joint position in this frame. This is the simplest strategy.
difference is that our system There are more sophisticated algorithms available such as
relies solely on colour features, Kalman Filter and Condensation algorithm. Since the
while Zhuang used both colour subject’s motion in the rehabilitation application is slow
and edge features. Figure 1 Colour belts and simply, this approximation is reasonable.
The correspondence problem is simplified in our tracking
2.2. Colour Model system because different colours are used to indicate
Colouring of interesting body joints partially simplifies different body joints. After using this tracking algorithm,
the visual tracking problem and allows fast processing, but we can obtain a 2D position trajectory of each colour belt.
colour tracking is sensitive to ambient lighting change. It is
important to choose a proper colour space, which can make 3. Implementation
the colour-based tracking robust to ambient lighting change.
There are lots of different colour spaces that have been used We have described the colour-based tracking algorithm
in colour-based tracking systems, such as the normalized in the previous section. It is important to know if the
RGB [r, g7, [H, S, and E Cr, Cb]. According to the proposed tracking algorithm has the acceptable accuracy or
comparison results in [2], HSV gives relatively better has the potential for the rehabilitation project. Here, we
tracking performance among all the colour spaces. So it is chose the target reaching movement to test the accuracy of
used in our tracking algorithm. For each objective colourk , the colour tracking system. Target reaching motion is
a region of interest is selected and used to create a colour mainly the motion of the arm [12], so markers and colour
model Mk before tracking. The following three equations belts are attached to the joints of shoulder, elbow, and wrist
are used to convert RGB to HSV. (see Figure 1). The original experiment involves only a
1 single video camera, so the target reaching motion is
-((R - G ) + ( R - B ) ) constrained in 2D and performed in the plane parallel to the
H = arccos 2 image plane.
(1)
,/((R-G)’ + ( R - B ) ( G - B ) )

774
3.1. Experimental Set-up 4. Accuracy Analysis
A marker based commercial system, Qnalisys, is A sequence of the target reachmg motion, which lasts
employed in our experiment for the comparison purpose. about 14.52 second, was captured in our experiments. The
The results from our colour-based tracking system are tracking result from the Qualisys is the 3D position
compared to the results from the Qualisys. In order to trajectory of each marker, represented as (x,,,(l),ym(l),zm(l)),
improve the accuracy of the comparisou and exclude the (xm(2),ym(Z),zm(Z)), ......(~~fi),y,,,fi).~~fi))~
where m is the
variation of the motion performed by the subject at different data from marker tracking system, and k is the frame index
time, we capture the subject’s motion by using a Qualisys in the image sequence. Since we constrain the motion in
and a colour tracking system simnltaneously. Figure 2 2D, the depth coordinates remain almost constant for each
shows the experimental set up viewed from the top side. marker during the image sequence and can be ruled out.
The subject wean both the colour belts and the markers. Accordingly, the result from the colour tracking system
The video camera and Qualisys cameras 1, 2 capture the is the 2D position trajectory of each colour belt, denoted as
subject’s motion at the same time. The origin of the (xC(l)~yd1)), (x&)~Y$)), ......( X ~ ~ L Y ~ ~where
))~ c
Qualisys coordinate system is also shown in Figure 2. represents the data from colour tracking system and j is the
frame index

f colour belts

a I
I
I
-- - l i t angle
calculation
video . I I
camera

median filter I
--_-_
Figure 2 Top view of the experimental set up

3.2. The Qualisys Tracking System 2 1

The Qualisys system uses small retro-reflective hall
markers, which are attached to the performer’s joints (see
Figure 3 (a)) and can reflect infrared light. It’s easy to
detect and locate these markers in the 2D images filmed by
different synchronized cameras because they are bright
points in the image planes (see Figure 3 (b) (c) (d)). Three 3
oidata
digital cameras fixed in the scene capture the arm of the
nerformer here. Figure 5 the procedures of data processing
Dircct comparison bencen the marker dah and the video
data seems unreasonable because the two tracking systems
have different coordinate system and units. A scries of data
processing procedures (see Figure 5 ) are employed to
proccss the tracking ruwlts from each s)stein, and finally
(a) (b) (C) (4 use them to conipare the tracking performance. These dat?
Figure 3 (a) Markers attached to the joints of the performer.
processing methods are described in detail in the following
(b)(c)(d) Marker points captured from three cameras.
subsections I, 2, and 3.
3D position of each marker (see Figure 4) is calculated
from the corresponding 2D 4.1. Angle Calculation and Data Smoothing
marker point in each image by
using the stereo correspondence In ow experiment, we compare the joint angle
method. At least two cameras trajectoncs instead of the joint position trajectories because
should be used in order to obtain of its invariant to the coordinate systems and units. Based
3D data. Each marker is tracked on the 2D position trajectory of each joint, we can calculate
from one frame to the next frame the shoulder angle and the elbow angle trajectories for the
and its 3D position trajectory of target reaching motion. We define the shoulder anglc 0 in
each marker can be obtained at thc 2D motion situation is the anglc between the upper arm
this step. Fig. 4 Markers in 3D and the horizontal axis. And the elbow angle Q, is thc angle

7‘75
between the upper arm and the forearm (see Figure 6). After is different. That's because the frame rates of cameras are
calculation in each h e , we can get the joint angle different, (23% in the colour tracking system, and 12Of/s in
trajectories of shoulder and elbow. They are represented as the marker tracking system). In this experiment, the number
follows for the marker tracking system: of image frames from colour tracking system is 363, and
1742 from marker tracking system. In order to compare the
(@,(1),0,(2), ......@,(k)) , (D,,(I),Dm(2),......~,,(k)) tracking results obtained from the two tracking systems, the
The procedures are the same for calculating the joint angles two sets of data are fitted with a cubic spline [l]
in colour based tracking system: respectively.
Cubic s p l i e interpolation is a piecewise interpolation
(0, (U, @, (21,......@, (i)), (Dc(11, (21,......'3, (i)) and is very useful to interpolate between known data points
(also known as breakpoints) due to its stable and smooth
characteristics. It fits a different cubic polynomial between
each pair of breakpoints. Considering tb: shoulder angle
data, we get a collection of shoulder angle data ( t o , O 0 ) ,
( t l , O l ),...( t i - l , @ i - l ) , ( t i , O i ),_..( f n , @,),wheretistbe

Iy Elbou

Figure 6 Shoulder angle and elhow angle in the image

coordinate system
time variable of the captured motion sequence, n is the
index of the number of h e s of data and 0 ;represents the
shoulder angle at time f,. A third degree polynomial (see
Equation (5)) is constructed between each pair of points
( ,l.,@ ) and ( r i ,Oi ). If we can calculate the 4 unknown
Since the processing are the same for both shoulder and coefficients (a, b. c, d,) for each third order polynomial,
elbow angle data, we will illustrate the procedures by taking we then found a cubic spline for our shoulder angle data set.
the shoulder angle data processing for example in the According to the properties of a cubic spline that it is
following sections. Figure 7(a) shows the shoulder angle continuous, and its first and second derivatives are
data obtained from each tracking system. They are continuous as well at each breakpoint, equations (6),(7) and
calculated according to the method discussed above. In (8) can be formulated for each polynomial.
order to smooth the noise, these joint angle data are filtered A third order polynomial, where 5 f 2 ti , i E [l,n]
by a median filter with size of 5 (see Figure 7(b))
f i ( t ) = ai + bit + c i t Z + dit3 (5)
The cubic spline is continuous at each breakpoint.
LUi) = L+I@i)= @ i (6)
The first and second derivatives are continuous at each
breakpoint.
h.'(ti)
=fi+IVi) (7)
L."(t;)
= L.+l"(fi) (8)
According to the analysis above, we can get 4 equations
for each cubic polynomial to solve 4 unknown coefficients.
(a) Shoulder angle data (marker & video) However, the first and last cubic polynomials don't have
adjoining cubic polynomials beyond the fnst and last
breakpoints, which means Equation (8) is not suitable for
the end points. There are a lot of different existing methods
to solve the problem. Here we use parabolic run-out splines,
which means that the second order derivatives of the spline
at the end points are the same as that at the adjacent points.
fi"(t0) = fi"(t*) (9)
f""(t.) = f""L1) (10)
So the data fitting with a cubic spline becomes solving
(b) Shoulder angle data after smoothing 4*n linear equations to get the 4*n unknown coefficients.
After solving a large set of linear equations, the cubic spline
Figure 7 Shoulder angle data of target reaching motion curve of the shoulder angle data can be found. Figure 8 (a)
shows the shoulder angle data from each system fitted with
4.2. Data Fitting cubic spline curves.
As we can see from Figure 7, the number of frames of We equally sample N data points from the corresponding
data for the same motion sequence captured by each system s p l i e curve. And the extracted data sel. in each system,
which has the same number (700 data points in this

776
experiment), is used to compare the tracking results. Figure sequence of about 14.5s, so the frequency of the motion is
8 b ) illustrates the sanided shoulder anele data after data
\ ~ ,
about 2.90s/cvcle.
fining.

Figure 9 Elbow angle after translation

(a) Shoulder angles fitted with cubic splines
4.3. Comparison of the two sets of data
Two indices are used to evaluate the performance of two
systems:
*The absolute difference
d, =Ixi - y i l i6(1,2, ....A’) (1 1)
*The correlation coefficient r

where x, y represent the sampled video data and sampled

(b) Re-sampled shoulder angle data after fitting marker data respectively. N is the number of points in each
set.
The absolute differences of the shoulder angle and elbow
angle for the two systems are illustrated in Figure 10. Most
of the differences for the shoulder angle are less than 2.5
degrees and less than 4 degrees for the elbow angle. The
correlation coefficient is also employed to evaluate the
similarity of the tracking results. According to Equation
(12), the correlation coefficient for the shoulder angle is
0.9998, and 0.9999 for elbow angle. The correlation ratio
p0.99) by using our colour based tracking algorithm is
much higher than the similar ratio (M.95) in [l], which is
(c) Shoulder angle after translation the result based on the marker data and the medial axis
transformation.
Figure 8 Shoulder angles of target-reaching motion The target reaching motions with various speeds are also
After the data processing procedures, these sampled data tested and the experimental results are shown in the Tahlel.
should be ready for comparison in theory. But as we can see From table 1, we can see the colour based tracking
from Figure 8@), there is a translation transformation algorithm does give promising tracking results. All the
between the marker data and the video data. This is because correlation coefficients are higher than 0.99. The absolute
of the layout difference of the markers and the colour belts difference increases as the speed of the target reaching
on the a n n (see Figure 1) and the synchronization problem motion increases. However, even for the highest speed in
between the two tracking systems. The markers and colour our experiments 1.55s/cycle, which is faster than the daily
belts are attached slightly away from each other for the life motion, the absolute difference for the shoulder angle is
same body joint in order to be visible to the Qualisys and about 5 degrees and 8 degrees for the elbow angle. For the
video cameras respectively. We can compensate the normal speed motion, 2.25slcycle and 2.90s/cycle, the
translation caused by the experiment set up by translating absolute difference is lower than 5 degrees. Since the
the video data towards the marker data (see Figure 8 (c)). rehabilitation motion of the patients is slower, or at most
Figure 9 illustrates the elbow angle trajectories of this equivalent to the speed of the daily life, which means that
motion sequence after being processed by the same the absolute difference is less than 5 degrees, the colour-
procedure as the shoulder angle trajectories. Note that there based tracking algorithm is suitable for the rehabilitation
are total 5 cycles of the target-reaching motion in this project under the constraints assumed in this paper.

777
 Acknowledgments
The authors would like to thank Ms. H m ,H a f d H, Dr
Martin Sellens and Prof. Ralph Beneke in the Biological
Sciences Department to allow us to us12 their Qualisys
system. Our thanks also go to Dr Nigel. Harris at Bath
University and the other members of I3PSRC EQUAL
Smart Rehabilitation Consoltium for useful discussion.

References
[I] A. Bharatkumar, K. Daigle, Q. Cai, J. K. Aggarwal, “Lower
Limb Kinematics of Human Walking with the Medial Axis
Transformation”.Proc. of Workshop on Motion of Non-Rigid
and Articulated Objects, Austin, Texas, USA, 1994
[2] B. D. Zarit, B. J. Super, and F. K. H. Quek, “Comparison of
Five Colour Models in Skin Pixel Classification”.
Proceedings of ICCV’99 Int’l Workshop on recognition,
analysis and tracking of faces and gestures in Real-Time
systems, 58-63, 1999.
[3] C. Bregler, and J. Malik, “Tracking People with Twists and
Exponential Maps”. Proc. of IEEE Int. Conf. on Computer
Vision and Pattem Recognition, 1998.
@) [4] C. Sminchisescu, “Estimation Algorithms for Ambiguous
Figure I O Absolute differences of shoulder & elbow angle Visual Models 3D Human Modeling & Motion
Reconstruction in Monocular Video Sequences’’: PhD Thesis,
Table 1 Experimental results of the target-reaching motion Inst. National Politechnique de Grenohle (INRIA), July 2002.
[SI C. Wren, A. Azarhayejani, T. Darrel, and A. Pentland,
I Speed I Absolute I Correlation I “Pfmder: Real-time tracking of the human body”.
I I Difference
Shoulder I Elbow
Angle Angle
I Coefficient
Shoulder I Elbow
Angle Angle
I Proceedings of SPIE, Bellingham, WA, 1995.
[6] D. Hogg, “Model-based vision: a progmm to see a walking
person”. J. of Image and Vision Computing, l(1): 5-20,1983,
1.55s/c <=5” <=8” 0.9994 0.9998 [7] D. M. Gavrila, “‘The Visual Analysis of Human Movement:
2.25dc <=4‘ <=So 0.9997 0.9999 A Survey”. Joumal of Computer Vision and Image
2.90s/c <=2Y <=4” 0.9998 0.9999 Understanding,Vo1.73, No1 ,pages 82-98, 1999.
[SI H. Sideuhladb, M. Black, and D. Fleet, “Stochastic Tracking
of 3D Human Figures Using 2D Image Motion”. European
5. Conclusions and Future Work Conf. on Computer Vision, 2000.
Traditionally, patients’ rehabilitation needs the help of [9] J. K. Aggarwal, and Q. Cai, “Human Motion Analysis: A
physiotherapists or expert carers. We proposed to develop a Review”. Journal of Computer Vi,sion and Image
Understanding. 1999.
visual-tracking system for the patients who sustain a stroke [IO] I. M. Rehg, and T. h a d e , “Model-Based Tracking of Self-
to do their rehabilitation exercise at a home environment Occluding Articulated Objects”. Proceedings of Intemational
without the need of physiotherapists’ presence. In this Conference on Computer Vision, Cambridge, MA, .~ pages
paper, the motion tracking module is developed by 612-617, June 20.23, i995.
employing a colour-based human motion tracking algorithm .1111_M. hard and A. Blake. “CONDENSATION -- conditional
which is simple and fast. Although only the constrained density propagation for visual tracking”. Int. Joumal of
tracking motion of a human arm was currently investigated, ComputerVision, 29, I, pages 5 - 2 8 , 1998.
it is a good starting point because tracking arm motion is [I21 N.Yang, M. Zhang, C. Huang, and D. Jin, “Motion Quality
very useful for the rehabilitation process. Evaluation of Upper Limb Target-Reaching Movements”.
Medical Engineering &Physics 24, pages 115-120,2002.
To improve the performance of the colour tracking [13] S. X. Ju, M. Black, and Y. Yacooh, “Cardhoard people: A
algorithm and make it more robust and reliable, the colour parameterised model of articulated motion”. 2nd Int. C o d . on
model problem and the occlusion problem will he Automatic Face- and Gesture-Recognition, Killington,
investigated in our future work. An adaptive colour model Vermont, pages 38-44, Ocl 1996.
is to be employed to deal with the lighting condition change [14] T. Moeslund, and E. Granum, “A Survey of Computer
in the environment. The occlusion problem is one of the Vision-Based Human Motion Capture”,ComputerVision and
most difficult problems in human motion tracking, which is Imageunderstanding (81), No 3, pages 231-268,2001.
to he dealt with by using multi-cameras and the b a n [IS] Y. Zhuang, Q. Zhu, and Y. Pan, “Hiemahical Model Based
Filter. Some kinematic constraints and motion constraints Human Motion Tracking”. ICIP 2000
of the human body can also he used to solve the ambiguities
occurred in the motion tracking.

778