2012 Second International Conference on Instrumentation & Measurement, Computer, Communication and Control

Detection and Segmentation of Occluded Vehicles

Based on Skeleton Features

Chunyu Chen Shihui Liu

Dept. Information and Communication Engineering Dept. Information and Communication Engineering
Harbin Engineering University Harbin Engineering University
Harbin, China Harbin, China
[email protected]

Abstract—Vehicle occlusion is an important problem for vehicle

detection even in the high-definition camera. The difficulty is to II. PREVIOUS WORK
obtain individual vehicle from one blob when some vehicles In order to solve the occlusion problem, notable early
merged together in the blob in the foreground mask. We present works have been done in the area of vehicle segmentation.
a method that can detect the number of occluded vehicles in one Beymer et al’s 1997 paper [2] describes pioneering studies that
blob and the segmentation points based on the effective skeleton
tracks feature points throughout the video sequence and then
nodes, and then segment occluded vehicles in one blob associated
cluster the features based on their motion. Goo Jun et al’s study
with concave spots. Experiments are performed on traffic video
sequences captured at two different locations, and the results in 2005 [3] is also based on a similar idea and further extends
appear to have good accuracy. Beymer et al’s approach. Their approach has difficulty
initializing and tracking partially occluded vehicles.
occluded vehicles; skeleton nodes; concave spots; Another study on vehicle segmentation is by Song and
segmentation; Nevatia [4]. In which a model-based approach was employed
using 3-D box models, and a probabilistic hypothesis
I. INTRODUCTION framework was proposed for the box models. One of the major
Recently, computer vision-based traffic monitoring systems drawbacks to model-based tracking is the large number of
have become popular because of their ease of installation, models needed due to differing vehicle shapes and camera
inexpensive price and ability to capture a rich description of the poses.
scene [1]. When the video is captured from those systems, With known vehicle orientation, Pang et al. [5] analyze the
moving regions can be detected from background. Due to the shape of the foreground blobs to detect occlusion and segment
height and angle of those systems, there are many vehicles merged vehicles. However, this method has difficulty in
occluding each other. And a motion blob is likely to contain segmenting vehicles with arbitrary orientations. The
multiple vehicles in it. Therefore, it is necessary to segment the segmentation algorithm presented in this paper also based on
occluded vehicles into individual ones to count and identify the foreground blobs, but our approach is purely based on
each vehicle in the image. skeleton feature extracted from it, without requiring known
In this paper, we present a novel method for segmenting vehicle orientation.
vehicles in a highway traffic monitoring video, which
occlusion is a serious issue. The approach is based on skeleton III. ALOGORITHM DESCRIPTION
features that are extracted from moving regions. The major An overview of proposed algorithm for occluded vehicle
difference with other related methods is that: segmentation is shown in Figure 1. For a given image, a
 The overall approach is able to segment vehicles even background subtraction algorithm associated with edge
when vehicles remain partially occluded throughout detection algorithm is applied for preprocessing, and then the
their appearance in the videoˊ foreground mask is obtained. Skeleton structure and some
effective skeleton nodes are extracted from the blobs in
 The method is able to segment vehicles using a single foreground mask. We can detect blobs that are suspected of
video frameˊ having more than one vehicle in them, and then find out the
segmentation points in the blobs. Finally, suspected blobs are
 We requires no prior models and does not need any segmented into single vehicle based on the boundary of the
trainingˊ blob associated with segmentation points. The details of these
 The algorithm can work robustly with arbitrary steps are described in the following subsections.
orientations and is not complexˊ
A. Preprocessing

This Research Project was fully sponsored by {The Fundamental Research Funds for the Central Universities} with grant number { HEUCFZ1102 }.

978-0-7695-4935-4/12 $26.00 © 2012 IEEE 1037

1055
DOI 10.1109/IMCCC.2012.249

Authorized licensed use limited to: Central Scientific Instruments Organisation. Downloaded on November 25,2020 at 08:59:04 UTC from IEEE Xplore. Restrictions apply.
 Image frame monitoring systems remains to be solved in the following
method.

Preprocessing

Skeleton nodes extraction

Detecting occluded vehicles Segmentation points Figure 2. (a)Original image. (b)foreground mask from background
subtraction. (c)foreground mask from enhanced preprocessing algorithm
proposed in this paperˊ

The blob boundary Concave points

B. Skeleton nodes extraction
As a result of the preprocessing which is proposed in this
paper above, we obtain a foreground mask without shadows.
Segmentation results While some blobs in foreground mask contain single vehicle,
some blobs contain multiple vehicles. In this section we extract
skeleton structure from blobs and find some effective skeleton
Figure 1. Overview of the propose system
nodes in order to detect and segment occluded vehicles using
the difference of skeleton nodes.
An enhanced preprocessing algorithm is proposed in this Shape analysis methods play an important role in systems
paper. Fist we use background subtraction to obtain the for object recognition and analysis [7]. A most representative
foreground mask. For background subtraction, an enhanced shape analysis method is skeleton extraction. Several theories
version of the adaptive mixture of Gaussian background model of skeleton extraction are briefly mentioned. The skeleton
is employed. In the adaptive mixture of Gaussian model, each extraction algorithm adopted in this paper is proposed by Pflata
pixel is assumed to have a mixture of Gaussian distribution. and Rosenfeld as described in [8]. The idea of the algorithm is
When the distribution is a single hump, the distributions with to represent a region by the locus of centers of maximal
highest weights are chosen as distributions of a background inscribed circle, the name “skeleton’’ has been suggested for it.
pixel. When the distribution exist more adjacent peaks, take the
While the skeleton structure has been extracted from blobs,
average distributions with highest weight as distributions of a
then we find the effective skeleton nodes. Since the skeleton
background. With our sequences, we found 10s of video to be
points is a single-pixel wide connected region, we can thus
sufficient for this task. It is applicable for which the traffic is
define the effective skeleton node is the one of the skeleton
moderately dense for some period of time. Once the
points and have three contiguous skeleton points in its eight
background is learned, the technique of background subtraction,
principal directions, including horizontal, vertical and diagonal.
including morphological operations and threshold values, is
Figure 3 shows the skeletons and the effective skeleton nodes
applied to each image of the sequence to yield a binary
of a regular region.
foreground mask that indicates whether each pixel is
foreground or back ground. One of the serious problems in
using background subtraction is the distraction caused by
moving shadows, which mistakenly appear as foreground
pixels. It is common for shadows to cause multiple nearby
vehicles to merge into a single blob.
Then we use edge detection to reduce the effects of
shadows from background subtraction. Kirsch [6] proposed a
direction edge detection operator on edge detection, which
algorithm has certain anti-noise ability. And the algorithm is Figure 3. (a)a regular region ˊ (b)the skeleton points extracted from the
employed in this paper on edge detection of image frames. An region.(c)the effective skeleton nodesˊ
effective combination of background subtraction and edge
detection is to test the outline of the moving vehicles and The above method is used to extract the skeleton and
reduce shadows. The prospect of foreground mask is obtained effective skeleton nodes that from each blob in foreground
through the technique of holds filling and morphological to the mask. The results are shown in Figure 4 in different
edge of moving vehicles. The results are shown in Figure 2 in conditions, showing the variety of scenarios considered.
comparison with the previous background subtraction
algorithm. Although the problem of shadow occlusion has been C. Detecting occluded vehicles
solved, the problem of occlusions due to angle of traffic

1056
1038

Authorized licensed use limited to: Central Scientific Instruments Organisation. Downloaded on November 25,2020 at 08:59:04 UTC from IEEE Xplore. Restrictions apply.
 D. Segment vehicles in one blob
This section describes the segmentation of vehicles in one
blob. The main idea presented in this section is finding
segmentation points based on each vehicle’s effective skeleton
nodes in the blob, and then finding the segmentation lines
based on the concave spot.
Now we obtain the segmentation points of the blob. From
the result of detection, we have known that there are N
th
vehicles in the blob and the j vehicle’s effective skeleton
k
nodes have obtained in the V j . Let’s assume that the

 j 1
th
segmentation point between the j th vehicle and the
Figure 4. The skeleton features of various blobs. (a)single vehicle in the vehicle is a0 j , then we have  N 1 segmentation points.
blobˊ(b)two occluded vehicles in the blobˊ(c)three vehicles occluded in the
blobˊ Let’s denote the coordinates of segmentation point a0 j as:

The objective of this section is to detect blobs composed of a0 j ( x, y )  (V j

max( k )
( x, y )  V j 1
min( k )
( x, y )) / 2 (2)
multiple vehicles. The detection algorithm is based on the
effective skeleton nodes extracted in the previous section. max( k )
Vj is the maximum coordinates of effective skeleton
Let’s assume that Mi denotes the effective skeleton nodes th min( k )
in one blob, 1  i  L . V j
k
denotes the j
th
vehicle’s k
th nodes in the j vehicle, and V j 1 is the minimal

 j 1
th
effective skeleton nodes in the blob. Assuming that coordinates of effective skeleton nodes in the vehicle.
Ms , ..., M s be similar a line, where
j 1 j Figure 6 shows two examples of segmentation points. The
s  [ s0 , s1 ,..., sN ] , s0  1 and s N  L . Then we have known left image of Figure 6 shows the segmentation point with two
that and the blob has N vehicles. Each vehicle’s effective vehicles occluded in the blob, and the right image of Figure 6
skeleton nodes are shows two segmentation points in the blob which contains
three occluded vehicles.
k
Vj  Mi ( s j 1  i  s j ,1  k  s j  s j 1  1,1  j  N ) (1)

 If N  1 , assign the blob has single vehicle.

 If N  2 , assign the blob has N occluded vehicles.
Under normal circumstances, the number of effective
skeleton nodes is proportional to the number of vehicles
included in the blob.
Figure 6. Detecting occluded vehicles
 If L  2 , the blob has single vehicle.
Then we extract the segmentation lines by finding the
 If L  4 , the blob has two occluded vehicles.
concave spot near by the segmentation points. Because the
 If L  6 , the blob has three occluded vehicles. concave spot is defined as the boundary point of the blob with
the minimum distance from the segmentation points.therefore,
That is N  L / 2 . we denote the concave spot upon the segmentation point as a1 ,
Figure 5 shows the result of detecting blobs with occlusions. and the concave spot under the segmentation point as a2 . Then
The blobs that we suspect to have single vehicle have contours
in red, and other blobs have green contours and the suspected the segmentation line is the connection between the two points,
number of vehicles are shown beside the contours. that is a1 and a2 .

We assume that Bu  {( xui , yui ) | i  1,..., Q} denotes all

the boundary points upon the the segmentation point a0 , where
xu  [ xu1 , xu2 ,..., xuQ ] and yu  [ yu1 , yu2 ,..., yuQ ] . And
Bd  {( xd j , yd j ) | j  1,..., P} denotes all the boundary points
under the the segmentation point a0 , where
Figure 5. Detecting occluded vehicles

1057
1039

Authorized licensed use limited to: Central Scientific Instruments Organisation. Downloaded on November 25,2020 at 08:59:04 UTC from IEEE Xplore. Restrictions apply.
 xd  [ xd1 , xd 2 ,..., xd P ] and yd  [ yd1 , yd 2 ,..., yd P ] . Let these two traffic videos differ by camera placement, field of
veiw and agle changes.we also can see that some vehicles in
Du  {Du1 ,..., DuQ } and Dd  {Dd1 ,..., Dd P } represent the adjacent lanes merge with each other due to shadow or angle,
distance from segmentation point a0 to the upper boundary and thus the system segment the occluded vehicles into single
one which is labeled by a green contours. Note that we
Bu and the lower boundary Bd , repectively. Therefore, Du successfully find and segmente merged vehicles in the
and Dd can be calculated by following frames, and the results appear to have good accuracy.
2 2
Dui  ( xui  xa )  ( yui  ya ) (3)
0 0

2 2
Dd j  ( xd j  xa )  ( yd j  ya ) (4)
0 0

Where [ xa , ya ] indicate the coordinate value of the

0 0

segmentation point a0 , which has been obtained above.  

Assuming that the minimum distance in Du and Dd are Frame #807 Frame #1083
th th
found, let’s denote the k distance in Du and the s
distance in Dd as the minimum diatance. Finally the concave
spots a1 and a2 are obtained by

a1  ( xuk , yuk ) (5)

a2  ( xd s , yd s ) (6)
Frame #1103 Frame #1126
Connecting a1 and a2 we can get the segmentation line.
Figure 8. Segmentation results from traffic1.avi
Figure 7 shows the result of segmentation of vehicles in one
blob. The right image of figure 7 has two vehicles in the blob
and the left image has three vehicles. The segmentation lines
can segment the occluded vehicles accurately and the result is
satisfactory.


Frame #124 Frame #175

Figure 7. The result of segmentation

IV. EXPERIMENTAL RESULTS

The experiments are performed using MATLAB’s imge
processing toolbox and the algorithm is tested on traffic video
sequences captured at two different locations in the daytime. Frame #232 Frame #345
The video is digitized at 336 by 448 resolution and 25 frames
per second. One of the video is taken from HD camera, and the Figure 9. Segegmentation results from traafic2.avi
other is taken from the ordinary camera. the total processing
time is about 2 seconds per frame on a PC(CoreTM2 at We evaluated both the detection performance and the
2,53GHz). segmentation performance of the proposed method, and the
We will show the result of dividing occluded vehicles into results are shown in Table1. Correct detection is defined as
individual ones with the algorithm in these two traffic videos. detecting a blob has single vehicle or occluded vehicles
To convey the varity of conditions in the processed videos, correctly. The detection rate is good for the two traffic
Some sample image frames are shown in Figure 8 from situations. Correct segmentations only includes correctly
traffic1.avi and Figure 9 from traffic2.avi. As can be seen, segmented vehiles from correct merged detections. As we can
seen from table1, the number of false segmentations is small.

1058
1040

Authorized licensed use limited to: Central Scientific Instruments Organisation. Downloaded on November 25,2020 at 08:59:04 UTC from IEEE Xplore. Restrictions apply.
 The total accuracy means the final rate of detecting each IEEE Trans. Intelligent Transportation Systems, vol.5, no.3, pp.129-141,
vehicle after segmentation which is quite high(95.7% for Sept. 2004.
traffic1.avi and 88.0% for traffic2.avi). [6] R. A. Kirsch, “Computer determination of the constituent structure of
biological images,” J. Computers in Biomedical Research, vol. 4, no. 1,
1971.
TABLE I. EVALUATION ON DETECTION AND SEGMENTATION [7] S. Loncaric, “A Survey of shape analysis techniques,” J. Pattern
Recognition, vol. 31, no. 8, pp. 983-1001, Aug. 1998.
Traffic1 Traffic2 [8] J. L. Pfaltz and A. Rosenfeld, “Computer representation of planar
Frames 1886 378 regions by their skeletons,” J. Communications of ACM, vlo. 10, no. 2,
pp. 119-122, Feb. 1967.
Vehicles 3702 1532

Correct Detections 3572 1380

(merged/single) (360/3212) (308/1072)

Detection Rate 96.5% 90.0%

Correct Segmentations 343 286
False Segmentations 21 32
Total accuracy 95.7% 88.0%

The error detection occurred is mostly due to the following

reason: the skeleton points are affected heavily by noise, and
each blob in foreground mask is not smooth.thus the skeleton
points extracted from the blob is not ideal.

V. CONCLUSION
In this paper, we propose a new method to segment
occluded vehicles based on skeleton analysis in order to detect
every individual vehicle in the foreground. our method fist
detects the number of merged vehicles in one blob and finds
the segmentation points based on the effective skeleton nodes,
and then segment the blob according to the concave spot.
Experimental results show that our approach can effectively
segment occluded vehicles. compared to the related methods,
our method is effective and not complex without requiring a
fixed size vehicle model and special region to initialize each
vehicle individually, since the skeleton features are affected by
noise easily, the detection rate need to be improved. we plan to
address the problem in our future work.

VI. ACKNOWLEDGMENT
This Research Project was fully sponsored by {The Funda
mental Research Funds for the Central Universities} with grant
number { HEUCFZ1102 }.

REFERENCES
[1] Wei Fang, Yong Zhao, Yule Yuan, and Kai Liu, “Real-time Multiple
Vehicles Tracking with Occlusion Handling,” IEEE Conf. Image and
Graphics, China: Shenzhen, August 2011, pp. 667-672.
[2] D. Beymer, P. Mclauchlan, B. Coifman, and J. Malik, ”A real time
computer vision system for measuring traffic parameters,” in Proc, IEEE
Conf. Computer Vision and Pattern Recognition, 1997, pp. 495-501.
[3] Goo Jun, J. K. Aggarwal, and Muhittin Gokmen, “Tracking and
Segmentation of Highway Vehicles in Cluttered and Crowded Scenes,”
IEEE Workshop on Applications of Computer Vision, January 2008,
pp.1-6.
[4] Xuefeng Song and Ram Nevatia, ” Detection and Tracking of Moving
Vehicles in Crowded Scenes,” IEEE Workshop on Motion and Video
Computing, Feb. 2007, pp.4.
[5] C.C.C. Pang, W. W. L. Lam, and N. H. C. Yung, “A novel method for
resolving vehicle occlusion in a monocular traffic-image sequence,”

1059
1041

Authorized licensed use limited to: Central Scientific Instruments Organisation. Downloaded on November 25,2020 at 08:59:04 UTC from IEEE Xplore. Restrictions apply.