Automatic detection, tracking and counting of birds

in marine video content

Roeland T’Jampens1 , Francisco Hernandez1 , Florian Vandecasteele2 and Steven Verstockt2
1 Flanders Marine Institute (VLIZ), Wandelaarkaai 7, 8400 Oostende, Belgium
2 Data Science Lab, UGent-iMinds, Sint-Pietersnieuwstraat 41, 9000 Ghent, Belgium
e-mail: [email protected]

Abstract— Robust automatic detection of moving objects in a Automated object detection in a maritime and marine en-
marine context is a multi-faceted problem due to the complexity vironment is a complex problem due to various factors that
of the observed scene. The dynamic nature of the sea caused complicate the general video analysis approach. Camera mo-
by waves, boat wakes, and weather conditions poses huge
challenges for the development of a stable background model. tion, variety of objects and their appearance, highly dynamic
Moreover, camera motion, reflections, lightning and illumination background, meteorological circumstances, geographical loca-
changes may contribute to false detections. Dynamic background tions and direction of the camera make the detection process
subtraction (DBGS) is widely considered as a solution to tackle challenging [1]. In order to deal with all these issues, an
this issue in the scope of vessel detection for maritime traffic appropriate background model [2] is needed in combination
analysis. In this paper, the DBGS techniques suggested for ships
are investigated and optimized for the monitoring and tracking with a classifier to discriminate between the objects of interest
of birds in marine video content. In addition to background (i.e. birds) and other moving foreground objects. Furthermore,
subtraction, foreground candidates are filtered by a classifier being able to track the objects of interest across consecutive
based on their feature descriptors in order to remove non-bird video frames will facilitate detection, spatio-temporal behavior
objects. Different types of classifiers have been evaluated and analysis and recognition. In this paper, we propose a method-
results on a ground truth labeled dataset of challenging video
fragments show similar levels of precision and recall of about ology combining these three techniques and we particularly
95% for the best performing classifier. The remaining foreground focus on the specific problem of bird detection in dynamic
items are counted and birds are tracked along the video sequence scenes. First of all, we build a background model to remove as
using spatio-temporal motion prediction. This allows marine much of the sea as possible, without removing any flying birds
scientists to study the presence and behavior of birds. or birds resting on the water surface. Secondly, an image tex-
Keywords— dynamic background subtraction, texture analysis, ture analysis is performed to classify the foreground candidates
image classification, object detection, tracking, seabirds, marine
environment.
as water or bird. We investigate whether false detections (like
water) can be eliminated, while maintaining the true detections
(i.e. birds). This results in a number of validated foreground
I. I NTRODUCTION items per frame. Finally a spatio-temporal prediction technique
The Flanders Marine Institute (VLIZ)1 has evolved into the is used to track these items along the video sequence.
central coordination and information platform for marine and The remainder of this paper is organized as follows. Related
coastal scientific research in Flanders, Belgium. The objective work in marine/maritime video analysis is discussed in Sec-
of VLIZ is to support and promote Flemish marine scientific tion 2. Subsequently, the three main building blocks of our
research and to participate in local, national and international algorithm are described in Section 3 to 5. Next, Section 6
projects. The proposed research is linked to the European Life- presents our manually annotated VLIZ dataset (shown in Fig.
Watch project2 for monitoring biodiversity on earth. In the 1), the evaluation process and results. Finally, Section 7 lists
context of LifeWatch, VLIZ installed two PTZ cameras in a the conclusions and points out directions for future work.
marine setting for visual surveillance. The first camera was
placed at the Spuikom (Ostend), which is a local water mass
attracting bird and people alike. The second camera is set upon
the railing of a wind mill on the Thornton (sand)bank. The
resulting video feeds, shown in Fig. 1, allow marine biologists
to track the presence and behavior of birds on those sites.
Manual operation of these camera systems, however, is not
efficient due to fatigue, stress and the limited ability of human
operators to perform this kind of tasks. To aid the scientists and
avoid manual tallying, an automatic processing of the imagery
is investigated.
1 http://www.vliz.be/en/mission
2 http://www.lifewatch.eu/ Fig. 1. VLIZ dataset with manually annotated bird labels.

978-1-4673-8910-5/16/$31.00 ©2016 IEEE

 II. R ELATED WORK
The majority of video-based detection methods in a ma-
rine/maritime environment focus on vessel detection. Com-
pared to birds, these objects are larger in size and have more
robust/distinct features. Bloisi et al. [3], for example, use com-
putational efficient Haar-like classification and spatio-temporal
filtering to discriminate between vessels and other objects, e.g.,
reflections and wakes on the water surface. For bird detection,
however, the discrimination step of this technique will fail.
Furthermore, no real background modeling is used because the Fig. 2. Genaral overview of our dynamic background subtraction workflow.
authors mention to have problems to cope with its dynamic
behavior. For similar reasons, Rodrigo Da Silva Moreira et
al. [4], mainly focus on maritime vehicles that arise above The closest match to our solution is the system described
the horizon line. In order to detect the horizon line, Libe et in [1]. The author brings up many issues that arise in an
al. [5], evaluated four different methods which tend to have intelligent video surveillance system for dynamic monitoring
a quasi-similar accuracy. However, only focusing on objects of objects in a maritime environment. Their system uses poly-
above the horizon line would limit the practical applicability nomial background estimation and classification for reduction
of our approach and would complicate the tracking of birds. of false detections. However, performance of the described
The number of studies focusing on (sea) bird detection detection and tracking algorithm may be limited in our bird-
is still rather small, however, more recenlty research in this based scenario with low-resolution objects of interest. With
domain is beginning to appear more frequently. The skeleton a precision of 0.388 and recall of 0.516 their classifier based
based flying bird detection of Qunyu Xu and Xiaofeng Shi [6], system is at the moment not nearly able of fully eliminating the
for example, is based on the fact that the skeletal structure false detections while remaining the true detections. However,
that most flying birds possess is rather similar and quite the proposed future work in which additional features and a
discriminative against other objects. SVM-based classification combination of classifiers for the sea- and air part will be
is used to label the simplified skeleton features with the examined looks promising.
appropriate object classes. The proposed method detects flying
birds in side view with 90% accuracy, but needs improvement III. DYNAMIC BACKGROUND SUBTRACTION
for adapting the wide variations in poses and viewpoints of Maritime environments represent one of the most challeng-
free-flying birds. Histograms of Oriented Gradients (HOG) and ing scenarios for background subtraction due to the complexity
Local Binary Pattern (LBP) are two other types of features that of the monitored scene [3]. Based on their performance
are commonly used to capture the edge, texture and local shape reported in other works [10], two types of dynamic background
information of birds. The boosted HOG-LBP approach of Qing subtraction (DBGS) techniques were selected and investigated
et al. [7], for example, detects 80% of seabirds correctly. False using the workflow shown in Fig. 2. The first type defines indi-
positives are mainly due to background problems. In order to vidual pixels as fore- or background and is based on Temporal
tackle this issue, active area monitoring (for static backgrounds Median Filtering (TMF). The second type determines a block
with a similar color distribution as the foreground objects) and of pixels and is based on Local Binary Patterns (LBP) [11].
dynamic background modeling can be used. The latter one is The latter block-based approach has a lower level of detail
integrated in our approach. HOG features are also used in (pixel block) and is more complex, which makes it slower. As
the method of Yoshihashi et al. [8]. This method serves as all investigated LBP techniques [12] did not perform well in
a basis for testing a novel bird dataset and is compared to a our experiments, as shown in Fig. 3, these were excluded from
convolutional neural network (CNN) based method in more further research.
recent work [9], showing the effectiveness of rising CNN for
general image classification tasks. CNN-based bird detection
is also slightly investigated within this paper and will be the
focus of future work.
In practical environments, such as our VLIZ context, birds
tend to appear in low resolution even in a high resolution
image, i.e. the monitoring system has to cover a wide field of
view in order to perform a wide-range of tasks. It is important
to take this into account when evaluating and comparing with
other state-of-the-art approaches. A similar remark is made
by Yoshihashi et al. [9]. The average size of the birds in their Fig. 3. False detection problem of Local Binary Patterns (LBP) based
dataset is around 25 pixels, requiring recognition techniques background subtraction: results on different types of maritime/marine video
that works on very low-resolution images. content show several detection problems for accurate bird counting/analysis.
Fig. 4. Results of the proposed dynamic background subtraction. Flying Fig. 5. Dilation improves the median filtering based foreground detection,
birds are more easily found than birds resting on the water surface. Different making it possible to detect and analyze stationary objects such as birds resting
background model updating and thresholding approaches are investigated. on the water.

The pixel-based TMF approach keeps a limited set of Median filtering is very accurate for detecting moving
video images in memory using a frame buffer. Based on this objects (see Fig. 4). However, for stationary objects, such
frame buffer we continuously update the background model as birds resting on the water, pixel values can be within the
with each incoming frame. For each pixel in the new frame, threshold of the median value. As a result, the foreground
we determine the median value of the corresponding pixels mask might not fully encompass all foreground objects. This is
in the frame buffer in order to perform a threshold-based solved by dilating the foreground mask. In this way, stationary
background/foreground detection. objects can be detected as foreground, as shown in Fig. 5.
Three different approaches for updating the buffer are inves-
tigated. By default, the buffer is a sliding window in which the IV. B IRD / WATER CLASSIFICATION
oldest frame is deleted first, i.e. a first-in-first-out strategy. An
alternative approach uses a memoryless buffer in which frames To detect as many birds as possible, i.e. to limit the number
are replaced at random, i.e. a Random Temporal Median Filter of false negatives, some false positives (such as waves) must
(RTMF) [13]. In this manner the background model covers be allowed in the dynamic background subtraction. In order
a longer period of time/motion without enlarging the buffer. to filter out these false positives, we propose a classification
The technique is called memoryless because there is no link mechanism based on image texture analysis (Fig. 6). Key
between the buffer index and the moment in time the frame points and corresponding gradient features are extracted from
was added to the buffer. The third option that we investigated each detection result and are transformed into lower dimen-
is recursive, as only one frame is kept in memory and adjusted sional code words using the k-means clustering results of our
to each new frame. Approximate Temporal Median Filtering training samples. Next, these code words are classified as
(ATMF) [14] saves the first frame received. If the new pixel ’water’ or ’bird’ by a linear SVM classifier (Support Vector
value is higher than the model, the model is incremented by Machines). During an offline training phase, labeled (bird,
one. If a new pixel value is lower, the model pixel is decreased water) code words are fed to the classifier.
by one. For each of the investigated TMF approaches, all new SIFT, i.e. the Scale-Invariant Feature Transform method of
frames are thresholded with the background model, resulting in Lowe [15], is used as our baseline feature detector to which
the foreground mask. Pixels which differ more than the exper- other approaches will be compared. SIFT describes an image
imentally defined optimal thresholds are seen as foreground. by its most representative local characteristics and can be used
Again, three different approaches were investigated. A first for image stitching, object detection and object classification.
type of threshold uses a static threshold. However, as video The SIFT keypoints, i.e. circular image regions with an orien-
content changes frequently (e.g. due to lighting and waves), the tation, are represented by 128-dimensional vectors where the
threshold should change accordingly. An alternative is using fastest varying dimension is the orientation. Some examples of
a relative threshold, in which the margin is expressed as a VLIZ images with SIFT keypoints and orientations are shown
percentage instead of an actual pixel value difference. The last in Fig. 7. Important to mention is that this technique is scale
and the best approach is using a normalized threshold based and rotation invariant in the 2D picture plane and to some
on the mean and standard deviations of all pixel differences. degree to rotations in 3D.
 Fig. 6. General overview of our image texture analysis workflow. Fig. 8. General overview of our spatio-temporal tracking workflow.

For a classifier to work properly, all input data should V. S PATIO - TEMPORAL BIRD TRACKING
have the same size and be sufficiently small to attain high
performance. As our detected regions vary in size, the number In order to track the detected birds across the video frames,
of keypoints will greatly differ. Furthermore, a SIFT feature individual detections must be linked to detections in previous
vector contains 128 bins, which is also rather big. Both frames. For each registered object, the new location is pre-
issues complicate the training/classification process. However, dicted on its last position and movement. If the bounding box
features can be transformed into a combination of a limited of a newly found detection intersects with this prediction, the
number of visual code words [16]. Features of our training new detection is assumed the same object and the trajectory
set are clustered with k-means, creating a small vocabulary of of the object is extended with the path to the new detection
visual code words. The classifier is then trained on pictures (shown in Fig. 9). Detections with no corresponding objects
described as a combination of these code words. For our are seen as a new foreground object. Objects which have not
baseline SIFT feature detector, a nonlinear SVM classifier with been detected in a series of consecutive frames are deleted.
histogram intersection kernel [17] and only five code words Before removal, object tracking information is written to a
is found to achieve the highest accuracy with a precision over JSON-structured output file, as shown in Fig. 10. For each
90% (as discussed in Section VI - Table 1). object, we log its first and last appearance and the spatio-
Alternative feature extraction approaches based on SURF temporal information of all its observations. These kind of
and HOG features have also been investigated. As shown in loggings facilitate the querying of the video data and allows
Section VI - Table 2, HOG features [18] seem to perform direct access to objects of interest, supporting VLIZ scientists
better than SIFT for the bird classification task. HOG is in their study of the presence and behavior of birds.
a dense feature extraction method for images that extracts
features for all locations in the image as opposed to only VI. E VALUATION PROCESS AND RESULTS
the local neighborhood of keypoints like SIFT. Since the
detected regions in our set-up can be very small, the number The proposed methodology is objectively evaluated using a
of SIFT features can be too low to discriminate between birds ground truth labeled dataset of five videos coming from the
and water. Contrarily, the HOG descriptor technique counts VLIZ set-up, containing data recorded in different scenarios
occurrences of gradient orientation in localized portions, i.e. with varying light and weather conditions (as shown in Fig.
cells of 8x8 pixels, over the entire region of interset, leading to 1). Different dictionary sizes, feature extraction methods and
a higher accuracy. Furthermore, it is also important to mention SVM kernels have been investigated.
that we don’t use k-means clustering in the HOG based
approach, which drastically decreases the computational cost
(up to 80%) compared to our baseline SIFT-based approach.

Fig. 7. Gradient analysis of bird and water examples from the VLIZ dataset. Fig. 9. Results of our spatio-temporal tracking of flying birds.
 Fig. 10. Output of the proposed algorithm: JSON output file and annotated video stream.

Table 1 lists the results for our baseline SIFT based ap- best with an average precision, recall and F1-score of 0.95.
proach with three types of SVM kernels (linear, intersect Important to mention is that no k-means clustering is used
and RBF) and a dictionary size of 5 and 10. A balanced in HOG based classification, reducing the computational cost
training set of approx. 600 bird/water images and a test with 80% compared to the SIFT-based approach and making
set of 10000 bird/water images were used in the evaluation it more suitable for real time monitoring.
process. In general, we achieve a high precision and recall. The confusion matrix of the best performing HOG - RBF
The best configuration consists of a dictionary size of 5 and an approach contains 2892 true positives, 173 false negatives, 150
intersection kernel, resulting in 92% and 90% bird and water false positives and 3333 true negatives. In order to further
accuracy. Performance results of the classification step show decrease the number of false positives/negatives, we will
that this configuration is also the fastest. We also have studied investigate to incorporate temporal tracking information in
the impact of the type of median filter for dynamic background the classification process. Furthermore, first tests have already
subtraction. No particular median filter can be seen as overall been performed on state-of-the-art CNN architectures [19] for
best, but a normalized threshold is clearly the better option in bird/water classification. The gaining importance of CNN for
terms of accuracy and performance. object detection and recognition will also be part of future
Precision, recall and F1-scores of alternative approaches for work. Preliminary results of the Pyfaster object detection
our baseline SIFT based approach are presented in Table 2. and recognition (shown in Fig. 11) with the COCO dataset
In this test, grid-search and shuffle split cross validation were (http://mscoco.org) show the feasibility of this approach.
used to avoid overfitting and to get the best parameters. A set The proposed detection results can be computed in quasi
of 3065 bird samples and 3483 water samples were used in real-time and with low memory requirements when number of
this evaluation. In addition to SIFT, SURF and HOG have been objects is low. However, the performance diminishes greatly
tested to because both are mentioned quite often in literature. if a lot of detections need to be classified. Redesigning the
Again, different types of SVM kernels were evaluated. HOG feature selection process in the bird/water classification, i.e.
features in combination with an SVM RBF kernel perform our computational bottleneck, will be further investigated.

Table 1. Accuracy and performance results for our baseline SIFT algorithm. Table 2. Precision, recall and F1-score for different feature extractors.

Dictionary Kernel Bird Water Time Feature Kernel Precision Recall F1

5 Linear 92 % 86 % 91 s Bird Water Bird Water Bird Water
5 Intersect 92 % 90 % 89 s SURF Linear 0.86 0.89 0.88 0.87 0.87 0.88
5 RBF 90 % 89 % 95 s SIFT Linear 0.93 0.93 0.92 0.94 0.93 0.93
10 Linear 89 % 89 % 108 s SIFT RBF 0.93 0.93 0.92 0.94 0.93 0.93
10 Intersect 91 % 91 % 107 s HOG Linear 0.93 0.93 0.91 0.94 0.92 0.93
10 RBF 91 % 92 % 107 s HOG RBF 0.95 0.95 0.94 0.96 0.95 0.95
 [4] R. Da Silva Moreira, N. F. Favilla Ebecken, A. Soares Alves, F. Livernet,
A. Campillo-Navetti. A survey on video detection and tracking of
maritime vessels. International Journal of Signal Processing, 1, 47–60,
2016.
[5] T. Libe, E. Gershikov and S. Kosolapov. Comparison of Methods for
Horizon Line Detection in Sea Images In CONTENT 2012 : The Fourth
International Conference on Creative Content Technologies, proceedings,
Nice-France, July 2012.
[6] Q. Xu and X. Shi. A Simplified Bird Skeleton based Flying Bird
Detection In 11th World Congress on Intelligent Control and Automation,
proceedings, Shenyang-China, June-July 2014.
[7] C. Qing, P. Dickinson, S. Lawson and R. Freeman Automatic Testing
Seabird Detection based on Boosted HOG-LBP Descriptors In 18th IEEE
International Conference on Image Processing, proceedings, Brussels-
Belgium, September 2011.
[8] R. Yoshihashi, R. Kawakami, M. Iida and T. Naemura. Construction
of a Bird Image Dataset for Ecological Investigations. In 22nd IEEE
Fig. 11. CNN Py-faster object detection with COCO dataset shows the
International Conference on Image Processing, proceedings, Quebec-
feasibility of CNN based bird/water classification .
Canada, September 2015.
[9] R. Yoshihashi, R. Kawakami, M. Iida and T. Naemura Evaluation of Bird
Detection using Time-lapse Images around a Wind Farm In EWEA Wind
VII. C ONCLUSION Energy Event, proceedings, Paris-France, November 2015.
[10] M. Cristani, M. Farenzena, D. Bloisi, and V. Murino Background
subtraction for automated multisensor surveillance: A comprehensive
This paper proposes a methodology for automatic detection, review. EURASIP Journal on Advanced Signal Processing 1(24), 2010.
tracking and counting of birds in marine video content and [11] M. Heikkila and M. Pietikainen A texture-based method for modeling
evaluates different state-of-the-art building blocks within this the background and detecting moving objects IEEE Trans. Pattern
Analysis and Machine Intelligence, vol. 28, no. 4, pp. 657-662, 2006.
context. A dynamic background subtraction technique based [12] J. Chen, G. Zhao and M. Pietikainen Unsupervised dynamic texture
on normalized median filtering gives fast and accurate re- segmentation using local spatiotemporal descriptors In International
sults for the foreground object detection task. For bird/water Conference on Pattern Recognition, proceedings, pp. 1-4, Florida, USA,
December 2008.
classification of these foreground objects, the combination of [13] O. Barnich and M. V. Droogenbroeck Vibe: A universal background
HOG features and SVM RBF kernel performs best. With a subtraction algorithm for video sequences IEEE Transactions on Image
best obtained precision and recall of 95% it is concluded Processing, vol. 20, no. 6, pp. 1709-1724, 2011.
[14] S. S. Cheung and C. Kamath Robust techniques for background
that our current classifier based system is not yet able to subtraction in urban traffic video In SPIE - The International Society
fully eliminating the false detections while remaining the true for Optical Engineering 5308, proceedings, 2003.
detections. However, two suggestions for further improvement, [15] D. G. Lowe Object recognition from local scale-invariant features In
International Conference on Computer Vision, proceedings, vol. 2, Corfu,
i.e. temporal tracking information and CNN object recognition, Greece, September 1999.
are discussed within this paper and will be focus of future [16] G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray Visual
work. Finally, in order to track the birds across the video categorization with bags of keypoints In Workshop on Statistical Learning
in Computer Vision, ECCV, pp. 1-22, Prague, Czech Republic, May 2004.
sequence, a bounding box based spatio-temporal tracker is [17] S. Maji, A. C. Berg, and J. Malik Classification using intersection kernel
proposed. This facilitates the querying of the video data and support vector machines is efficient In IEEE Computer Vision and Pattern
allows direct access to objects of interest, supporting scientists Recognition, proceedings, Alaska, USA June 2008.
[18] N. Dalal and B. Triggs Histograms of oriented gradients for human
in their study of the presence and behavior of birds. detection In 2005 IEEE Computer Society Conference on Computer
Vision and Pattern Recognition, Proceedings, pp. 886-893, San Diego,
ACKNOWLEDGMENT CA, USA, June 2005.
[19] S. Ren, K. He, R. Girshick and J. Sun Faster R-CNN: Towards real-
The research activities as described in this paper were time object detection with region proposal networks Advances in neural
information processing systems, pp. 91-99, 2015.
funded by the Flanders Marine Institute (VLIZ), LifeWatch
(http://www.lifewatch.eu/), Ghent University, iMinds, Flanders
Innovation & Entrepreneurship (VLAIO), the Fund for Scien-
tific Research-Flanders, the Belgian Federal Science Policy
Office and the EU.

R EFERENCES
[1] M. Hartemink. Robust Automatic Object Detection in a Maritime
Environment. Master of Science Thesis, Faculty of Electrical Engineer-
ing, Mathematics and Computer Science (EEMCS), Delft University of
Technology, Delft-the Netherlands, 2012.
[2] M. Piccardi Background subtraction techniques: a review In IEEE
International Conference on Systems, Man and Cybernetics, proceedings,
vol. 4, pp. 3099-3104, The Hague, Netherlands, October 2004.
[3] D. Bloisi, L. Iocchi, M. Fiorini and G. Graziano. Camera Based Target
Recognition for Maritime Awareness. In 15th International Conference on
Information Fusion (FUSION), proceedings, pp. 1982-1987, Singapore,
July 2012.