Vision Processing in Intelligent CCTV for Mass

Transport Security
Abbas Bigdelil, Brian C. Lovell" Conrad Sanderson', Ting Shan' and Shaokang Chen'
'Safeguarding Australia program
NICTA, 300 Adelaide Street, Brisbane
QLD 4000, Australia
2ITEE, University of Queensland, Brisbane
QLD 4072, Australia

Abstract- Intelligent Surveillance Systems is attracting acquired from CCTV cameras in public spaces as the subjects
unprecedented attention from research and industry. In this are not usually aware of camera placement. Other capabilities
paper, we describe a real-life trial system where various video
analytic systems are used to detect events and objects of interests that are being trialed and developed include 1) robust
in a mass transport environment. The system configuration and detection of background changes, 2) tracking and
architecture of this system is presented. In addition to identification of people by their appearance across multiple
implementation and scalability challenges, we discuss issues cameras, 3) detecting suspicious events such as left luggage or
related to on-going trials in public spaces incorporating existing the dangerous behavior of people, and 4) video summarization
surveillance hardware.
to produce brief video summaries of activity.
In this paper, the configuration of the trial system and some
early results from commercial and NICTA research systems is
I. INTRODUCTION presented. We also discuss the implementation and scalability
CCTV surveillance systems are now seen as a major tool for challenges, as well as issues related to on-going real life trials
counterterrorism activities after they were shown to be able to in public spaces using existing surveillance hardware. The
successfully track the movements of the four suicide bombers main capabilities that are currently offered by leading
in the days before their attack on the London Underground in Intelligent Surveillance software vendors are demonstrated.
July 2005. Despite their usefulness, most current surveillance Technology gaps are identified and opportunities for computer
systems only provide reactive security by enabling the vision and pattern recognition research in the field of ICCTV
analysis of events after the terrorist attack has already are discussed.
occurred what is really needed by the counterterrorism II. RELATED WORK
community is proactive security to help prevent future attacks.
Intelligent Closed-Circuit TV (ICCTV) systems use powerful After the 9/11 attack on the World Trade Center in 2001,
computers to analyze video feeds to assist human operators to there has been a significant rush in both the industry and the
detect events of interest as they occur an example might be research community to develop advanced surveillance systems.
recognizing the face of a suspected terrorist or person of In particular, developing total solutions for protecting critical
interest in a crowded railway station. In 2006 NICTA was infrastructure has been on the forefront of R&D activities in
awarded a research grant to conduct long term trials of this field[ 1]. The solution is far much more complex than only
advanced ICCTV technologies in important and sensitive video analysis. It must cover activity detection through to
public spaces such as major ports and railway stations. The control room decision making.
trial will highlight operational and capability deficiencies in As far as Video Analysis is concerned, today there are a
current ICCTV systems and will focus NICTA's research on number of products already available in the market from a
capability gaps. The project is thus a unique collaboration of variety of vendors. Some of the most noticeable ones are
researchers, vendors, and user agencies aimed at delivering iOmniscient, NICE Systems, iSentry and Clarity. In the
advances in computer vision and pattern recognition for research arena, one of the closest works to ours was reported
human activity recognition. by Velastin et al[2] in a project called PRISMATICA. This
A primary objective of this work is the field trialing and on- project was part of a major European initiative on intelligent
going development of a system for the robust detection and transport systems. They describe an architecture that takes into
identification of persons of interest in a crowd. These people account the distributed nature of the detection processes and
will often have non-frontal facial presentation, be the need to allow for different types of devices and actuators.
photographed under various lighting conditions, and will The main contribution was a computer vision module used in
exhibit natural expressions such images are typically the system and its particular ability to detect situations of

1-4244-1226-9/07/$25.00 ©2007 IEEE.

Authorized licensed use limited to: University of Queensland. Downloaded on May 3, 2009 at 17:56 from IEEE Xplore. Restrictions apply.
interest in busy conditions. They report that the system
components have been implemented, integrated, and tested in
real metropolitan railway environments. This is clearly a first
step towards providing ambient intelligence in such complex Video Switch
scenarios. In [3] the same group presented some of the key
computer vision algorithms that were employed in
PRISMATICA and also published a survey on the current IDVR
state-of-the-art in the development of automated visual
surveillance systems [4]. This was quite useful in providing
researchers in the field with a summary of progress achieved Detected Alarms
to date and to identify areas where further research is needed.
In this survey, the authors have examined a wide range of
capabilities such as the ability to recognize objects and Display for Operator
humans Describing human actions and interactions from Figure 1. System Configuration
information acquired by sensors is essential for automated
visual surveillance. However the emphasis of their review was Camera Management Layer
on discussion of the creation of intelligent distributed The Camera Management Layer ingests the video feeds
automated surveillance systems. from all the cameras connected to the system. This layer is
Video analysis covers a wide range of applications such as aware of the GIS correct location of each camera with respect
tracking [5], pedestrian detection [6], face recognition[7], as to the 3D model. This is needed for correct registration of the
well as more complex detections such as events of interest as camera view on top of the 3D model. The camera
reported in [8]. management layer also handles the stitching and overlay of
In addition to the video analysis that creates the initial the overlapping cameras. Where a PTZ camera is used, this
alarms, an immersive 3D visual assessment [1] can be layer manages the camera control as the operator, for example,
employed for situational awareness and to manage the reaction follows a moving object from one camera to the next.
process. This can then be coupled with wide area command
and control capabilities to allow control from a remote Alarm Management and Presentation Layer
location. The Praetorian suit of software packages provides The second major module is the Alarm Management and
such an environment. The open architecture of Praetorian in Presentation layer. At this layer, various alarms that are
essence works as an operating system that can absorb alerts generated by one or more of the video analytic software
generated from various video analytic systems. systems are handled. The alarms are overlaid on the 3D model.
III. SYSTEM CONFIGURATION An incident can be indicated by simply jumping to the camera
view that has created the alarm.
Figure 1 shows the basic configuration of one of the real life
IV. FIELD TRIALS
trial systems. As the existing cameras were all analogue, a
video switch was used to digitize the feed and recorded In our trials, we used four leading intelligent video
footage from all camera was stored on a digital video recorder. surveillance systems to analyze the video sequences obtained
The recorded footage on the DVR can be accessed by the from the railway platform. Table 1 shows the various alarms
video analytic software. As mentioned earlier, a number of that can be generated from the four systems. It should be noted
commercially available video analytic software systems were that certain specialized capabilities are only available in some
used in this exercise. We also used some of the experimental and not all systems.
systems that are being developed at NICTA labs. These TABLE I: ExiSTING AND DESIRED ALARMS AVAILABLE IN SOME
include Robust Face Recognition as well as appearance-based INTELLIGENT SURVEILLANCE SYSTEMS
tracking of individuals. The 3D Immersive Video Presentation Common Capabilities Specialized Capabilities
layer uses the followings sources of information: original Left Object Detection Association of object with owner
video feeds, the pre-produced 3D model of the premises, and Appearance Model based Tracking
the generated alarms as they are detected. The output of the (tracking individuals)
Presentation Model is what is displayed on an operator's Tripwire
screen. This is done to put the alarm in the context of the 3D Slip and Fall Detection
environment an essential step if responders are to react Behavior Analysis
appropriately in real-time.
The software that creates the 3D Immersive Presentation is One of the "test-beds" we are using for our field trials is a
quite substantial and a quick overview of the main modules is railway station in Brisbane (Australia), which provides us with
presented below. Interested readers are referred to [1] for more implementation and installation issues that can be expected to
details. arise in similar mass-transport facilities.

Authorized licensed use limited to: University of Queensland. Downloaded on May 3, 2009 at 17:56 from IEEE Xplore. Restrictions apply.
 Capturing the feed in a real-world situation can be
problematic, as there should be no disruption in operational
capability of existing security systems. The optimal approach
would be to simply use an IP camera feeds. However, in many
existing surveillance systems the cameras are analogue and
often their streams are fed to relatively old digital recording
equipment. Limitations of such systems can include low
resolution, recording only a few frames per second, non-
uniform time delay between frames, and proprietary codecs.
To avoid disruption while at the same time obtaining video
streams which are more appropriate for an intelligent
surveillance system, it is useful to tap directly into the
analogue video feeds and process them via dedicated
analogue-to-digital video matrix switches.
Apart from the technical challenges, issues in many other
domains may also arise. Privacy laws or policies at the
national, state, municipal or organizational level may prevent
surveillance footage being used for research even if the video
is already being used for security monitoring the primary
purpose of the data collection is the main issue here. Moreover, Figure 2. NICTA's real-time face detection and identification of persons of
without careful consultation and/or explanation, privacy interest in footage from a CCTV camera in a major railway station
groups as well as the general public can become
uncomfortable with security research. Some people may
simply wish not to be recorded as they have no desire in
having photos or videos of themselves being viewable by
other people. Plaques and warning signs indicating that
surveillance recordings are being gathered for research
purposes may allow people to consciously avoid monitored
areas, possibly invalidating results.
Research code is often written by scientists/engineers (not
necessarily professional programmers) for the explicit purpose
of evaluating new methods. While this is sufficient to obtain
experimental results which can be published, there can be little
incentive to keep the code in a maintainable state or to
guarantee that the underlying algorithm implementation is
actually stable. Furthermore, research code is often written in
Matlab which requires a nontrivial conversion into a language Figure 3. NICTA's tracking and labeling of commuters on a railway
such as C++ to allow faster processing and integration with platform using CCTV cameras
other software (e.g. via an SDK). The conversion may be quite
difficult if, for example, the experimental implementation
relies on elaborate functions and toolkits included with Matlab.

Preliminary Observations
In our initial field trial for public dissemination, we
recorded 20 minutes of footage from seven cameras on a
single platform in a major metropolitan railway station.
During the recording time, a range of scenarios were staged on
the platform. These include:
* left objects (of various sizes and position at different
locations)
* crossing yellow safety lines
* attempting to break the vending machines
* other normal as well as abnormal passenger behavior
on the platform Figure 4. Commercial left object detection on a railway platform using
CCTV cameras

Authorized licensed use limited to: University of Queensland. Downloaded on May 3, 2009 at 17:56 from IEEE Xplore. Restrictions apply.
 The recorded footage was then fed through commercial as robust face and number plate recognition, human activity
well as research video analytic software. Screen shots of some recognition etc. The adoption of Smart cameras is resulting in
of the examples are given in Figures 2-4. Below we have a paradigm shift from a central to a distributed control
summarized the initial observation from this exercise: surveillance system. The main motivation for this shift is
* Set up and calibration of commercial systems for different increasing surveillance systems' functionality, availability,
cameras and different type of alarms is time consuming and autonomy.
and requires a fair bit of trial and error. VI. CONCLUSION
* The level of capabilities offered by various commercial
software companies varies considerably. Intelligent Surveillance Systems is attracting much attention
* There are many areas where the research community can from research and industry. In this paper, we described a real-
make significant contributions. NICTA is currently life trial system located on a public railway platform. We
progressing towards meeting two of these areas: robust applied various video analytic systems both from commercial
face recognition and human behavior analysis. vendors as well as those under development in the research
community to detect events and objects of interests. Some of
the challenges involved in implementation, integration and
system configuration were examined.
ACKNOWLEDGMENTS
This project is supported by a grant from the Australian
Government Department of the Prime Minister and Cabinet.
NICTA is funded by the Australian Government's Backing
Australia's Ability initiative, in part through the Australian
Research Council.
REFERENCES
[1] Francisco, G., Roberts, S., Hanna, K. and Heubusch, J., "Critical
infrastructure security confidence through automated thermal imaging,"
In Proceedings SPIE Vol. 6206, Infrared Technology and Applications
XXXII, May 2006.
Figure 5. Immersive 3-D Visual Presentation of Camera View and 3-D [2] Velastin, S.A., Boghossian, B.A., Lo B.P.L., Jie Sun, Vicencio-Silva,
model on a railway platform M.A., "PRISMATICA: toward ambient intelligence in public transport
environments," Systems, Man and Cybernetics, Part A, IEEE
V. EMERGING TECHNOLOGIES Transactions on, Vol 35 (1), Jan. 2005, pp. 164- 182.
[3] Black, J., Velastin, S., Boghossian, B., "A real time surveillance system
Smarter Cameras for metropolitan railways," Advanced Video and Signal Based
Smart Cameras are slowly being introduced in the emerging Surveillance, IEEE Conference on, 15-16 Sept. 2005, pp. 189- 194.
surveillance systems. They usually perform a set of low-level [4] Valera, M. and Velastin, S.A., "Intelligent distributed surveillance
systems: a review," Vision, Image and Signal Processing, IEE
image processing operations on the input frames at the sensor Proceedings, Vol 152(2), 2005, pp.192- 204.
end. The types of computation performed on Smart Cameras [5] Fuentes, L.M. and Velastin, S.A., "From tracking to advanced
vary between applications. In most cases, it is limited to surveillance," Image Processing, 2003. ICIP 2003. Proceedings. 2003
International Conference on, 2003, Vol 3, pp. III- 121-4.
compressing images and transmission to the host computer. In [6] Coimbra, M. and Davies, M., "Segmentation of moving pedestrians
more recent years however, a wide range of applications have within the compressed domain," Acoustics, Speech, and Signal
emerged. Algorithms such as face recognition are good Processing, 2004. Proceedings. (ICASSP '04). IEEE International
Conference on, Vol 3, 2004, pp. iii- 605-8.
examples of a Smart Camera application. Fatemi et al. [7] Shan, T., Lovell, B.C., Chen, S. and Bigdeli, A., "Reliable Face
introduced a real-time face recognition System on a Smart Recognition for Intelligent CCTV," In Proceedings of RNSA Security
Camera [9] while Kleihost et al [10] took this concept further Technology Conference, Canberra 2006, pp. 356-364.
[8] Ziliani, F., Velastin, S., Porikli, F., Marcenaro, L., Kelliher, T.,
by developing a dedicated Smart Camera for face recognition. Cavallaro, A. and Bruneaut, P., "Performance evaluation of event
Ozer & Wolf [ 1] have implemented Human Activity detection solutions: the CREDS experience" Advanced Video and Signal
recognition on a Smart Camera. Based Surveillance. IEEE Conference on, 2005, pp. 201- 206.
[9] Fatemi, H., Kleihorst, R., Corporall, H., and Jonker, P., "Real-time Face
With growing interest in security and surveillance, Smart Recognition on Smart Camera," Proceedings of Advance Concepts for
Cameras are finding their way into intelligent surveillance Intelligent Vision Systems, Belgium, September 2003, pp. 222-227.
systems too. Matsushita et al [12] used Smart Camera for [10] Kleihorst, R., Reuvers, M., Krose, B. and Broers, H., "A Smart Camera
For Face Recognition," Proceedings of International Conference on
scene capturing and identity recognition. The majority of the Image Processing (ICIP 2004), 2004, pp. 2849-2852.
video processing and analysis in the existing surveillance [11] Ozer, B. and Wolf, W, "A smart camera for real-time human activity
systems is executed at a central host using standard recognition," In Proceedings, SIPS-O1, IEEE. 2001, pp. 217-225.
[12] Matsushita, N., Hihara, D., Ushiro, T., Yoshimura, S., Remkimito, J.
workstation racks. Up-and-coming embedded computing and Yamamoto, Y, "ID CAM: A Smart Camera for Scene Capturing and
technology is being used to implement Smart Cameras with ID Recognition," In proceedings of the Second IEEEIACM International
greater processing capabilities. Such capabilities include more Symposium on Mixed and Augmented Reality (ISMAR '03). October
2003, pp. 227- 236.

Authorized licensed use limited to: University of Queensland. Downloaded on May 3, 2009 at 17:56 from IEEE Xplore. Restrictions apply.