A Framework for Intelligent Sensor Network with Video Camera for

Structural Health Monitoring of Bridges

Arslan Basharat Necati Catbas Mubarak Shah
Dept. of Computer Science, Dept. of Civil Engineering, Dept. of Computer Science,
University of Central Florida, University of Central Florida, University of Central Florida,
Orlando, FL-32816 Orlando, FL-32816 Orlando, FL-32816
[email protected] [email protected] [email protected]

Abstract condition of a structure. Although these tools might be

useful for local damage detection and analysis but they
Wireless sensor network (WSN) gives the are not suitable for a global monitoring system. To
characteristics of an effective, feasible and fairly implement a SHM successfully, we would like to
reliable monitoring system which shows promise for propose monitoring the civil structures globally as
Structural Health Monitoring (SHM) applications. well as locally, using sensors and cameras, and finally
Monitoring of civil structures generates a large detect any abnormal behavior in real time to mitigate
amount of sensor data that is used for structural any catastrophic failures. Of course, this monitoring
anomaly detection. Efficiently dealing with this large system has excellent potential to improve the regular
amount of data in a resource-constrained WSN is a operation and maintenance of the structures.
challenge. This paper proposes a, WSN based, novel Comprehensive systems use a large number of
framework that triggers smart events from sensor sensors to monitor structure’s health. Typically these
data. These events are useful for both intelligent data systems comprise of vibration sensors, strain gauges
recording and video camera control. The operation of and other similar sensors that are wired through cables
this framework consists of active & passive sensing to a central data acquisition system or a network of
modes. In passive mode, selected nodes can data acquisition systems. The data acquisition systems
intelligently interpret local sensor data to trigger not only record all the data but also facilitate data
appropriate events. In active mode, most of the interpretation. Problems with these systems include
sensing nodes perform high frequency sampling and high initial cost, large complicated mesh of cables,
record useful data. Unnecessary data is suppressed difficult installation and maintenance, etc.
which improves the lifespan of the network and SHM can benefit from the recent research and
simplifies data management. development in the area of Micro-Electro-Mechanical
System (MEMS) and Wireless Sensor Network (WSN).
MEMS technology has greatly improved the
miniaturized sensors of different types including the
1. Introduction accelerometers and tilt-meters that are extensively
used in civil SHM. These new sensors consume lesser
The process of continuously monitoring the status energy, are relatively inexpensive and have been
of a structure to detect damage can be defined as shown to give promising responses when compared
Structural Health Monitoring (SHM). The importance with the conventional wired sensors.
of health monitoring of civil structures has gained A WSN is composed of a large number of small
considerable attention over the last two decades. A nodes that have sensing, processing and wireless
variety of methods have been employed for SHM, communication capabilities. Use of WSN for SHM
which have improved over time with the evolution of gives us many advantages such as:
technology. One of the conventional methods is visual
x System setup & maintenance cost is remarkably
inspection by humans for any signs of apparent
reduced.
damage. But this technique does not prove to be very
x No cables are required for data transfer because
useful because it is limited to the visible defects only.
the communication is wireless.
Human judgment error and labor-intensive nature are
other drawbacks of this approach. Another set of tools x Data processing & interpretation can be
such as tomography equipment analyzes the internal distributed across the network nodes.

Proceedings of the 3rd Int’l Conf. on Pervasive Computing and Communications Workshops (PerCom 2005 Workshops)
0-7695-2300-5/05 $20.00 © 2005 IEEE
 x System becomes more fault-tolerant. In case of a many different areas. Due to the limitation of the
partial system failure the rest of the system is resources available on the WSN nodes, only the
capable of performing its task independently. applications that have low sampling rates and energy
x Overall system response time improves due to requirements have been reported. The examples of
anomaly detection through data processing on the these applications are habitat monitoring [3], product
nodes instead of central base station. quality monitoring through atmosphere control [2]. On
the other hand, the data rate is much higher in SHM
WSN based framework does not come only with application. Some researchers have developed new
the advantages, there are some limitations that have to platforms to overcome the limitations of sensor node
be considered and further evaluated. hardware [4, 5].
x Processing power & communication bandwidth In the area of WSN, researchers have been
available on the nodes are very limited. working on the services that are required in most of
x Each node has restricted battery life that has to be the applications. Time synchronization between the
preserved by efficient consumption. nodes of the network is one such service. Data from
x A multi-hop protocol is to be established for different sensor nodes can be accurately synchronized
communicating with the central base station. if a good time synchronization service is functional.
Ganeriwal et al. [9] propose a hierarchical approach for
Along with these, there are some other drawbacks time synchronization among nodes. It is called
in the use of WSN based system for SHM Timing-sync Protocol for Sensor Networks (TPSN).
applications. The research continues to address these Similarly, there are other key issues related to system
problems and to make WSN based solution more deployment and operation. These are discussed in [6].
robust and reliable for this domain. The system has to The research community in SHM has been in the
perform the regular sensing operation not only under process of evaluating and exploring the features of
the normal circumstances but also during extreme monitoring systems with WSN. We are mainly
weather and environmental conditions as well as after interested in addressing real life problems using novel
hazards occur. Another very important consideration is technologies within context of a complete SHM
the maintenance cycle span and overall life of the system as discussed in [10].
system. As part of this effort, first, we would like to
In this paper, we propose the basic characteristics address the issue of efficiently handling large amount
of an efficient framework that can be employed to of vibration data on resource-constrained WSN. In this
perform SHM for bridges. We use a WSN (MICA paper, we propose techniques that improve data traffic
motes [1]) that exploits the distributed processing on the network. At the same time, we emphasize the
available on its nodes. Distributed data interpretation usefulness of the proposed framework for interpreting
is used to intelligently detect local data trends (events) data to trigger certain events related to the data trend.
instead of just sending raw data to the base station in a Another novel contribution of this study will be the
dumb fashion. Examples of these events are “normal integration of video processing unit that enhances the
vibration detected”, “abnormal vibration detected” or capability of the SHM. Autonomous camera control
“large structure tilt detected”. Using this technique we and data synchronization between the video and sensor
can also reduce network traffic and improve the data are the key features of our framework.
battery life of the WSN. When a critical event is
detected on the bridge, we can prompt a video camera 3. Proposed system architecture
to pan/tilt/zoom into the local area and monitor
activity. Camera is attached to a video processing 3.1 Apparatus
module that can provide video frames of the area of
interest, synchronized with other sensor data. This way For testing and evaluation, MICA series motes
we can improve the conventional SHM system have been used for this study. This platform is
utilizing new technology. This framework has been equipped with Atmel’s microcontroller running on
tested in the laboratory environment. 8MHz, 868/916 MHz multi-channel RF transceiver
with effective data rate of 4-10Kbps, 512KB of serial
2. Related work flash memory for data storage and can run up to one
year on AA Batteries (using sleep modes) [1]. A
The area of WSN has greatly benefited from the sensor board, with different sensors, is attached to the
research carried out towards Smart Dust by Pister et main mote. Acceleration and temperature sensors are
al. [12]. The application of WSN has been shown in used in this framework. TinyOS, a component based

Proceedings of the 3rd Int’l Conf. on Pervasive Computing and Communications Workshops (PerCom 2005 Workshops)
0-7695-2300-5/05 $20.00 © 2005 IEEE
 operating system, runs on motes [7]. Information message to the GMs in range and confirms their
about these motes shows that they have limited availability. In case of an absent GM, it takes its
computing resources available. Motes build a network responsibilities in the cluster and tries to continue the
and send data to the base station. It records this data normal operations.
into a local data store and also transmits to a remote
observatory through wireless internet access. Figure 1
illustrates the point.

Mote
Base Station

Local Database
Figure 1. Apparatus deployment on the bridge

The video cameras are used to monitor the traffic

Sensing Mote Backup Mote Base Station
and other activities on the bridge from a remote Gateway Mote Cluster Data path
observatory. The set of cameras should be deployed so
that using pan/tilt/zoom complete area of the bridge Figure 2. Network architecture on the bridge
can be covered. For simplification, we have used only
one camera in our experiments in the lab. A standard
PC was used as the video processing unit. It can send
4. System operation
& receive messages from the WSN nodes on the
bridge. These messages are routed through the base 4.1 Initialization
station of the WSN. This is further explained in
section 4. There are some essential services that have to be
incorporated for a WSN to operate in a meaningful
3.2 Network architecture fashion. We cover sensor localization and time
synchronization here.
The network of motes is organized into a group of GM occupies a critical position in the network as
clusters. Each cluster has one functional Gateway explained in the previous section. Localization is
Mote (GM) at a given time and about 4-8 Sensing carried out through a handshaking protocol.
Mote (SM). Responsibility of a GM is to coordinate In the first phase, base station initiates a
the communication between the SMs and base station. localization message for the GMs, which propagate
At the same time, GM is responsible for passing the through the GMs and each one of them establishes a
messages between the GMs of neighboring clusters, neighbor-list. Since this message is initiated from base
thus completing the route to the base station. Certainly station, each GM knows which neighbors to use for
the GMs will have to cope up with the network traffic reaching base station.
from within the cluster and from the other neighboring In the second phase, GMs build up their local SM-
clusters. This means that the GM should have higher list through a similar handshake with SMs. Every GM
network bandwidth. Considering the current uses an assigned time-slice for this phase. This ensures
availability of the hardware, we suggest the use of that an SM lying in area of coverage of more than one
MICAZ Motes. They have 2.4GHz radio transceiver GMs only gets first cluster assignment. But this SM
giving network bandwidth nearly 6 times that of can store a list of GMs that are in communication
MICA2 Motes. Figure 2 show the top view of range, which will help in the future in case of primary
suggested WSN layout. The dotted line shows the GM failure. Later during normal operation of WSN
cluster boundary and arrows show the direction of data some other housekeeping messages are periodically
flow. transferred which help in both time synchronization
Backup Motes are available for redundancy in and sensor localization.
case of a node failure. The backup mote normally Time synchronization is one of the critical
stays in sleep mode but periodically sends a heartbeat services of a WSN when it is being used for SHM.

Proceedings of the 3rd Int’l Conf. on Pervasive Computing and Communications Workshops (PerCom 2005 Workshops)
0-7695-2300-5/05 $20.00 © 2005 IEEE
 The data from different sensors has to be synchronized assumed that we are not interested in any vibration
precisely. A scheme proposed by Ganeriwal et al. [9] readings below TL, therefore, we can reduce the
works fairly well for this application. sampling frequency but we do not want to go too low
so as to miss high frequency data in the start. We have
4.2 Event detection used values close to 80Hz for the Passive Mode
sampling frequency.
WSN can start sensing operation once the initial Once a SENSE_EVENT has been detected by the
setup services have completed. The sampling rate of sensing SM in a cluster, it sends the message in its
the temperature data is very low and only one mote cluster to the other SMs to switch to wakeup mode and
from a cluster has to send the temperature once after start sensing at a higher frequency (we used 150Hz).
every 5 minutes. On the other hand, the accelerometer All the SMs in that cluster switch to Active Mode
measures vibration and can also be used for tilt now. They remain in Active Mode for a limited time
detection. The sampling frequency on accelerometer but before going back to Passive Mode the mean shift
may be over 100Hz for SHM applications depending value will be checked and the decision will be taken
on the frequency band of interest. If all the SMs in the accordingly. During the Active Mode SMs use local
network sense simultaneously at this rate, we will have flash to store the data and start sending the data to the
too large a data to handle efficiently. local GM during the allotted time slice. Figure 3
We propose an event detection technique that also shows single and two mote case of SENS_EVENT
helps in reducing unnecessary network traffic. We and change in mode is depicted through color coding.
assume that vibration data is unnecessary if the
activity metric value is below a low threshold (TL)
value. We have used mean shift vector as the activity
metric.
nx
M ( P ) 1 / n x ¦ ( xi  P o )
i 1

where
Po is initial mean value, nx is the number
x
of data points and i is one data point. In our
experiments, we have used 15 for n x and 10 for TL but
these are parameters that can be manually tweaked to
suite noise conditions or user preference. The values a.
can also be changed on run-time throughout the
network. According to the tests, this metric performs
reasonably well in the lab setup. Figure 3.a shows the
vibration data from one sensor.
Using above mentioned method we detect two
events in the vibration data.
x First one is “normal vibration detected”.
SENSE_EVENT label is assigned to this event,
which is triggered when mean shift vector
magnitude is higher threshold value of TL. (See
Figure 3).
x Second one is “abnormal vibration detected”.
CAMERA_EVENT label is used for this event,
shows that mean shift value is higher than another b.
threshold TH. We have used TH = 25 in our Figure 3. Vibration detection (Time vs. Sensor reading)
a. Single mote data: The color change from green to red
experiments. depicts that mean shift exceeded the threshold value TL = 10.
b. Two time synchronized motes. SENSE_EVENT detected
During the normal mode of operation, with no simultaneously and the change is color coded.
vibrations in a cluster, the SMs are in Passive Mode.
Only one SM is performing sensing and others are in CAMERA_EVENT is triggered when the activity
sleep mode. This SM uses a lower sampling frequency metric is higher than some threshold value TH. It
for conservation of the battery life. Since we had means that there is significant vibration in that section,

Proceedings of the 3rd Int’l Conf. on Pervasive Computing and Communications Workshops (PerCom 2005 Workshops)
0-7695-2300-5/05 $20.00 © 2005 IEEE
 therefore, it should be visually observed. This event system. To further enhance the efficiency of this
message is passed to the base station through the system domain specific data compression on the nodes
intermediate GMs, then the base station uses the could be helpful.
location information about the sending mote and Many factors affect the performance of a SHM
requests camera to pan/tilt/zoom into the system. This includes environmental conditions, which
corresponding section of the bridge. A controller at a have to be observed in a real-life outdoor scenario
remote location can look at the live video if he/she over a longer period of time. It also remains to be seen
wishes; otherwise the video clip related to an activity that how well these algorithms perform in real-world
can be recorded into the database. Since we have the situation with problems like node failures, complex
data about the SM that triggered this event, we can civil structures, extreme weather conditions etc. Only
synchronize this data with the video frames. Figure 4 then a complete transition from the conventional SHM
shows a camera overlooking the structure being used to the WSN based SHM can be made.
in the lab. We have tested the triggering of a video camera
using event detection on mote data. Another
Wireless
possibility is doing the reverse of this, i.e. triggering
Wired
sensors in an area if unusual activity is observed in the
video. There is interesting work in computer vision
community for activity recognition and detection of
abnormal activities in videos [11]. For example, the
Video
normal activity on a bridge is the traffic in a
continuous flow but if there is a heavy truck broken
down on the bridge that will be abnormal activity.
This might induce heavy loading of the bridge
structure which could affect its health. We will further
Data Acquisition System evaluate system trigger based on real time processed
streaming video data.
Actuator

6. References
Figure 4. Lab setup (in parallel with
conventional sensors)
[1] Crossbow Technology Inc. (http://www.xbow.com)
[2] Wireless Vineyard (www.intel.com/labs/features/ rs01031.htm)
4.3 Optimization techniques
[3] A. Mainwaring, J. Polastre, R. Szewczyk, D. Culler, and J.
Anderson, “WSNs for habitat monitoring”, WSNA '02, ACM, 2002.
To improve the performance of the WSN, we
propose some optimizations. [4] Straser, E. G. and A. S. Kiremidjian, “A modular Wireless
Damage Monitoring System for Structures”, Report #128, John A.
x To uniformly use the resources within a cluster, Blume Earthquake Engineering Center, 1998
we switch the passive sensing responsibility
[5] J. P. Lynch, A. Sundararajanb, K. H. Law, H. Sohn and C. R.
among the local SMs in a round-robin manner. Farrar, “Design of a Wireless Active Sensing Unit for Structural
x To reduce the amount of data being transmitted, a Health Monitoring”, International Symposium on Smart Structures
packet of data gets a common time-stamp instead and Materials, San Diego, CA, 2004.
of individual one’s for each data values. [6] S Sundresh, G. Agha, Ki. Mechitov, W. Y. Kim, and Y. Kwon,
Timestamp for individual data values can be “Coordination Services for Wireless Sensor Networks”, Intl.
recovered on the base station using the current Workshop on Advanced Sensors, SHM and Smart Structures, 2003.
sampling frequency information. [7] TinyOS (http://www.tinyos.net)
x Each value of accelerometer data is in 10-bit [8] A. Kottapalli, A. S. Kiremidjian, J. P. Lynch, E. C., T. W.
digital form. By default the TinyOS component Kenny, K. H. Law, Y. Lei, “Two-tiered wireless sensor network
returns this data in a 16-bit variable and about architecture for structural health monitoring”, Proc. SPIE Smart
Structures and Materials 2003.
37.5% of the bit-space is wasted. We recover this
bit space through data packing on the sending end [9] S. Ganeriwal, R. Kumar, M. B. Srivastava, “Timing-sync
Protocol for Sensor Networks”, SenSys ‘03, 2003.
and then unpacking on the receiving end.
[10] N. Catbas, M. Shah, J. Burkett, and A. Basharat, “Challenges in
Structural Health Monitoring”, Fourth IWSC, 2004
5. Conclusion
[11] I. Junejo, O. Javed, M. Shah, “Multi Feature Path Modeling for
Video Surveillance”, ICPR, 2004.
Experiments show that effective distributed
processing can enhance the effectiveness of the SHM

Proceedings of the 3rd Int’l Conf. on Pervasive Computing and Communications Workshops (PerCom 2005 Workshops)
0-7695-2300-5/05 $20.00 © 2005 IEEE