A Non-Intrusive Multi-Sensor System for Characterizing Driver Behavior

Jo ao G. P. Rodrigues

, Fausto Vieira

, Tiago T. V. Vinhoza

, Jo ao Barros

and Jo ao P. Silva Cunha

AbstractUnderstanding driver behavior is critical towards

ensuring superior levels of safety and environmental sustain-
ability in intelligent transportation systems. Existing solutions
for vital sign extraction are generally intrusive in that they
affect the comfort of the driver and may consequently lead
to biased observations. Moreover, low-complexity devices such
as GPS receivers and the multitude of sensors present in
the vehicle are yet to be exploited to the full extent of
their capabilities. We present a real-life system that combines
wearable non-intrusive heart wave monitors with a wireless
enabled computing platform capable of gathering and process-
ing the data streams of multiple in-vehicle sources. Observed
variables include electrocardiogram, vehicle location, speed,
acceleration, fuel consumption, and pedal position, among
others. Preliminary results show that the proposed system is
well suited not only for characterizing driver behavior but
also for identifying and mapping potentially dangerous road
segments and intersections.
I. INTRODUCTION
Improvements in road safety and transportation efciency
by means of intelligent transportation systems (ITS) are ar-
guably dependent on our ability to characterize the behavior
of different classes of drivers and their response to various
events, technologies and travel conditions. Fortunately, the
vehicles of today are equipped with a large number of
sensors that can be leveraged to extract behavioral patterns.
Typical instances, for example [1], integrate on-board sensors
with human-computer interfaces and estimate driver stress
levels by means of pressure sensors in the steering wheel
and on the gearshift knob. Although capable of detecting
aggressive braking, acceleration, steering, or inefcient fuel
consumption, among other patterns, the system forces drivers
to hold the steering wheel in a particular position, which
modies their natural behavior and affects the outcome of
the measurement process.
Another limitation of typical systems aimed at character-
izing driver behavior is their lack of connectivity. Typically,
such devices are operated as stand-alone on-board units that
store the gathered data for future ofine processing. With
the advent of vehicular sensor networks [2], which consist
of cars, buses and other vehicles capable of collecting sensor
measurements and transmitting the data over mobile ad-hoc
networks, it is now possible to grant remote access to in-
vehicle sensing systems and observe the collected data in

Instituto de Telecomunicac oes, Departamento de Engenharia Elec-

trot ecnica e Computadores, Faculdade de Engenharia da Universidade do
Porto, Porto, Portugal

Dep. de Electr onica, Telecomunicac oes e Inform atica and IEETA, Uni-
versidade de Aveiro, Aveiro, Portugal
This work was supported in part by FCT and the international partnership
program MIT|Portugal under the MISC project MIT-PT/TS-ITS/0059/2008.
real-time. The application described in [3] gathers informa-
tion from Global Positioning System (GPS) and On-Board
Diagnostics (OBD) devices from several cars, which is then
sent to a central data collection point over wireless channels
with some delay tolerance. Although known to be largely
dependent on the mobility patterns [4], wireless interfaces
for inter-vehicle and vehicle-to-infrastructure communication
can be deemed as robust enough for real-time monitoring of
vehicle and driver behavior.
Ultimately, we would like to correlate positioning, time
and road map information with the stress levels and emo-
tional response of drivers. From the analysis of the daily
annotations of a large number of drivers one can infer that
near accident incidents are highly correlated with feelings of
anger [5]. From [6], it becomes clear that some locations are
more likely to induce stress than others, induced either from
feelings of risk or from extra driving difculty [7]. It has
also been suggested that some cardiac waves are altered by
driving events [8]. Thus it is reasonable to assume that overall
road safety can be increased, if such feelings are detected in
time using biomedical signal processing (see e.g. [6] or [9])
in combination with positioning information and in-vehicle
sensing.
Seeking to overcome the aforementioned drawbacks of
existing driver monitoring systems and simultaneously lever-
age the benets of vehicular sensor networks, we set out to
develop a solution that satises the following criteria:
Non-intrusive: Driver behavior should not be affected
by the devices that are used to acquire the necessary
biomedical signals.
Comprehensive: The system should exploit the data
collected from a large number of heterogeneous sensors
and correlate the different readings.
Remotely Accessible: The collected measurements
should be transmitted to a remote location for analysis
and processing.
Real-time Enabled: The information must reach the
remote location within certain deadlines to enable real-
time decision-making.
User-friendly: Visualization tools should make it easy
for a human user to interpret the acquired temporal,
spatial and sensing information.
Our main contribution is a system architecture that meets
these criteria by combining wearable heart wave monitoring
technology, on-board sensing units and wireless networking
capabilities. The proposed system, which was implemented
in a real vehicle, aggregates data from a GPS receiver and an
OBD system. This information is correlated with the signals
2010 13th International IEEE
Annual Conference on Intelligent Transportation Systems
Madeira Island, Portugal, September 19-22, 2010
WA4.2
978-1-4244-7658-9/10/$26.00 2010 IEEE 1620
obtained from a special kind of garment that features an
embedded ElectroCardioGram (ECG) biosignal monitor. The
collected sensor data includes time, location, speed, fuel ef-
ciency, pedal position, temperature and drivers ECG. The in-
vehicle unit is capable of transmitting the data via wi access
points or third generation mobile communication systems.
By allowing non-intrusive, continuous monitoring of driver
behavior, the proposed system enables easier prototyping
and testing of on-board safety systems. As in [10], other
potential benets include the possibility of improving drivers
skills towards higher fuel efciency, stronger breaking safety
and more effective defensive driving. Some advanced meth-
ods of characterizing drivers behavior, such as time to
collision and time to line crossing, are not exploited in
this paper because they require information from additional
sensors not usually present in the vehicles, such as lateral
video-cameras, steering angle information, frontal radar or
distance sensors.
In Section II, we introduce the system architecture, includ-
ing the sensing capabilities, and address the main implemen-
tation issues. Preliminary results and analysis are presented
in Section III. In section IV we conclude the paper by
discussing a number of potential services and applications
for the proposed testbed.
II. SYSTEM ARCHITECTURE
The basic design of the proposed system, which can be
integrated in any vehicle, is represented in Fig. 1. It is
divided into three main physical blocks: the sensors, the in-
vehicle units and the central server. Data acquisition from
the sensors is performed by the in-vehicle units. These units
are also responsible for aggregating, scheduling the data and
Fig. 1. The main components of the proposed system architecture.
transmitting the more important data to the central server
through the network. The rest of the data processing is
performed remotely at the central server. Data visualization
is offered at the central server, or from any other place for
example via the Internet. In the following section, we explain
the functionalities and operation modes of the different
system components.
A. Wearable Technologies
Driver behavior in urban transportation systems is tra-
ditionally measured by means of travel logs and inquiries
or polls [5], [6]. However, wearable technologies are now
available that make it possible to monitor the heart wave
in a non-intrusive fashion. For estimating the human stress
or instantaneous emotional responses we use the so-called
VitalJacket R [11], [12], which consists of a smart t-shirt
designed for bio-monitoring (see Fig. 2). Tests show that it is
capable of continuously monitoring the ECG wave, returning
relevant information about the fatigue and stress levels of the
user.
Fig. 2. Smart t-shirt used to register ECG and monitor heart rate. Electrodes
are only felt during the initial minutes, after which the garment becomes
completely inconspicuous. [11], [12]
The textile fabric of the garment connects three light
electrodes (placed on the chest of the driver) with a small
electronic device (66 x 38 x 16 mm) located in a front
pocket. The device, which weights 50 g, broadcasts through
bluetooth not only the wearers heart rate but also the
complete heart wave. The device has been designed to have a
battery autonomy of up to 48 hours of continuous operation
(including wireless communication).
B. In-vehicle Sensors
The sensors perform the data gathering operations of the
system. The rst prototype consists of a GPS, an OBD and
an ECG recording device, which are connected via USB
or bluetooth to a netbook with a wireless interface. Our
system is able to gather information from a large variety of
in-vehicle sensors, through the standard OBD protocol. For
communication with these sensors we use a commercially
1621
available OBD-tool capable of gathering data from more than
50 different sensors. The tool connects to the computer via
USB and allows the user to query data from any sensor,
processing up to ten queries per second. Some of the data
gathered from the OBD connection are: in-vehicle air tem-
perature (

C), outside air temperature (

C), engine rotations

per minute (RPM), vehicle speed (m/s), instantaneous fuel
economy (/100km) and accelerator pedal position (%). The
data returned by these sensors is typically conveyed in 16 bit
codewords.
The GPS device is connected via USB or bluetooth and
is compatible with National Marine Electronics Association
(NMEA) specications. It returns information about latitude,
longitude, altitude, speed, road inclination and timestamp
with 32 bits of precision. The connection to the GPS is
controlled by an intermediate daemon called gpsd [13],
which can be queried at a socket connection for the GPS
related data.
C. Data Processing
The data from the sensors is gathered, stored and pro-
cessed by the in-vehicle units. It may also be transmitted in
real-time for further processing at the central server.
Synchronization
The data-mining module has to ensure the chronological
consistency of the data collected by different sensors. The
gpsd daemon is used to synchronize the system clock
between units with an estimated precision of only a few
milliseconds. When the daemon is used to retrieve location
information, the data can have a latency of up to one second,
and new data is usually available with a clock of 1 Hz.
At normal vehicle speeds 1 second of error can represent
more than 10 meters of location error, so the GPS time
and clock is used to synchronize the data from the other
sensors. The OBD protocol is based on query-response and
the ECG information is continuously saved in the bluetooth
driver buffer without time information. We use the system
time at the moment of the query or buffer-read operation to
synchronize the information from the OBD and VitalJacket
respectively.
Storage
After the synchronization process, the gathered data is
stored in a local relational database. For the rst experiments,
we used SQLite [14], since it stores the data in a single
le, simplifying data transfer from the netbook (prototype)
to another computer. The database structure is organized as
follows: a new table is created at every trip, and added one
new row for every second (sampling rate of 1 Hz), with a
xed number of columns, one for each type of information.
From Table I we can see that each new line will have
approximately 2000 bits of data, resulting in about 1 MB
of data per hour of driving. The database structure may
be changed to integrate new types of sensor data, such as
readings from accelerometers, number of detected bluetooth
devices and wi hotspots.
Communications
The unit classies the data according to its real-time
relevance and schedules the transmission to the central
server. The current scheduler algorithm is based on priorities
for each sensor, but it can be easily changed to allow for
other techniques, such as queries from the central server. As
showed in Table I, the bandwidth needed to transmit all the
information gathered is around 2 kbps. This requirement can
be lowered to around 328 bps by classifying the heart wave
as not real-time relevant. The transmission is encrypted in
order to ensure the privacy and integrity of the collected
information. The transmitted data is marked as sent, but it is
kept in the local database until a reliable communication can
be established and the data acknowledged. At user-dened
intervals, all the data is sent to the server or central entity,
for example through a high-speed wireless connection near
a stop. The databases are then merged in a complementary
fashion.
TABLE I
BANDWIDTH USED BY EACH TYPE OF SENSOR
System Sensors Size (bits) Bandwidth (bps)
Unit Time 32 32
GPS Lat, Lon, Alt, Speed, Incl 32 160
VJ Heart rate and wave 8 + 1600 8 or 1608
OBD 10 different sensors 16 160
Total bandwidth 360 or 1960 bps
The central server is also responsible for the data fusion,
analysis and visualization methods. The Keyhole Markup
Language (KML) Generator implemented in this entity is
vital for the visualization technique described in the next
subsection. Data fusion is carried out at the server offering
new insights by means of statistical analysis, as shall be
presented in section III.
D. Data Visualization
Another goal of our system is to provide a user-friendly
interface to visualize and analyze the gathered data. Google
Earth [15] was chosen to serve as the visualization platform
since it provides a straightforward way to overlay spatial
data and correlate different types of information. The Google
Earth GUI uses KML les, based on eXtensible Markup Lan-
guage (XML), to import placemarks and other information
that is easily visualized in four dimensions. We present the
data as small one second-distance line-segments where the
spatial information is the actual line location in the map. The
color and height (altitude) are used to represent two different
variables, such as heart rate and speed. The time toolbar (top-
left corner of Fig. 4) allows a time interval window to be
selected, in order to replay the data evolution over time.
In Fig. 3 we can observe the registered heart rate of a trip
over a highway bridge. The driver is heading south and his
heart rate increases when he approaches the highway exit
near the end of the bridge. Fig. 4 was taken from a closeup
of the same bridge, where two trips are visible. We can see
that in both datasets the drivers heart rate increased as he
approached the highway exit.
1622
Fig. 3. Overview of a trip over Freixo Bridge. Hot colors represent higher
values of heart rate while light colors represent lower values. [15]
Fig. 4. Closeup of the same bridge with two trips. Brightness represents
heart rate, height represents speed. [15]
Using the presented method of data visualization it is easy
to correlate high volumes of data. The user can select the
variables to be shown as well as the color, the height and the
time range he wishes to see. For example, visualizing data
from two months in a one year-long dataset is as easy as
moving two sliders on the time toolbar to set the beginning
and the ending date. That toolbar also has a play button
that allows the user to see a temporal animation of the
values, such as showing the average fuel consumption and
the outside air temperature on the streets along the various
months of the year.
In the central server the database is very large and with
many daily-recorded trips. This visualization method with
one line per trip is only t to analyze up to a few trips in
the same road. For analyzing many spatially-concurrent trips
the lines have to represent the time-average of the observed
values over the trips made during the prescribed time interval.
For example, the server can generate a KML le where the
color of the lines represent the average speed of the trips in
that road, during a chosen interval of time. This data fusion
method is harder to perform, since the KML Generator must,
for every road, search in the large database for the records of
the same road, organize them by timestamp, and compute the
average with a chosen time window. This way it is possible
to preserve the Google Earth temporal-animation capability.
However, it becomes hard to include or exclude more trips,
since the process must be restarted.
III. EXPERIMENTS AND RESULTS
Our preliminary experiments show that the proposed sys-
tem is well suited not only for characterizing driver behavior
but also for identifying and mapping potentially dangerous
road segments and intersections.
The proposed testbed is not designed to detect drivers
individual stress factors, such as favorite music playing on
the radio, a goal against the supported soccer team, or a
passenger who just did something unpredictable. However,
these factors can be analyzed with further processing of
the data in parallel with some control questionnaires of
psychological nature.
Random errors in the data sets caused by occasional and
time-limited sensor faults, driver stress induced by other
sources unrelated to driving or any other odd events become
progressively less signicant as the database grows larger (as
per the law of large numbers).
We deem the testbed data visualization to be mature
enough to detect a road or cross-road that causes emotional
responses in drivers in a non-random fashion (as seen in
Figs. 3 and 4), and to rank the best roads for example based
on fuel consumption, average speed, number of stops or
induced stress levels.
As a practical example, after one hour of measurements
and simple data fusion, we could already detect a strong
relationship between acceleration and stress. Fig. 5 shows a
graph that illustrates this relationship. To produce the graph,
for each second of the trip the heart rate variation (y-axis)
was calculated as the increase in heart rate in relation to the
average of the last 20 seconds. The vehicle acceleration (x-
axis) was calculated as the increase in speed on the last 5
seconds. The graph shows the histogram of the heart rate
variation across the different values of acceleration. Or in
other words, it plots the different values of acceleration
versus the average of the heart rate variation corresponding to
that acceleration value. Curve-tting indicates that those two
measures can be related approximately through a quadratic
polynomial. As common sense would suggest, breaks and
accelerations are stress inducing, - or, alternatively, stressful
events may cause a driver to break or accelerate - while, on
average, constant speed is more relaxing. With larger datasets
it is possible to draw some conclusions regarding this relation
to see if it holds for every type of road (streets or highways)
and for a larger universe of drivers.
Other results related to outside or in-vehicle temperatures,
air pollution and route durations can also be analyzed.
1623
Fig. 5. Average heart rate increase caused by different vehicle accelerations
IV. APPLICATIONS AND CONCLUSIONS
We presented a prototype for a driver monitoring system
that combines wearable non-intrusive heart wave monitors
with a multitude of in-vehicle sensors. Preliminary results
showed that a testbed with these characteristics can be very
useful for ITS in that it can assist in the characterization of
driver behavior as well as in the identication and mapping
of hazardous locations.
In addition, the system allows for easier prototyping and
testing of on-board security systems. Notice that large-
scale data gathering and data mining is also possible, if
we consider not one but a large number of vehicles that
gather data sets on a daily basis, which can be analyzed for
different purposes. Examples include real-time information
about the drivers condition, the state of the roads, trip
times, transportation delays, driving styles, congestion, fuel
consumptions and air pollution. Decision makers can exploit
the wealth of collected data for urban planning, risk assess-
ment, adaptive transportation scheduling or human resources
management. More specically, real-time data about driver
fatigue can be used by public transportation authorities to
improve job assignment and driver rotation schedules, thus
increasing the safety of the transportation system. Route
optimization based on fuel consumption or congestion is also
an example of a strategic decision that could be taken based
on the collected data.
In the future we plan on improving our prototype to
accommodate all the functionalities explained in this paper
and make it programmable over the air to avoid mainte-
nance overheads. We also plan on integrating more types of
sensors and devices, such as smart phones or PDAs. These
devices incorporate many different sensing capabilities, like
GPS, accelerometers, audio and video. The audio and video
recording of the driver is extremely useful in characterizing
driver behavior because it can allow the analysis of human
responses before and after a detected event. Such events can
be detected through abnormal responses from other sensors,
such as quick breaking or high bio-response. If the video
camera is pointed at the outside then further information can
be collected about the event. This allows for the estimation of
reaction time and awareness level of the driver by correlating
the recorded events with the responses detected by the on-
board systems and ECG device.
Trafc black spots like junctions, lane merge sections,
road works and others can be identied by statistically
abnormal high emotional responses (e.g. transient heart rate
variations) or trafc-related (e.g. sudden breaks) coupled
with the positioning information. This information can be
used to improve road signals, to increase drivers awareness
of dangerous places or to detect and prevent imminent
accidents. Other extensions that can be easily incorporated in
our testbed include human interfaces for alerting the driver
of his physical condition, dangerous events or inter-vehicle
warnings transmitted over VANETs [10] [2].
ACKNOWLEDGMENTS
The authors would like to thank Biodevices for granting
us some VitalJackets and for the help with its integration.
REFERENCES
[1] J. F. Pompei, S. Taly, S. J. Buckley, and J. Kemp, An automobile-
integrated system for assessing and reacting to driver cognitive load,
in Proceedings of Convergence, Detroit, Michigan, USA, Oct 2002,
pp. 411416.
[2] M. Nekovee, Sensor networks on the road: the promises and chal-
lenges of vehicular adhoc networks and vehicular grids, Edinburgh,
UK, May 2005.
[3] B. Hull, V. Bychkovsky, Y. Zhang, K. Chen, M. Goraczko, A. Miu,
E. Shih, H. Balakrishnan, and S. Madden, Cartel: a distributed
mobile sensor computing system, in SenSys 06: Proceedings of the
4th international conference on Embedded networked sensor systems.
New York, NY, USA: ACM, 2006, pp. 125 138.
[4] U. Lee and M. Gerla, A survey of urban vehicular sensing platforms,
Computer Networks, vol. 54, no. 4, pp. 527 544, 2010.
[5] G. Underwood, P. Chapman, S. Wright, and D. Crundall, Anger
while driving, Transportation Research Part F: Trafc Psychology
and Behaviour, vol. 2, no. 1, pp. 55 68, 1999.
[6] J. Healey and R. Picard, Detecting stress during real-world driving
tasks using physiological sensors, IEEE Transactions on Intelligent
Transportation Systems, vol. 6, no. 2, pp. 156 166, Jun 2005.
[7] R. Fuller, Towards a general theory of driver behaviour, Accident
Analysis & Prevention, vol. 37, no. 3, pp. 461 472, 2005.
[8] E. Simonson, C. Baker, N. Burns, C. Keiper, O. H. Schmitt, and
S. Stackhouse, Cardiovascular stress (electrocardiographic changes)
produced by driving an automobile, American Heart Journal, vol. 75,
no. 1, pp. 125 135, 1968.
[9] G. Rigas, C. Katsis, P. Bougia, and D. Fotiadis, A reasoning-
based framework for car drivers stress prediction, in Proceedings of
16th Mediterranean Conference on Control and Automation, Ajaccio,
Corsica, France, Jun 2008, pp. 627 632.
[10] P. Papadimitratos, A. De La Fortelle, K. Evenssen, R. Brignolo, and
S. Cosenza, Vehicular communication systems: enabling technolo-
gies, applications, and future outlook on intelligent transportation,
IEEE Comm. Mag., vol. 47, no. 11, pp. 84 95, Nov 2009.
[11] Biodevices. (2010, Mar) Vitaljacket
R
, a wearable ambulatory
ecg medical device. Aveiro, Portugal. [Online]. Available: http:
//www.vitaljacket.com
[12] J. P. S. Cunha, B. Cunha, A. S. Pereira, W. Xavier, N. Ferreira, and
L. A. Meireles, Vital jacket: A wearable wireless vital signs monitor
for patients mobility in cardiology and sports, in Proceedings of the
4th International ICST Conference on Pervasive Computing Technolo-
gies for Healthcare, Munchen, Germany, Mar 2010.
[13] Berlios. (2010, Mar) GPSD. [Online]. Available: http://gpsd.berlios.de
[14] SQLite. (2010, Mar). [Online]. Available: http://www.sqlite.org
[15] Google. (2010, Mar) Google Earth. [Online]. Available: http:
//earth.google.com
1624