IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 66, NO.

6, JUNE 2019 1769

Heart Rate Variability-Based Driver Drowsiness

Detection and Its Validation With EEG
Koichi Fujiwara , Member, IEEE, Erika Abe , Member, IEEE, Keisuke Kamata, Chikao Nakayama,
Yoko Suzuki, Toshitaka Yamakawa , Member, IEEE, Toshihiro Hiraoka , Member, IEEE,
Manabu Kano , Member, IEEE, Yukiyoshi Sumi, Fumi Masuda, Masahiro Matsuo, and Hiroshi Kadotani

Abstract—Objective: Driver drowsiness detection is a key According to a study by Gottlieb et al., the risk of traffic acci-
technology that can prevent fatal car accidents caused dent occurrence increases regardless of the drivers’ subjective
by drowsy driving. The present work proposes a driver sleepiness when they have sleep apnea or their sleep duration is
drowsiness detection algorithm based on heart rate vari-
ability (HRV) analysis and validates the proposed method by insufficient [2]. In order to prevent accidents caused by drowsy
comparing with electroencephalography (EEG)-based sleep driving, a driver-assistance system that detects drowsy driving
scoring. Methods: Changes in sleep condition affect the au- and provides a warning would be effective.
tonomic nervous system and then HRV, which is defined In sleep medicine, electroencephalography (EEG) recording
as an RR interval (RRI) fluctuation on an electrocardio- is necessary for sleep scoring because sleep onsets and sleep
gram trace. Eight HRV features are monitored for detecting
changes in HRV by using multivariate statistical process stages are defined based on EEG [3]. Although EEG-based
control, which is a well known anomaly detection method. drowsiness detection methods have been developed [4]–[7], it
Result: The performance of the proposed algorithm was is difficult to record EEG accurately during driving since EEG
evaluated through an experiment using a driving simula- recording is intolerant to motion artifacts and puts significant re-
tor. In this experiment, RRI data were measured from 34
strictions on the body. Thus, various types of driver drowsiness
participants during driving, and their sleep onsets were de-
termined based on the EEG data by a sleep specialist. The detection systems that do not use EEG have been developed [8].
validation result of the experimental data with the EEG data Driver face image analysis and vehicle travel data analysis are
showed that drowsiness was detected in 12 out of 13 pre- used for detecting driver drowsiness [9]–[14]; however, these
N1 episodes prior to the sleep onsets, and the false posi- methods require installing special devices in a vehicle, such as
tive rate was 1.7 times per hour. Conclusion: The present a camera for face image acquisition or a data logging device for
work also demonstrates the usefulness of the framework of
HRV-based anomaly detection that was originally proposed accessing vehicle travel data.
for epileptic seizure prediction. Significance: The proposed Instead of installing devices in a vehicle, physiological infor-
method can contribute to preventing accidents caused by mation other than EEG can be used for drowsiness detection if
drowsy driving. drivers agree to wear a sensor that measures their physiolog-
Index Terms—Drowsy driving detection, heart rate vari- ical signals. Changes in sleep condition affect the autonomic
ability analysis, electroencephalography, anomaly detec- nervous system (ANS) as well as cardiac activities [15], and
tion, multivariate statistical process control. cardiac signals can be used for drowsiness detection. Chui et al.
proposed a drowsiness detection method based on an electro-
I. INTRODUCTION
cardiogram (ECG) taken from drivers [16]. In addition, some
HE risk of traffic accidents in drowsy drivers is estimated
T to be four to six times higher than in awake drivers [1].
researchers have analyzed photoplethysmography (PPG) signals
for detecting drowsy driving [17]. Although they reported that
their proposed methods were able to achieve good performance,
Manuscript received May 9, 2018; revised September 7, 2018 and Oc- it is difficult to obtain good ECG or PPG signals stably due to
tober 21, 2018; accepted October 26, 2018. Date of publication Novem- motion artifacts. Besides, these methods would require a heavy
ber 2, 2018; date of current version May 20, 2019. This work was sup-
ported in part by the JST A-STEP under Grant 12103409, in part by the computational load because the sampling rate of ECG is usually
JSPS KAKENHI under Grant 17H00872, in part by the Hattori Hokokai more than several hundred Hz.
foundation, in part by the SECOM science and technology foundation, Heart rate variability (HRV), which is the RR interval (RRI)
and in part by the SEI Group CSR Foundation. (Corresponding author:
Koichi Fujiwara.) fluctuation in an ECG, is a well-known physiological phe-
K. Fujiwara is with the Department of Systems Science, Kyoto Univer- nomenon which reflects activities of ANS [18]. Fujiwara et al.
sity, Kyoto 606-8501, Japan (e-mail:, [email protected]). proposed an epileptic seizure prediction algorithm utilizing
E. Abe, K. Kamata, C. Nakayama, T. Hiraoka, and M. Kano are with
the Department of Systems Science, Kyoto University. HRV analysis [19]. HRV changes before an epileptic seizure
Y. Suzuki is with the Tokyo Medical and Dental University. because changes in cardiovascular regulation begin ten minutes
T. Yamakawa is with the Kumamoto University. to several seconds before seizure onsets [20], [21]. In addition,
Y. Sumi, F. Masuda, M. Matsuo, and H. Kadotani are with the Shiga
University of Medical Science. several studies have reported changes in HRV associated with
Digital Object Identifier 10.1109/TBME.2018.2879346 sleep stage transitions [22]–[24].

0018-9294 © 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution
requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
1770 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 66, NO. 6, JUNE 2019

In the present work, a new HRV-based driver drowsiness During N1, the muscles are still active, the eyes open and close
detection algorithm is proposed by utilizing the framework of moderately, and persons can be easily awakened by a sensory
HRV-based epileptic seizure prediction [19]. Abnormalities in stimulus. Thus, driver drowsiness should be detected prior to
the HRV data of drivers are monitored by multivariate statis- the N1 onset (sleep onset), when a driver can be easily wakened
tical process control (MSPC), which is a well-known anomaly by a stimulus.
detection algorithm used in manufacturing industries [25]–[27]. It is noteworthy that we cannot define a sleep onset with the
The proposed algorithm is simpler than previous HRV-based accuracy of less than 30 seconds because sleep scoring is based
methods [28]–[30] because the number of HRV features it uses on the 30-second EEG epoch-based method.
is just eight and MSPC is a linear method. Driving simulator
experiments were performed to validate the proposed method,
in which EEG-based sleep scoring by a sleep specialist was used B. Heart Rate Variability Analysis
as a reference. The R wave is the highest peak on an ECG, and the RR
interval (RRI) [ms] is defined as the interval between an R
II. RELATED WORKS wave and the next R wave. HRV is the fluctuation of RRI,
which is a physiological phenomenon reflecting ANS activities.
HRV-based drowsiness detection methods have been pro- Thus, HRV analysis has been used for monitoring stress, and
posed. Vicente et al. proposed a drowsiness detection method cardiovascular disease [33], [34].
that uses HRV analysis and linear discriminant analysis Although there are two types of HRV features–linear fea-
(LDA) [28]; however, the method uses ECG-derived respiratory tures and nonlinear features–this work uses the former, simply
information in addition to HRV, and ECG signal analysis is still because the extraction of nonlinear features requires a long-
needed. Although Li et al. proposed a drowsiness detection sys- term RRI measurement for stable calculation [35], which is not
tem that utilizes the support vector machine (SVM), their system appropriate for real-time applications like drowsy driving de-
utilizes driver face images as well as HRV [29]. A neural net- tection. The linear HRV features are classified into time domain
work (NN)-based drowsiness detection model was developed features and frequency domain features [18].
by Patel et al. which uses the power spectral density (PSD) The following time domain features can be calculated from
of RRI fluctuation as input variables of the NN model [30]. the original RRI data [18].
Their method would require a large amount of computational r MeanNN: Mean of RRI.
resources because the NN model is complicated and its number r SDNN: Standard deviation of RRI.
of input features is 900. A simple methodology for detecting r RMSSD: Root means square of the difference of adjacent
drowsy driving should be developed for realizing a wearable RRI.
drowsy driving detection system. r Total Power (TP): Variance of RRI.
r NN50: The number of pairs of adjacent RRI whose dif-
III. METHOD ference is more than 50 ms within a given length of mea-
Although EEG measurement is necessary for detecting sleep surement time.
onsets [3] in sleep medicine, it is difficult to measure EEG Frequency domain features cannot be extracted since the raw
during driving. The proposed algorithm adopts HRV instead of RRI data are not sampled at equal intervals. Thus, the raw RRI
EEG, and EEG-based sleep scoring is used as the reference for data are interpolated by using spline and resampled at equal
the proposed algorithm. This section explains EEG-based sleep intervals. The following frequency domain features can be ob-
scoring and HRV briefly and proposes an HRV-based drowsiness tained from the power spectrum density (PSD) of the resampled
detection algorithm. RRI data, and the PSD can be calculated by using Fourier anal-
ysis or an autoregressive (AR) model [18].
r LF: Power of the low-frequency band (0.04 Hz–0.15 Hz)
A. EEG-Based Sleep Scoring in a PSD. LF reflects both the sympathetic and parasym-
Sleep consists of REM (rapid eye movement sleep) and pathetic nervous system activities.
NREM (non-REM sleep), which is categorized into three lev- r HF: Power of the high-frequency band (0.15 Hz–0.4 Hz)
els: N1, N2, and N3 [3]. N1 is also called transitional sleep or in a PSD. HF reflects the parasympathetic nervous system
light sleep. According to the sleep scoring manual [31], sleep activity.
stages are discriminated based on the 30-second epoch-based r LF/HF: Ratio of LF to HF. LF/HF expresses the balance
EEG scoring method. The N1 onset (sleep onset) is defined by between the sympathetic nervous system activity with the
the epoch in which α wave (8–13 Hz) activity is attenuated parasympathetic nervous system activity.
and replaced by low-amplitude, mixed-frequency activities that The guideline recommends that RRI is measured for two to
occupy more than 50% of the epoch. five minutes for HRV analysis, and the sampling rate of ECG
Drivers may feel drowsiness shortly before N1, which causes should be more than 200 Hz for precise R wave detection [18].
mild cognitive dysfunction, and some researchers have at- A precise RRI sensor is needed in order to realize an HRV-
tempted drowsy EEG identification [32]. On the other hand, based drowsy driving detection system. Although the Holter
falling asleep directly contributes to traffic accidents. N1 usu- monitor is generally used for measuring ECG outside hospi-
ally occurs between wakefulness and deeper sleep stages. tals, its use in daily life is difficult since the Holter monitor
FUJIWARA et al.: HEART RATE VARIABILITY-BASED DRIVER DROWSINESS DETECTION AND ITS VALIDATION WITH EEG 1771

requires operation skills. Many types of wearable devices such

Algorithm 1: Drowsy Detection Preparation.
as smartwatches have PPG sensors that can also be used for
pulse detection; however, it is notably difficult for PPG to de- 1: for all i such that 1 ≤ i ≤ I do
{i}
rive RRI precisely enough to carry out HRV analysis [36]. 2: Extract the ith driver awake HRV feature X̃ from
A wearable RRI sensor developed by Yamakawa et al. eas- the ith driver awake RRI data y {i} .
ily measures accurate RRI based on ECG. If an HRV-based 3: end for
{1} {I }
drowsy driving detection algorithm can be implemented in such 4: Merge matrixes X̃ , . . . , X̃ into one matrix X̃.
a device, a wearable drowsy driving detection system can be 5: Preprocess X̃, which is referred to as X.
realized. 6: Derive ΣR and V R from X as Eq. (1)
Changes in HRV associated with sleep stage transitions have 7: for all i such that 1 ≤ i ≤ I do
been reported [22]–[24]. Bonnet and Arand reported that heart 8: Define the control limits of the T 2 and Q statistics for
rates vary depending on sleep latency [38]. Chua et al. showed the ith driver, T̄ 2{i} and Q̄{i} .
that there is a correlation between changes in subjective sleepi- 9: end for
ness and HRV through sleep deprivation experiments [39]. Since
HRV alteration begins prior to a sleep onset [23], drowsy driving
may be detected by monitoring HRV. proposed method, eight HRV features are adopted as input vari-
ables. The extracted HRV features are merged into one matrix
in Step 4. Then, in Step 5, the merged matrix X̃ is preprocessed
C. Drowsy Driving Detection
for model construction. There are various preprocessing meth-
In drowsy driving detection, the awake data and the drowsy ods, and thus an appropriate method should be chosen by taking
data are regarded as normal data and anomalous data, respec- account of the characteristics of the problem and the data. In
tively. To build an accurate discriminant model by using both this work, each column of X̃ is standardized so that each HRV
the awake data and the drowsy data, a sufficient amount of feature has zero mean and unit variance. In Step 6, the singular
drowsy data needs to be collected from drivers. However, in value matrix ΣR and the loading matrix V R are derived from
practice, collecting such drowsy data is more difficult than the the preprocessed HRV feature matrix X̃. In other words, PCA
awake data. Thus, drowsy driving detection is formulated as an is applied to X̃, and the correlations among eight HRV features
anomaly detection problem, in which a model is developed from are modeled. In this step, the number of principal components
the awake data only. R has to be selected appropriately to realize precise drowsiness
Fujiwara et al. developed an epileptic seizure prediction algo- detection. The next step is to calculate the T 2 and Q statistics
rithm based on multivariate statistical process control (MSPC) and to define their control limits.
to detect abnormalities in HRV [19]. MSPC detects a sample There is considerable individual variability in HRV. Changes
that does not follow the major trend in the modeling data as an in HRV are different for every person, which changes with age,
anomaly based on principal component analysis (PCA), which and the variation of the T 2 and Q statistics is also different for
has been widely used as fault detection and identification tech- every person. Hence, the control limits have to be determined for
nique in multivariate processes [25]–[27]. Since drowsy driving each driver in Steps 7–9. The control limits can be determined
detection is a similar problem to epileptic seizure prediction, we as the α% confidence of each driver.
use MSPC. Before driver drowsiness monitoring starts, the initial RRI
The proposed algorithm discriminates between driver statuses data of a driver have to be stored for more than the window
of ‘awake’ and ‘drowsy,’ where ‘drowsy’ means that the driver size W to calculate HRV features. After the initial RRI data
is close to or in N1. In the proposed method, eight HRV features collection, driver drowsiness can be monitored by following
described in Section III-B are used and their abnormalities by Algorithm 2. y[t] ∈  denotes the tth RRI and t is the number of
sleepiness are monitored using MSPC. The detail of MSPC is sampling from the monitoring start. τ is a time counter variable,
explained in the Appendix. and C denotes the binary driver status C = {A, D} where A
For HRV feature extraction, a rectangular moving window and D are ‘awake’ and ‘drowsy,’ respectively. That is, ¬A =
whose window size is three minutes is used. Li et al. compared D and vice versa. In Step 5, the extracted HRV feature x̃ is
the one-minute and the three-minutes windows in HRV extrac- preprocessed in the same manner in Algorithm 1.
tion, and they reported that the latter was better for drowsiness To realize accurate drowsy driving detection, it is crucial to
detection [29]. Time domain features are extracted from the raw decrease false positives. In drowsy driving detection, false pos-
RRI data. For frequency domain feature extraction, the raw RRI itives are mainly caused by ECG artifacts, which significantly
data need to be arranged at equal intervals. The raw RRI data are affect the T 2 and Q statistics. Hence, the driver status is de-
interpolated by using the third-order spline, and the interpolated termined as ‘drowsy’ only when either the T 2 or Q statistic
RRI data are resampled at 4 Hz. An AR model of order 40 is continuously exceeds its control limit for more than the prede-
used to calculate the PSD [19]. fined period τ̄ . Conversely, to change the status from ‘drowsy’
The proposed drowsy driving detection algorithm is described to ‘awake,’ both statistics have to continuously stay below their
in Algorithm 1, in which y {i} is the awake RRI data recorded control limits for more than τ̄ . In Steps 7–14, the driver sta-
from the ith driver and I is the number of drivers. First, awake tus is discriminated. A warning is given to the driver when the
HRV features are extracted from y {i} in Steps 1–3. In the algorithm detects drowsiness, that is, C = D.
1772 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 66, NO. 6, JUNE 2019

Algorithm 2: Drowsy Driving Detection.

1: set τ [0] ←− 0, C[0] ←− A.
2: while do
3: Collect the newly measured tth RRI y[t].
4: Extract HRV feature x̃[t].
5: Preprocess x̃[t], which is denoted as x[t].
6: Calculate the tth T 2 and Q statistics, T 2 [t] and Q[t]
from x[t] by using (4) and (3).
7: if ((T 2 [t] > T̄ 2 ∨ Q[t] > Q̄) ∧ (C[t − 1] = A)) Fig. 1. Electrode allocations for sleep scoring: EEG (left) and
EOG (left).
∨((T 2 [t] ≤ T̄ 2 ∧ Q[t] ≤ Q̄) ∧ (C[t − 1] = D))
then
8: τ [t] = τ [t − 1] + y[t]. In this work, participants drove a driving simulator con-
9: else structed based on a commercial racing simulator (GRAN
10: τ [t] = 0. TURISMO 5, Sony Interactive Entertainment Inc.), which was
11: end if used in some researches of driving physiology [40]–[42]. The
12: if τ [t] ≥ τ̄ then simulator equips with an LCD display, a steering, an accelerator,
13: C[t] = ¬ C[t − 1] and τ [t] = 0. and a brake pedal. Thus, it can simulate driving operation.
14: end if Before experiments, we confirmed that EEG during driving
15: Wait until the next RRI data y[t + 1] is measured. can be measured precisely enough for sleep scoring in this
16: end while simulator.
The participants drove twice on a course that simulated a
nighttime, monotonous highway loop line for 1.5 hours in a dark
D. Participants room, resting and taking lunch for an hour between the two tri-
The inclusion criteria for the participants were non- als. There were no other vehicles and it took about ten minutes
professional drivers, with a valid driving license. The exclusion to cycle the loop line at 80 kilometers per hour. The first and
criterion was having a chronic illness that may affect HRV such second trials started from around 11 am and after lunch, respec-
as cardiovascular disease, arrhythmia, epilepsy, or sleep disor- tively. This setting was determined in consideration of avoiding
ders. These experiments and analyses were approved by the Re- participants’ excessive fatigue due to extended experiments as
search Ethics Committee of the Graduate School of Science and well as with the expectation that the participants may become
Technology, Kumamoto University. Written informed consent drowsy.
was obtained from each participant prior to the experiments. The EEG data during driving were recorded for sleep scor-
ing using a digital EEG recording system (Grapevine, Ripple),
whose sampling frequency was 1,000 Hz. Although the Inter-
IV. RESULT national 10–20 system is a standard scalp electrode allocation
This section evaluates the performance of the proposed of EEG recording, the number of electrodes was reduced in this
drowsy driving detection algorithm through an application of work with reference to polysomnography (PSG) tests performed
real RRI data obtained from an experiment using a driving sim- in sleep laboratories. Fig. 1 (left) shows the adopted electrode
ulator. allocation, in which Fp1, Fp2, C3, C4, O1, O2 were EEG elec-
trodes and earlobes A1 and A2 were for reference. This electrode
allocation is enough for sleep scoring. Electrooculogram (EOG)
A. Data Collection
was also recorded during driving for making sleep scoring easy.
The RRI data and the EEG data were collected from experi- The EOG electrode allocation is shown in Fig. 1 (right) and a left
ment participants (drivers) while they drove a virtual vehicle on earlobe A1 was for reference. The RRI data were obtained for
a simulator. Participants A, . . ., Z, α, . . ., θ, 25 males and nine HRV analysis by using a wearable RRI sensor [37]. In addition,
females aged 18–36 years (mean 22.7 years) participated in this participant video during driving was recorded for confirming
experiment. In order to avoid effects on HRV, participants were participant behaviors after the experiments.
instructed not to take alcohol, caffeine or smoke for one night Since artifacts were generated when participants moved dur-
before the experiments. Before the experiment, the participants’ ing driving, data in which either the RRI data or the EEG
driving careers, health statuses, mediations, and sleep habits data were contaminated with strong artifacts were eliminated
were checked by means of a questionnaire. The questionnaire before analysis. A sleep specialist certified by the Japanese
collects age, sex, weight, height, occupation, medical history Society of Sleep Research determined sleep onsets of partic-
(hypertension, diabetes, cardiovascular disease, epilepsy, etc.), ipants by visual check of the EEG data based on the 30-second
meditation, and habits about breakfast, exercise, sleep, caffeine, epoch-based scoring method recommended in the sleep scoring
and smoke. In addition, we asked the sleep time of the previous manual [31].
night. As a result, all participants were healthy and took enough As a result of sleep scoring, 12 participants were scored in
sleep (>6 hours), and there were no rejected participants. N1 during driving. The data 15 minutes before and 5 minutes
FUJIWARA et al.: HEART RATE VARIABILITY-BASED DRIVER DROWSINESS DETECTION AND ITS VALIDATION WITH EEG 1773

TABLE I
PARTICIPANT DEMOGRAPHICS AND COLLECTED EPISODES

Fig. 2. RRI data of L2 (top) and Ld1 (bottom). Fig. 3. RRI data of N2 (top) and Nd2 (bottom).

after sleep onset were stored as pre-N1 episodes, following the

report by Jurysta et al. on sleep stage transition that cardiac
activities precede 9–20 minutes (mean 12 minutes) before EEG
changes [23]. In addition, the data that were not scored as sleep
were clipped as awake episodes.
Consequently, we collected 13 pre-N1 episodes named Ad1,
Bd1, . . ., Zd1 and β d1, and 91 awake episodes named A1,
A2, . . ., θ 1, and θ 2, which are shown in Table I. Their total
lengths of awake and pre-N1 episodes were 66.8 and 4.3 hours,
respectively.
Fig. 4. HRV features derived from L2.
B. RRI Data and HRV Features
The raw RRI data of participants L and M in awake and
drowsy periods are shown in Figs. 2 and 3. In these figures, an C. Drowsy Detection Preparation
orange colored band denotes the N1 epoch. Eight HRV features Driver drowsiness detection was prepared according to
described in Section III-B were extracted. Figs. 4–7 are the Algorithm 1. Table II shows 34 awake episodes used for mod-
HRV features extracted from the RRI data shown in Figs. 2 eling. The total recorded length of the analyzed episodes was
and 3. Although frequency domain features seemed to change 26.7 hours.
before sleep onset in Figs. 5 and 7, a similar fluctuation is also All HRV features calculated in Section IV-B were used as
observed in the awake HRV features in Figs. 4 and 7. inputs. In MSPC, the number of retained principal components
These results show that it is difficult to detect drowsiness by R was determined so that the cumulative proportion reached
monitoring changes in respective HRV features, and suggest that more than 90%, and R = 3. The control limits of the T 2 and Q
multiple HRV features should be monitored simultaneously. statistics were defined for each participant so that they represent
1774 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 66, NO. 6, JUNE 2019

Fig. 8. Detection result of pre-N1 episode Ld1 (orange band: N1 epoch,

green band: detected drowsy period).
Fig. 5. HRV features derived from Ld1.

Fig. 9. Detection result of pre-N1 episode Nd2.

Fig. 6. HRV features derived from N2. the 90% confidence limits. Although the 99% or the 95% confi-
dence limits are usually adopted in MSPC for suppressing false
positives, this research used the 90% confidence limits because
it is important to prevent erroneous drowsiness detection from
the viewpoint of safety. The parameter τ̄ was determined as ten
seconds according to [19].

D. Drowsy Driving Detection Results

All of the drowsy and awake episodes that were not used
for modeling were monitored by Algorithm 2. The numbers of
validated awake and pre-N1 episodes were 57 and 13, respec-
tively. The total length of the validated awake episodes was 40.1
hours. Here, drowsiness detection success means that drowsi-
Fig. 7. HRV features derived from Nd2. ness is detected from 15 minutes before to just before a sleep
onset.
TABLE II From the pre-N1 episode results, the Q statistic detected 12
AWAKE EPISODES USED FOR MODELING out of 13 pre-N1 episodes excluding episode Bd1. On the other
hand, the T 2 statistic detected 8 out of 13 pre-N1 episodes
excluding episodes Ad1, Ld1, Nd2, Pd1, and Zd1. As a result,
the sensitivity of the T 2 and Q statistics are 62% and 92%,
and the mean and the standard deviation of the first drowsiness
detection time by the T 2 and Q statistics were 484 ± 383 and
642 ± 401 seconds before sleep onsets, respectively.
Detection results of pre-N1 episodes Ld1 and Nd2 are shown
in Figs. 8 and 9, in which horizontal dashed lines express the con-
trol limits of the T 2 and Q statistics. Orange and green colored
bands denote the N1 epochs scored by the sleep specialist and
drowsy periods detected by the proposed method, respectively.
According to Algorithm 2, driver drowsiness is detected only
when either T 2 or Q statistic exceeds its control limit continu-
ously for more than τ̄ = 10 seconds. Although the T 2 statistic
FUJIWARA et al.: HEART RATE VARIABILITY-BASED DRIVER DROWSINESS DETECTION AND ITS VALIDATION WITH EEG 1775

TABLE III
FALSE POSITIVES

Fig. 10. Detection result of awake episode L2.

Fig. 11. Detection result of awake episode N2.

around 900 seconds in Fig. 8 exceeded its control limit; it was

not detected as drowsiness because it did not exceed its control
limit for more than ten seconds continuously. Figs. 10 and 11
show detection results of awake episodes L2 and N2. There were
no false positives in episode L2 while a false positive occurred
according to the T 2 statistic in episode N2.
Table III summarizes the number of false positives (#FP) and
false positive (FP) rates, which are defined as #FP per hour. The
#FP in all awake periods (total 40.1 hours) by the T 2 and Q
which was performed after lunch. Although all participants in
statistics were 105 and 70, and the FP rates were 2.6 and 1.7
the experiment were healthy, sleep loss and daytime sleepiness
times per hour, respectively.
in the Japanese adult population are common [43]. Thus, 1.5
hours of driving after lunch may have been sufficient to induce
V. DISCUSSION drowsiness for some participants.
In the experiment, we collected HRV data from a total of 34 α waves, which indicate sleep-related brain activities, were
participants, and drowsiness detection was based on the T 2 and observed in correspondence with the increase of the Q statistic.
Q statistics. In order to confirm the validity of these statistics, Fig. 12 shows α waves recorded during the awake period, which
sample powers were calculated. The sample powers of the T 2 corresponded to a false positive by the Q statistic in episode
and Q statistics were 0.09 and 0.71, respectively. Thus, the X6. According to EOG of this period, the participant repeatedly
sample size was not insufficient for the T 2 statistic; however, blinked. Such awake α waves associated with the increase of the
the sample size was adequate when the Q statistic was used for Q statistic were observed in episodes J2, M2, P2, δ4 as well as
drowsiness detection. In fact, the drowsiness detection of the Q X6. These correspondences between changes in the Q statistic
statistics was higher than the T 2 statistic in the experiment. and α wave appearance support the validity of the proposed
The collected data in the experiment consist of a total of 66.8 method.
hours of awake episodes and a total of 4.3 hours of pre-N1 Microsleep, which is sudden short sleep lasting for a fraction
episodes. This unbalanced ratio of awake to pre-N1 episodes of a second or up to 30 seconds, is a well-known phenomenon
justifies the adoption of the anomaly detection framework of in sleep science [44]. Persons with microsleep often remain un-
MSPC, in which a drowsiness detection model is developed aware of it. Its causes are, for example, sleep apnea syndrome,
from the awake episodes only. narcolepsy, mental fatigue, and insufficient sleep. Many mi-
One driving trial was limited to 1.5 hours, and there was crosleep identification methods have been proposed, and there
a one-hour rest including lunch between two trials, in consid- is little agreement on which is the best for use at this time [44].
eration of the burden of the participants. As a result, 12 out It is possible that the awake α wave appearance associated with
of 34 participants were scored by N1 during driving. Ten out an increase of the Q statistic was microsleep, although this is
of thirteen pre-N1 episodes were observed in the second trial, difficult to confirm.
1776 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 66, NO. 6, JUNE 2019

Fig. 12. The awake α waves associated with a false positive by the Q statistic in episode X6.

The occurrence of microsleep may be dangerous particularly useful for drowsiness detection, although it cannot be used for
in situations that demand constant alertness, such as driving sleep stage scoring.
or working with heavy machinery. Microsleep decreases the The sensitivity of the proposed method achieved almost the
capability for task execution, which is equivalent to a mild cog- same level as conventional camera-based drowsiness detection
nitive dysfunction; however, falling asleep directly contributes methods [8]; however, most of the latter use facial expression
to the occurrence of traffic accidents, since N1 sleep is equiv- evaluation by referees or subjective evaluation by questionnaires
alent to complete cognitive dysfunction. In fact, the multiple instead of EEG-based sleep scoring for driver sleepiness eval-
sleep latency test (MSLT), which measures the elapsed time uation. Because it is impossible to detect precise N1 onsets,
from wakefulness to N1 onset, is correlated to the risk of traffic microsleep, and awake α waves by facial expression evalua-
accidents [45], [46]. Thus, the target of the proposed algorithm tion and questionnaires, there is a possibility that previous re-
is the N1 onset instead of microsleep. Future studies are needed searchers overlooked such sleep-related phenomena.
to evaluate whether the proposed methodology for detecting the It is concluded that the proposed HRV-based drowsy driving
N1 onset is also applicable to microsleep detection. detection algorithm is more promising than other conventional
According to an evaluation of the EEG data and videos, most methods with respect to accuracy as well as practical use.
false positives occurred in correspondence with participant’s
motion. The drowsy driving detection model was constructed by
using HRV data collected during driving in which participants VI. CONCLUSION AND FUTURE WORK
sat in a seat and rarely moved. Because few HRV data with A driver drowsiness detection method was proposed utilizing
body motion were contained in the modeling data, fluctuations the framework of epileptic seizure prediction, by which multi-
of HRV caused by body motion in the validation data were ple HRV features are extracted from the RRI data and MSPC
detected as false positives. For example, there were eight false monitors abnormalities in the extracted HRV features. The ex-
positives of the T 2 statistic in participant F, seven of which perimental result showed that 12 out of 13 pre-N1 episodes were
occurred when his EEG was contaminated with electromyogram detected prior to sleep onsets, and the false positive rate was
(EMG) artifacts caused by body motion. about 1.7 times per hour. The experimental result was discussed
Xaio et al. developed an HRV-based sleep stage scoring from the viewpoint of sleep science. This work demonstrated
method using random forest (RF), which is an ensemble learn- the usefulness of the framework of HRV-based anomaly detec-
ing technique using multiple decision trees [47]. Their method tion because it can be applied to driver drowsiness detection as
classifies sleep condition into awake, REM, and NREM and well as epileptic seizure prediction.
the accuracy of their method was 72%–88%, even though it Limitations of the study include the properties of the col-
uses a total of 41 HRV features consisting of nonlinear fea- lected experimental data, such as a highly controlled laboratory
tures, time domain features, and frequency domain features as environment, the limited number of participants, and the fact
input variables [48]. It is difficult to compare their sleep stage that all participants were young Japanese persons. Accordingly,
scoring method with the proposed drowsiness detection algo- more studies are required to confirm our results by using well-
rithm from the viewpoint of performance, because the purpose matched groups of participants in a real driving environment.
of their method is to discriminate sleep stages, while that of the The proposed method requires drivers to put some elec-
proposed method is to detect drowsiness prior to the N1 onset; trodes on the skin before driving, because precise RRI mea-
however, the proposed drowsiness detection method is much surement based on ECG is needed for HRV analysis. Since it
simpler than their method since the proposed algorithm uses is burdensome for drivers to attach electrodes before driving, a
only eight HRV features and a linear model. This indicates that new type of electrode that is easy to use should be developed.
the framework of HRV-based anomaly detection by MSPC is Tsukada et al. developed a new wearable textile electrode using a
FUJIWARA et al.: HEART RATE VARIABILITY-BASED DRIVER DROWSINESS DETECTION AND ITS VALIDATION WITH EEG 1777

should be taken into account for detecting anomalies that do not

follow the major trend in the data.
In MSPC, the correlation among variables is modeled by
using principal component analysis (PCA) [50], which finds
linear combinations of variables that describe major trends in
a dataset as shown in Fig. 13(b). Let us assume a normal data
matrix X ∈ N ×M whose nth row is the nth sample xn ∈ M ,
wherein samples are mean-centered and scaled appropriately,
and M and N denote the number of variables and samples,
respectively. Such a matrix X can be factorized by singular
value decomposition (SVD) as follows:
X = U ΣV T
 
  ΣR 0  T
= UR U0 VR V0 (1)
0 Σ0
where U , Σ, and V are the left singular matrix, the diagonal
matrix whose diagonal elements are singular values, and the
right singular matrix, respectively. Using Eq. (1), the matrix
Fig. 13. Schematic diagram of MSPC.
factorization of X by PCA is defined as:
X = T R V TR + E (2)
conductive fiber [49], and a smart shirt woven with textile elec-
trodes has been developed for ECG measurement. Therefore, it where T R ∈ N ×R is the score matrix and T R ≡ U Σ. E
will be easy for drivers to use the proposed HRV-based drowsy ∈ N ×M is an error matrix. The column of V R spans the
driving detection method when the smart shirt becomes avail- subspace Π which expresses the correlation among variables
able. In addition, the proposed algorithm can be easily imple- as shown in Fig. 13(c). R(≤ M ) is the number of principal
mented into mobile computers, such as a smartphone, since the components retained in the PCA model.
computational load of the proposed method is much lighter than The T 2 statistic is used for monitoring anomalies in Π,
the camera-based methods that use real-time video analysis. which is defined as follows:
In future works, additional experimental data must be col- R
 t2r
lected to improve the drowsiness detection performance, and T2 =
the system under development will be tested in a real driving σ2
r =1 t r
environment.
= xT V R Σ−2 T
R V Rx (3)
APPENDIX where σt r denotes the standard deviation of the rth score tr and
MULTIVARIATE STATISTICAL PROCESS CONTROL (MSPC) x is a newly measured sample. Since the T 2 statistic is the Maha-
The appendix explains multivariate statistical process control lanobis distance, which is defined as the distance normalized by
(MSPC) used in the drowsiness detection algorithm. the standard deviations of the scores, it defines a circular nomi-
The proposed drowsiness detection algorithm described in nal region as shown in Fig. 13(d). Hence, the sample is close to
Section III-C detects driver drowsiness as anomalies in HRV. the mean of the modeling data when the T 2 statistic is small.
The simplest way of detecting anomalies is to check whether The Q statistic is defined as the squared distance between the
or not all variables are within their upper and lower bounds. sample and Π as
This simple method is called univariate statistical process con- M
 M

trol (USPC); it is also known as control charts and Shewhart Q= e2m = (xm − x̂m )2
charts. When multiple variables are monitored simultaneously, m =1 m =1
the nominal region of USPC becomes rectangular as shown in = x (I − V R V TR )x.
T
(4)
Fig. 13(a). USPC is intuitive and easy-to-use, and therefore has
been widely used in various fields including the manufacturing The normal operating condition (NOC) is defined using the
industry. However, it cannot detect an anomaly that does not T 2 and the Q statistics [51]. Fig. 13(c) shows the image of NOC
satisfy normal correlation among variables. In Fig. 13(a), for in MSPC. Since the monitored subspaces by two statistics are
example, USPC cannot detect the anomaly , which does not orthogonal to each other, NOC can be considered as a cylinder
follow a positive correlation between variables 1 and 2, because and the control limits of the T 2 and Q statistics correspond to
it is located within the normal rectangular area of USPC. If the its diameter and height, respectively. MSPC usually detects an
normal ellipsoid area defined by the dashed line is used instead anomaly when either the T 2 or Q statistic exceeds the prede-
of the normal rectangular area, the anomaly can be detected. fined control limit. Thus, the control limits of the T 2 and Q
This example demonstrates that the correlation among variables statistics have to be determined carefully, which can be set as
1778 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 66, NO. 6, JUNE 2019

α% confidence limits, and usually, the 99% or 95% confidence [24] E. Tobaldini et al., “Heart rate variability in normal and pathological
limits are adopted. sleep,” Front. Physiol., vol. 4, 2013, Art. no. 294.
[25] P. Nomikos and J. F. MacGregor, “Monitoring batch processes using mul-
In order to detect anomalies by MSPC, the number of prin- tiway principal component analysis,” AIChE J., vol. 40, no. 8, pp. 1361–
cipal components R also has to be appropriately determined. 1375, 1994.
The number of principal components R can be determined so [26] J. F. MacGregor and T. Kourti, “Statistical process control of multivariate
processes,” Control Eng. Pract., vol. 3, no. 3, pp. 403–414, 1995.
that the cumulative proportion of principal components reaches [27] M. Kano et al., “Comparison of multivariate statistical process monitoring
a predefined value, such as 80% or 90%. methods with applications to the eastman challenge problem,” Comput.
Chem. Eng., vol. 26, no. 2, pp. 161–174, 2002.
[28] J. Vicente et al., “Drowsiness detection using heart rate variability,” Med.
REFERENCES Biol. Eng. Comput., vol. 54, no. 6, pp. 927–937, 2016.
[29] G. Li and W.-Y. Chung, “Detection of driver drowsiness using wavelet
[1] S. G. Klauer et al., “The impact of driver inattention on near-crash/crash
analysis of heart rate variability and a support vector machine classifier,”
risk: An analysis using the 100-car naturalistic driving study data,”
Sensor, vol. 13, no. 12, pp. 16 494–16 511, 2013.
National Highway Traffic Safety Administration, 2006.
[30] M. Patel et al., “Applying neural network analysis on heart rate variability
[2] D. J. Gottlieb et al., “Sleep deficiency and motor vehicle crash risk in
data to assess driver fatigue,” Expert. Syst. Appl., vol. 38, no. 6, pp. 7235–
the general population: a prospective cohort study,” BMC Med., vol. 16,
7242, 2011.
no. 1, 2018, Art. no. 44.
[31] The AASM Manual for the Scoring of Sleep and Associated Events: Rules,
[3] M. H. Kryger et al., Principles and Practice of Sleep Medicine. 6th ed.,
Terminology and Technical Specifications, 2nd ed., Darien, IL, USA:
New York, NY, USA: Elsevier, 2017.
American Academy of Sleep Medicine, 2016.
[4] H. Su and G. Zheng, “A partial least squares regression-based fusion
[32] J. Santamaria and K. H. Chiappa, The Eeg of Drowsiness. New York, NY,
model for predicting the trend in drowsiness,” IEEE Trans. Syst., Man,
USA: Demos Medical Pub, 1987.
Cybern. A, Syst. Humans, vol. 38, no. 5, pp. 1085–1092, 2008.
[33] R. E. Kleiger et al., “Decreased heart rate variability and its association
[5] C.-T. Lin et al., “A real-time wireless brain–computer interface system for
with increased mortality after acute myocardial infarction,” Am. J. Car-
drowsiness detection,” IEEE Trans. Biomed. Circuits Syst., vol. 4, no. 4,
diol., vol. 59, no. 4, pp. 256–262, 1987.
pp. 214–222, Aug. 2010.
[34] A. Malliani, “Cardiovascular neural regulation explored in the frequency
[6] F.-C. Lin et al., “Generalized EEG-based drowsiness prediction system by
domain,” Circulation, vol. 84, no. 2, pp. 482–492, 1991.
using a self-organizing neural fuzzy system,” IEEE Trans. Circuits Syst.
[35] J. M. Yentes et al., “The appropriate use of approximate entropy and
I, Reg. Papers, vol. 59, no. 9, pp. 2044–2055, Sep. 2012.
sample entropy with short data sets,” Ann. Biomed. Eng., vol. 41, no. 2,
[7] A. Hashemi et al., “Time driver’s drowsiness detection by processing the
pp. 349–365, 2013.
eeg signals stimulated with external flickering light,” Basic Clin. Neurosci.,
[36] G. Lu and F. Yang, “Limitations of oximetry to measure heart rate vari-
vol. 5, no. 1, pp. 22–27, 2014.
ability measures,” Cardiovasc. Eng., vol. 9, no. 3, pp. 119–125, 2009.
[8] S. Kaplan, “Driver behavior analysis for safe driving: A survey,” IEEE
[37] T. Yamakawa et al., “Accuracy comparison of two microcontroller-
Trans. Intell. Transp. Syst., vol. 16, no. 6, pp. 3017–3032, Dec. 2015.
embedded r-wave detection methods for heart-rate variability analysis,” in
[9] R. O. Mbouna et al., “Visual analysis of eye state and head pose for driver
Proc. APSIPA ASC, 2015, pp. 1010–1013.
alertness monitoring,” IEEE Trans. Intell. Transp. Syst., vol. 14, no. 3,
[38] M. H. Bonnet and D. L. Arand, “Activity, arousal, and the MSLT in
pp. 1462–1469, Sep. 2013.
patients with insomnia,” Sleep, vol. 23, no. 2, pp. 205–212, 2000.
[10] J. Jo et al., “Detecting driver drowsiness using feature-level fusion and
[39] E. C. Chua et al., “Heart rate variability can be used to estimate sleepiness-
user-specific classification,” Expert Syst. Appl., vol. 41, no. 4, pp. 1139–
related decrements in psychomotor vigilance during total sleep depriva-
1152, 2014.
tion,” Sleep, vol. 35, no. 3, pp. 325–334, 2012.
[11] B. Cyganek and S. Gruszczynski, “Hybrid computer vision system
[40] M. Ohsuga et al., “Estimation of driver’s arousal state using multi-
for drivers’ eye recognition and fatigue monitoring,” Neurocomputing,
dimensional physiological indices,” in Proc. EPCE, 2011, pp. 176–185.
vol. 126, no. 27, pp. 78–94, 2014.
[41] S. Bando and A. Nozawa, “Detection of driver inattention from fluctua-
[12] J. H. Yang et al., “Detection of driver fatigue caused by sleep depriva-
tions in vehicle operating data,” Artif. Life Robot., vol. 20, no. 1, pp. 28–33,
tion,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 39, no. 4,
2015.
pp. 694–705, Jul. 2009.
[42] T. Nguyen et al., “Utilization of a combined EEG/NIRS system to predict
[13] D. Sandberg et al., “Detecting driver sleepiness using optimized nonlinear
driver drowsiness,” Sci. Rep., vol. 7, 2017, Art. no. 43933.
combinations of sleepiness indicators,” IEEE Trans. Intell. Transp. Syst.,
[43] X. Liu et al., “Sleep loss and daytime sleepiness in the general adult
vol. 12, no. 1, pp. 97–108, Mar. 2011.
population of Japan,” Psychiatry Res., vol. 93, no. 1, pp. 1–11, 2000.
[14] P. M. Forsman et al., “Efficient driver drowsiness detection at moder-
[44] M. J. Sateia, “International classification of sleep disorders-third edition:
ate levels of drowsiness,” Accident Anal. Prev., vol. 50, pp. 341–350,
highlights and modifications,” Chest, vol. 146, no. 5, pp. 1387–1394, 2014.
Jan. 2013.
[45] F. Pizza et al., “Daytime sleepiness and driving performance in pa-
[15] J. Trinder et al., “Autonomic activity during human sleep as a function of
tients with obstructive sleep apnea: Comparison of the MSLT, the MWT,
time and sleep stage,” J. Sleep Res., vol. 10, no. 4, pp. 253–264, 2001.
and a simulated driving task,” Sleep, vol. 32, no. 3, pp. 382–391,
[16] K. T. Chui et al., “An accurate ECG-based transportation safety drowsiness
2009.
detection scheme,” IEEE Trans. Ind. Inform., vol. 12, no. 4, pp. 1438–1452,
[46] P. Philip, “Can the MSLT be a useful tool to assess motor vehicle crash
Aug. 2016.
risk in sleepy drivers?” Sleep, vol. 33, no. 6, pp. 729–730, 2010.
[17] B. G. Lee et al., “Real-time physiological and vision monitoring of vehi-
[47] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32,
cle driver for non-intrusive drowsiness detection,” IET Commun., vol. 5,
2001.
no. 17, pp. 2461–2469, 2011.
[48] M. Xiao et al., “Sleep stages classification based on heart rate variability
[18] A. J. Camm et al., “Guidelines heart rate variability—standards of mea-
and randomforest,” Biomed. Signal Process Control, vol. 8, no. 6, pp. 624–
surement, physiological interpretation, and clinical use,” Eur. Heart J.,
633, 2013.
vol. 115, no. 5, pp. 354–381, 1996.
[49] S. Tsukada et al., “Conductive polymer combined silk fiber bun-
[19] K. Fujiwara et al., “Epileptic seizure prediction based on multivariate
dle for bioelectrical signal recording,” PLoS One, vol. 7, no. 4, 2012,
statistical process control of heart rate variability features,” IEEE Trans.
Art. no. e33689.
Biomed. Eng., vol. 63, no. 6, pp. 1321–1332, Jun. 2016.
[50] M. Kano et al., “A new multivariate statistical process monitoring method
[20] D. H. Kerem and B. Geva, “Forecasting epilepsy from the heart rate
using principal component analysis,” Comput. Chem. Eng., vol. 25, no. 7–
signal,” Med. Biol. Eng. Comput., vol. 43, pp. 230–239, 2005.
8, pp. 1103–1113, 2001.
[21] K. Kato et al., “Earlier tachycardia onset in right than left mesial temporal
[51] J. E. Jackson et al., “Control procedures for residuals associated with
lobe,” Neurology, vol. 83, pp. 1332–1336, 2014.
principal component analysis,” Technometrics, vol. 21, no. 3, pp. 341–
[22] F. Versace et al., “Heart rate variability during sleep as a function of the
349, 1973.
sleep cycle,” Biol. Psychol., vol. 63, no. 2, pp. 146–162, 2003.
[23] F. Jurysta et al., “A study of the dynamic interactions between sleep EEG
and heart rate variability in healthy young men,” Clin. Neurophysiol.,
vol. 114, no. 11, pp. 2146–2155, 2003.