Biomedical Signal Processing and Control 18 (2015) 360–369

Contents lists available at ScienceDirect

Biomedical Signal Processing and Control

journal homepage: www.elsevier.com/locate/bspc

Yawn analysis with mouth occlusion detection

Masrullizam Mat Ibrahim, John J. Soraghan, Lykourgos Petropoulakis,
Gaetano Di Caterina ∗
Department of Electronic and Electrical Engineering, University of Strathclyde, 204 George Street, G1 1XW Glasgow, United Kingdom

a r t i c l e i n f o a b s t r a c t

Article history: One of the most common signs of tiredness or fatigue is yawning. Naturally, identification of fatigued
Received 11 August 2014 individuals would be helped if yawning is detected. Existing techniques for yawn detection are centred on
Received in revised form measuring the mouth opening. This approach, however, may fail if the mouth is occluded by the hand, as it
10 December 2014
is frequently the case. The work presented in this paper focuses on a technique to detect yawning whilst
Accepted 9 February 2015
also allowing for cases of occlusion. For measuring the mouth opening, a new technique which applies
Available online 6 March 2015
adaptive colour region is introduced. For detecting yawning whilst the mouth is occluded, local binary
pattern (LBP) features are used to also identify facial distortions during yawning. In this research, the
Strathclyde Facial Fatigue (SFF) database which contains genuine video footage of fatigued individuals is
used for training, testing and evaluation of the system.
© 2015 Elsevier Ltd. All rights reserved.

1. Introduction involuntary action where the mouth opens wide and, for this
reason, yawning detection research focuses on measuring and clas-
Fatigue is usually perceived as the feeling of tiredness or drowsi- sifying this mouth opening. Frequently, however, this approach
ness. In a work related context, fatigue is a state of mental and/or is thwarted by the common human reaction to hand-cover the
physical exhaustion that reduces the ability to perform work safely mouth during yawning. In this paper, we introduce a new approach
and effectively. Fatigue can result due to a number of factors includ- to detect yawning by combining mouth opening measurements
ing inadequate rest, excessive physical or mental activity, sleep with a facial distortion (wrinkles) detection method. For mouth
pattern disturbances, or excessive stress. There are many condi- opening measurements a new adaptive threshold for segmenting
tions which can affect individuals and which are considered directly the mouth region is introduced. For yawning detection with the
related to fatigue such as visual impairment, reduced hand–eye mouth covered, local binary patterns (LBP) features and a learning
coordination, low motivation, poor concentration, slow reflexes, machine classifier are employed. In order to detect the wrinkles,
sluggish response or inability to concentrate. Fatigue is consid- the edge detector Sobel operator [30] is used. Differently from [32],
ered as the largest contributor to road accidents, leading to loss where the mouth covered detection technique is applied to static
of lives. Data for 2010 from the Department of Transport, UK images only, the yawn analysis approach proposed in this paper
[1], points to 1850 people killed, and 22,660 seriously injured, describes a complete system for yawn detection in video sequences.
with fatigue contributing to 20% to the total number of accidents In this research, genuine yawning fatigue video data is used for
[2]. US National Highway Traffic Safety estimates suggest that training, testing and evaluation. The data is from the Strathclyde
approximately 100,000 accidents each year are caused by fatigue Facial Fatigue (SFF) video database, which was explicitly created
[3]. to aid this research, and which contains series of facial videos
Fatigue is identifiable from human physiology such as eye and of sleep deprived fatigued individuals. The, ethically approved,
mouth observations, brain activity, and by using electrocardio- sleep deprivation experiments were conducted at the University
grams (ECG) measuring, for example, heart rate variability. The of Strathclyde, with the aid of twenty volunteers under controlled
physical activities and human behaviour may also be used to conditions. Each of the 20 volunteers was sleep deprived for peri-
identify fatigue [4–6]. In this paper yawning, indicating fatigue, ods of 0, 3, 5 and 8 h on separate occasions. During each session the
is detected and analysed from video sequences. Yawning is an participants’ faces were recorded while they were carrying out a
series of cognitive tasks.
The remainder of this paper is organised as follows. Section 2
∗ Corresponding author. Tel.: +44 (0) 141 548 4458. discusses the related work of this research, while Section 3 explains
E-mail address: [email protected] (G. Di Caterina). the SFF database. Section 4 discusses the overall system, while

http://dx.doi.org/10.1016/j.bspc.2015.02.006
1746-8094/© 2015 Elsevier Ltd. All rights reserved.
 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369 361

analysis and results are reported in Sections 5 and 6, respectively. detect the mouth region. The yawn is classified using colour prop-
Conclusions to the paper are provided in Section 7. erties to measure the breadth of the mouth opening. Elsewhere,
Gabor wavelet features that represent texture have been used by
Xiao et al. [9] for evaluating the degree of mouth opening. The tex-
2. Related work ture features are extracted from the corner regions of the mouth.
The texture of the corner is different for mouth opening and clos-
Yawning is a symptom indicated visually by a mouth open state. ing. The yawn is classified using linear discriminant analysis (LDA).
Yawning detection research focuses mostly on the measurement Yang et al. [17] employed a back propagation neural network for
of the mouth opening [7–9]. The algorithms used must there- yawn classification in three states: mouth closed, normal open, and
fore be able to differentiate between normal mouth opening and wide open. The mouth region features are extracted based on the
yawning. Yawn detection can be generally categorised into three RGB colour properties. For this approach a huge number of datasets
approaches; feature based, appearance based, and model based. In for images is required in order to obtain adequate results.
the feature based method, the mouth opening can be measured The model based approach requires that the mouth or lips are
by applying several features such as colour, edges and texture, modelled first. An active shape model (ASM) for instance, requires
which are able to describe the activities of the mouth. Commonly, a set of training images and, in every image, the important points
there are two approaches to measure the mouth opening, mainly that represent the structure of the mouth need to be marked. Then,
by tracking the lips movement and by quantifying the width of the marked set of images is used for training using a statistical
the mouth. Since the mouth is the wider facial component, colour algorithm to establish the shape model.
is one of the prominent features used to distinguish the mouth García et al. [18] and Anumas et al. [19] train the structure
region. Yao et al. [10] transfer RGB colour space to LaB colour using ASM, and the yawn is measured based on the lips open-
space in order to use the ‘a’ component, which is found as accept- ing. Hachisuka et al. [20] employ active appearance model (AAM),
able for segmenting the lips. The lips represent the boundary of which annotates each essential facial point. For yawning detection
the mouth opening that is to be measured. YCbCr colour space three points are defined to represent the opening of the mouth. In
was chosen by Omidyeganeh [11] for their yawning detection. this approach a lot of images are required since every person has
In that work, the claim is that colour space is able to indicate different facial structure.
the mouth opening area, which is the darkest colour region, by
setting a certain threshold. Yawning is detected by computing
3. Strathclyde Facial Fatigue (SFF) database
the ratio of the horizontal and vertical intensity projection of the
mouth region. The mouth opening is detected as yawn when this
The Strathclyde Facial Fatigue database was developed in order
ratio is above a set threshold. Implementation using the colour
to obtain genuine facial signs of fatigue. Most of the fatigue related
properties is easy but, to make the algorithm robust, several chal-
researchers have used their own datasets for assessing their devel-
lenges must be addressed. The lips colour is different for everyone,
oped system. For example, Vural et al. [21] developed their fatigue
and due to the different lighting conditions it is necessary to ensure
database by using video footage from a simulated driving environ-
that the threshold value is adapted to the changes.
ment. The participants had to drive for over 3 h and their faces were
Edges are another mouth feature able to represent the shape
recorded during that time. Fan et al. [22] recruited forty partici-
of the mouth opening. Alioua et al. [12] extracted the edges from
pants and recorded their faces for several hours in order to obtain
the differences between the lips region and the darkest region of
facial signs of fatigue. Some researchers only test their algorithms
the mouth. The width of the mouth is measured in consecutive
based on video footage of persons who pretend to experience
frames, and yawn is detected when the mouth is continuously
fatigue symptoms. For example, researchers in [8,11,12] detected
opening widely more than a set number of times. When using the
yawn based on the width of the mouth opening and the algorithms
edges to determine yawn, the challenge is that the changes of the
were not tested on subjects experiencing natural yawning. The SFF
edges are proportional to the illumination changes, making it dif-
database provides a genuine medium of facial fatigue video footage
ficult to set the parameter of an edge detector. In order to locate
from sleep deprivation experiments which involved twenty partic-
accurately the mouth boundary, the active contour algorithm can
ipants.
be used as implemented in [8]. However, the computational cost
The sleep deprived volunteers were 10 male and 10 female with
of this approach is very high. The mouth corners have also been
ages ranging between 20 to 40 years old. Each volunteer had to per-
employed by Xiao et al. [9] to detect yawn. The mouth corners can
form a series of cognitive tasks during four separate experimental
be reference points in order to track the mouth region. Yawning
sessions. Prior to each session, the participants were sleep deprived
is classified based on the change of the texture in the corner of
for 0 h (no sleep deprivation), 3, 5 and 8 h under controlled condi-
the mouth while the mouth is opened widely. Moreover, as in the
tions. The cognitive tasks were designed to test (a) simple attention,
case implemented for measuring eye activities, Jiménez-Pinto et al.
(b) sustained attention and (c) the working memory of the partici-
[13] have detected the yawn based upon the salient points in the
pants. These cognitive tasks are associated with sensitivity to sleep
mouth region. These salient points introduced by Shi and Tomasi
loss and are designed to accelerate fatigue signs. Fig. 1 shows some
[14] are able to describe the motion of the lips movement, and yawn
examples of video footage in the SFF database.
is determined by examining the motion in the mouth region. In
[7,11] yawn is detected based on a horizontal profile projection.
The height of the mouth opening in profile projection represents 4. System overview
the mouth yawn. However, this approach is inefficient for beard
and moustache bearing individuals. A block diagram of the new yawn analysis system is shown in
In appearance based method, statistical learning machine algo- Fig. 2. The system begins with an initialisation operation carried out
rithms are applied and distinctive features need to be extracted in in the first frame of the video. This consists of face acquisition and
order to train the algorithms. Lirong et al. [15] combined Haar-like region initialisation algorithms. During face acquisition, the face,
features and variance value in order to train their system to classify eye and mouth detection algorithms are performed sequentially
the lips using support vector machine (SVM). With the additional before a region of interest initialisation algorithm is executed.
integration of a Kalman filter their algorithm is also able to track Two regions of interest are initialised: focused mouth region
the lip region. Lingling et al. [16] also applied Haar-like features to (FMR) and focused distortion region (FDR). These two regions are
362 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369

Fig. 1. Example images from video footage SFF.

Faace acquisition

Face Detection

Eye Detection

Mouth Detection

Regions Interest
Initialisation

Fig. 3. (a) Anthropometric measurement; (b) focused mouth region (FMR); (c)
Yawn analysis operations focused distortion region (FDR).

Fig. 2. Yawn analysis system overview. From this information the coordinates of the FMR (Fig. 3(b)) are
empirically defined as:
   
x1 xR
to be tracked in all the following frames. The FMR region is used = (2)
in measuring the mouth opening and to detect when the mouth y1 yR + 0.75EMD
is covered, and FDR is used in measuring the distortions during    
yawning. x2 xL
= (3)
y2 yL + 0.75EMD + 0.8ED

4.1. Region of interest evaluation where (xR , yR ) and (xL , yL ) are the centre points of the right eye iris
and left eye iris, respectively. The FMR depends on the location and
As is indicated in [32] a facial region of interest needs to be distance of the eyes. When the face moves forward the eye dis-
established first. Fig. 3 depicts a face acquisition operation where tance increases and so does the FMR. Conversely, the FMR reduces
the eyes and mouth are sequentially detected using a Viola Jones following a backward move.
technique [23,24]. Training is performed through a cascade clas-
sifier applied to every sub-window in the input image in order 4.1.2. Focused distortion region (FDR)
to increase detection performance and reduce computation time. The focused distortion region (FDR) is a region in the face where
The SFF video footage provided the large number of face, eyes changes of facial distortion occur. The region, which is most likely
and mouth data needed to train the Haar-like features [25,26]. to undergo distortions during yawning, was identified based on
In the present method, the face region is first detected and then, conducted experiments using the SFF database. The coordinates of
the eyes and mouth areas are identified within it. The acquisition the FDR (Fig. 3(c)), are empirically defined as:
of the distinctive distances between these facial components is    
effected through anthropometric measurements, thus accounting x1 xR
for human facial variability. This information is then used in the = (4)
y1 yR − 0.75EMD
formation of the two following regions of interest.
   
x2 xL
= (5)
4.1.1. Focused mouth region (FMR) y2 yL + 0.75EMD
This is formed based upon the detected locations of the eyes and
mouth. The distance between the centre of the eyes (ED), and the The threshold values 0.75 and 0.8 used in Eqs. (2)–(5) are based
distance between the centre of mouth and the mid-point between on observation tests, undertaken on the SFF database.
the eyes (EMD) are obtained (Fig. 3(a)). The ratio of these distances
REM is then computed as: 4.2. Yawn analysis operations

ED As indicated in Fig. 2, when all regions of interest have been iden-

REM = (1)
EMD tified, three operations are carried out using these regions: mouth
 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369 363

Fig. 6. Focused mouth region (FMR).


Fig. 4. A linear transform that remaps intensity level input image in between Gmin

and Gmax .

3000
40

30 2000 Fig. 7. FMR segmentation using adaptive threshold.

20
1000 total number of index pixels in the histogram bins between 0 and
10
255 as illustrated in Fig. 5(b).
0 0
0 50 100 150 200 250 0 50 100 150 200 250
(a) (b) 
i
Hi = hj (7)
Fig. 5. (a) Enhanced image histogram; (b) cumulative histogram. j=1

Consequently, the initial adaptive threshold is calculated by ref-

erence to the darkest area as shown in Fig. 6. On average, the darkest
opening measurement, covered mouth detection, and distortion area constitutes approximately 5%, or less, of the entire obtained
detection. region of interest. Based on this percentage, the darkest area in the
histogram is situated between 0 and X intensity levels. The inten-
sity value X can be obtained by selecting the first 5% of the total
4.2.1. Mouth opening measurement number of pixels contained in the enhanced histogram (Fig. 5(a))
Yawning is an involuntary action that causes a person to open and reading the corresponding right-most intensity value from the
the mouth widely and breathe in. The technique in this paper x-axis.
detects yawning based upon the widest area of the darkest region Then, the adaptive threshold Ta value is calculated as follows:
between the lips. As previously discussed when applied techniques X − Pmin
Ta = (8)
are based on intensities and colour values, the challenge is to obtain Pmax − Pmin
an accurate threshold value which will adapt to multi-skin and lips
where Pmax and Pmin are the maximum and minimum pixel values,
colouration, as well as to different lighting conditions. The proposed
respectively.
algorithm applies a new adaptive threshold in order to meet this
After the adaptive threshold Ta is computed, the FMR is seg-
challenge.
mented and the mouth length LM is obtained as illustrated in Fig. 7.
The adaptive threshold value is calculated during the initialisa-
The length of LM must be within the range indicated in the following
tion operation after the FMR is formed. Due to the dependence of
equation:
the method on the intensity value of the region, it is necessary to
improve the face image in order to contrast enhance the darkest 0.7ED < LM < 0.9ED (9)
and the brightest facial regions. The enhancement is implemented
If LM is outside this range, the value Ta is increased or decreased
by transforming the histogram of the image to spread the level of
accordingly, in steps of 0.001. Then the segmentation process is
colour values evenly but without changing the shape of the input
repeated until the value of LM is within range as in Eq. (9).
histogram. This is computed using the following transformation
The adaptive threshold is applied to the next sequence of frames
[27]:
for segmenting the darkest region between the lips. In order to
 
determine the yawning, the height of the mouth HM , based on the
Gmax − Gmin
g  (x, y) = 
[g (x, y) − Gmin ] + Gmin (6) darkest region in FMR, is measured as shown in Fig. 8. Then, the
Gmax − Gmin Yawn Ratio (RY ) of the height HM over the height of FMR is computed
as follows:
The linear transformation illustrated in Fig. 4 stretches the his- HM
togram of the input image into the desired output intensity level RY = (10)
FMRheight

of the histogram between Gmin 
and Gmax , where Gmin and Gmax are
the minimum and maximum input image intensities, respectively. A value of RY higher than a certain threshold denotes a mouth
From this enhanced histogram illustrated in Fig. 5(a), the cumu- open. Such a threshold has been computed empirically as reported
lative histogram Hi is calculated as in Eq. (7), thus generating the in Section 4.1.
364 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369

Fig. 8. The measurement of high of mouth opening. Fig. 11. Example of extended LBP operator. (a) (8,1); (b) (16,2) and (c) (24,3) neigh-
bourhoods.

where the notation (P, R) represents a neighbourhood of P sampling

points on a circle of radius R.
The LBP result can be formulated in decimal form as:

P−1
LBPP,R (xc , yc ) = s (iP − ic ) 2P (11)
P=0

where ic and iP denote gray level values of the central pixel and
P represent surrounding pixels in the circle neighbourhood with
radius R. Function s(x) is defined as:

1, if x ≥ 0
s(x) = (12)
0, if x < 0

The LBP operator, as formulated in Eq. (11), has a rotational

effect. Hence, if the image is rotated, then, the surrounding pixels in
each neighbourhood are moving accordingly along the perimeter
of the circle. In order to remove this effect Ojala et al. [29] proposed
a rotational invariant (ri) LBP as follows:
Fig. 9. (a) Uncovered mouth; (b) covered mouth.   

LBPri
P,R = min ROR LBPP,R , i i = 0, 1, ..., P − 1 (13)

where ROR(x, i) performs a circular bitwise right shift on the P-bit

number x with i time. A LBP uniform pattern (u2), LBPu2(P,R) was also
proposed. The LBP is called uniform when it contains a maximum
of two bitwise transitions from 0 to 1 or vice versa. For example,
1111111 (0 transition) and 00111100 (2 transitions) are both uni-
form, but 100110001 (4 transitions) and 01010011(6 transitions)
Fig. 10. Example of the basic LBP operator. are not uniform. Another LBP operator, which is a combination of
a rotational invariant pattern with a uniform pattern, is LBPriu2
(P,R) .
This operator is obtained by simply counting the 1’s in uniform
4.2.2. Covered mouth detection pattern codes. All non-uniform patterns are placed in a single bin.
It is not unusual to occlude the mouth with the hand during An example of an LBP histogram of the three types of LBP operators
yawning. This means that information from Section 4.2.1 can no for covered and uncovered mouth region is shown in Fig. 12.
longer be used to detect yawning. The approach first introduced in A classifier then uses the LBP features to detect mouth covering,
[32] on still images can also be used in video sequence footage. The as explained in Section 6.2.
approach involves that the FMR is examined to determine whether
the region is covered or not. Fig. 9 displays images of FMR (a) not 4.2.3. Distortions detection
covered and (b) covered. From these images the different texture Yawning cannot be concluded solely on the detection of a
of the FMR regions is clearly obvious in the two cases. In order to covered mouth region. It is possible that the mouth is covered acci-
differentiate between these two cases, local binary patterns (LBP) dentally or even intentionally for reasons unrelated to yawning. To
are used to extract the FMR texture pattern. LBP features have minimise the possibility of false yawn detection, the metric of dis-
been proven highly accurate descriptors of texture, are robust to tortions in a specific region of the face, most likely to distort while
the monotonic gray scale changes, as well as simple to implement yawning, is also used [32]. This region is identified based on obser-
computationally [28]. vations and the experiments conducted using the video footage
Local binary patterns (LBP) was originally applied on texture from the SFF database.
analysis to indicate the discriminative features of texture. With ref- The region most affected by distortion during yawning is the
erence to Fig. 10, the basic LBP code of the central pixel is obtained focused distortion region (FDR) (Section 4.1.2). From Fig. 13b and c
by subtracting its value from each of its neighbours and, if a negative it is obvious that there are significant visual differences in the FDR
number is obtained, it is substituted with a 0, else it is a 1. area between normal conditions and during yawning.
The LBP operator is limited since it represents a small scale fea- To evaluate the changes occurring in FDR, edge detection is used
ture structure which may be unable to capture large scale dominant to identify the distortions. Here any edge detector can be used.
features. In order to deal with the different scale of texture, Ojala et However, based on experiments carried out, the Sobel operator [30]
al. [29] presented an extended LBP operator where the radius and was chosen since it detected most of the required edges, at a rea-
sampling points are increased. Fig. 11 shows the extended operator sonable computational cost. The Sobel operator first calculates the
 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369 365

Fig. 14. FDR with input image and edges detected image. (a) Normal; (b) yawn.

Fig. 12. LBP histogram for not covered mouth and covered mouth regions.

Fig. 15. Plot of normalised FDRSAD values during yawning.

During yawning, the numbers of the detected edges increase as

shown in Fig. 15.


FDRSAD = I1 (i, j) − I2 (x + i, y + j) (16)
i,j

FDRSAD
Normalised FDRSAD = (17)
255 × W × H

Fig. 13. (a) Focused distortion region (FDR); (b) normal condition in FDR; (c) yawn-
ing in FDR. 5. Yawn analysis

In general, the action of yawning can be categorised into three

intensity gradient at each point in the region of interest. Then, it situations: (a) mouth not-covered and wide open, (b) mouth wide
provides the direction of the largest possible increase from light to open for a short time before it gets covered and, (c) mouth
dark and the rate of change in the horizontal and vertical directions. totally covered with invisible mouth opening. These situations are
The Sobel operator represents a partial derivative of f(x, y) and of based on the observations made from the video footage of the
the central point of a 3 × 3 area of pixels. The gradients for the hor- SFF database. A yawn analysis algorithm, as depicted in Fig. 16,
izontal Gx and vertical Gy directions for the region of interest are was therefore developed to deal with these cases by introducing
then computed as [31]: the aforementioned procedures, i.e. mouth opening measurement,

mouth covered detection and distortions detection.
Gx = f (x + 1, y − 1) + 2f (x + 1, y) + f (x + 1, y + 1) In this algorithm, the status of mouth opening, mouth covered,

(14) and distortions are examined periodically. The chosen time frame
− f (x − 1, y − 1) + 2f (x − 1, y) + f (x − 1, y − 1)
for measurements, based on observations from the SFF database,

is 30 s—a time interval reached by examining the SFF video data.
Gy = f (x − 1, y + 1) + 2f (x, y + 1) + f (x + 1, y + 1)
This assumes that yawning does not occur twice within this time

(15)
− f (x − 1, y − 1) + 2f (x, y − 1) + f (x + 1, y − 1) period.
The algorithm starts by first examining the mouth opening.
The FDR distortions are detected based on the gradients in both Yawning is identified through the mouth opening for longer than a
directions and the results are shown in Fig. 14. The number of edges fixed interval , usually between 3 and 5 s. If this shows no yawning,
detected under normal conditions, as illustrated in Fig. 14(a), is sig- then the mouth covered detection is triggered. The status of distor-
nificantly different compared to yawning (Fig. 14(b)). In order to tions in FDR is only examined when mouth covering is detected.
identify the distortion in the FDR during yawning, the sum FDRSAD In the situation where yawning occurs with the mouth wide
of the absolute values obtained in Eq. (16) is used to compute the open for a short time and then hand-covered quickly, the algorithm
number of wrinkles in the region. The normalised FDRSAD is calcu- checks the sequence of events, i.e. mouth open wide followed by
lated in Eq. (17) where W and H denote the width and height of the mouth covered. As a result of this analysis, a yawn status is detected
FDR, respectively. and the period of yawning is also obtained.
366 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369

Mouth Opening
(mo)

Covvered No
mo outh mo >
deteection
Yes

YAWN

Mouth
vered
cov No

NOT YAWN Fig. 17. Mouth opening segmentation images.

Yes
6.2. Mouth covered detection
Distoortion
deteection In order to detect the covered mouth, the FMR is examined in
every video frame. The LBP features are extracted from the region,
and the learning machine algorithm decides the status. The clas-
sifiers tested, to determine their suitability for use in the analysis
algorithm, comprised support vector machines (SVM) and neural
networks (NN).
No
Disttortion A total of 500 covered mouth images and 100 non covered
mouth images of size 50 × 55 pixel have been used to train the NN
and SVM classifiers. The LBP operators used for feature extraction
Yes were: rotational invariant (ri), uniform (u2), and uniform rotational
NOT YAWN
invariant (riu2). The radii of the regions examined were either 8, 16,
or 24 pixel.
YA
AWN
Classification was performed on 300 images. Four different ker-
nel functions were tested for the SVM classifier and two different
Fig. 16. Yawn analysis algorithm. network layers for the NN. Analytically these were: SVM with a)
radial basis function (SVM-rbf), (b) linear (SVM-lin), (c) polynomial
(SVM-pol), and (d) quadratic (SVM-quad). The neural networks
tested had 5 (NN-5) and 10 neurons (NN-10), respectively, in their
6. Experimental results
hidden layer.
It is usual to present such results using graphs of sensitivity
The performance of the developed algorithms was tested by
(ordinate) against 1-specificity (abscissa), where sensitivity and
using the full SFF video footage. The results presented here were the
specificity are given, respectively, by:
outcome of substantial real-time experimentation using the video
sequences from the whole of the database. TP
Sensitivity = (18)
TP + FP
TN
6.1. Mouth opening measurement Specificity = (19)
TN + FN
The mouth opening measurement algorithms are tested using where Tp represents the number of true positives, Fp the number of
the SFF database. In order to evaluate the accuracy of the mouth false positives, Tn the number of true negatives, and Fn the number
opening segmentation using the proposed adaptive threshold, the of false positives. Sensitivity represents the ability of the developed
actual input image is compared with the segmented region of algorithm to identify the covered mouth. The specificity relates to
mouth opening. Fig. 17 shows examples of mouth opening images the ability of the algorithm to detect the non-covered mouth. For
paired with the output image of mouth opening segmentation. A maximum accuracy and minimum number of false positives, max-
total of 145 mouth opening images under varying lighting condi- imum values of sensitivity and specificity are required. The ROC
tions were used for testing. From these tests, it was shown that the curves from the results of the classifications are shown in Fig. 18.
algorithm was able to segment the darkest region between the lips The legends shown in Fig. 18(a–c) indicate the classifiers used
as required for detecting yawning. in each case and their respective outputs.
As discussed in Section 4, yawning is determined by measuring In Fig. 18 the best performance is provided by the LBP-classifier
the height of the segmented region of the mouth opening. combination which is closer to the upper left part of the graphs
In order to obtain the applicable threshold value that represents where the sensitivity approaches the value of 1 and the value of
yawning, 25 of the real yawning scenes from the SFF database were 1-specificity approaches 0. On this basis, the best performance is
used. From the yawning scenes it was found that the minimum illustrated in Fig. 18(c), which corresponds to the combinations of
value of RY which denotes yawning is 0.5. This ratio value is applied LBP with NN10, NN5 and SVM-linear as classifiers, with results close
in mouth opening measurements during yawn analysis. to perfection.
 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369 367

Fig. 19. Plots of yawning detection with mouth not occluded.

0.5. Based on the results obtained and shown in Fig. 19, for mouth
covered classification the LBP uniform pattern (u2) with radius 16
and a neural network classifier with 5 neurons in the hidden layer
are chosen to determine the status of FMR.
Genuine yawning images from sequences of 30 video clips
from the SFF database are used for evaluation of the algorithm’s
performance. These videos contain 10 yawning sequences with
non-covered mouth scenes, 14 with covered mouth scenes and
6 scenes with both situations present. Examples of the detection
results in each situation of yawning are shown in Figs. 19–20.
When the mouth is not occluded (Fig. 19), yawning is detected
when the height of the mouth opening is equal to or greater than the
threshold value. The yawning period is measured from the begin-
ning of the intersection point between the threshold value and
the ratio of the mouth opening height to FMR height RY . When
the RY value is below the threshold (Fig. 20), this could be due
to mouth being covered during yawning. In this case, the mouth
covered detection part of the system is triggered. If mouth occlu-
sion is detected, facial distortion during the period of occlusion is
checked. Yawning is assumed to occur when distortion increases
substantially within that period.
When the mouth is opened wide over a short period of time
before it is occluded (Fig. 21), the value of RY exceeds the thresh-
old value for a short period before it drops quickly. In this case,
Fig. 18. ROC curves for LBP operator: (a) rotational invariant (ri); (b) rotational
the mouth occlusion detection will be triggered when the RY value
invariant with uniform pattern (riu2); (c) uniform pattern (u2) with zoom-in insert
of upper left corner. drops below the threshold. If occlusion is detected, then the period
of yawning is measured from the beginning of the first intersection
of RY with the threshold value, until the end of the mouth occlusion
6.3. Yawn analysis detection.
From the 30 video scenes of yawning, 28 of them were detected
Yawn analysis is a combination of mouth opening measurement, successfully with the period measured accurately. However the
mouth covered detection, and distortions detection. algorithm failed to detect two yawning scenes where the mouth
These operations are examined every 30 s. For mouth opening is completely covered. This is because the FDR did not indicate any
measurements, the best threshold value that represents yawning is distortion.
368 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369

7. Conclusion

A yawn analysis algorithm designed to deal with both occluded

and uncovered mouth yawning situations was presented. This
algorithm is a combination of techniques for mouth opening mea-
surement, mouth covered detection and distortions detection.
The mouth opening is measured based on the darkest region
between the lips. The darkest region is segmented using an adaptive
threshold which is introduced in this paper. For detecting mouth
occlusion, an LBP uniform operator is applied to extract features of
the mouth region, and the occlusion is evaluated using a neural net-
work classifier. To ensure that the mouth occlusion is indeed due
to yawning and no other reason, the wrinkles of specific regions on
the face are measured. Yawning is assumed to have occurred when
distortions in these regions are detected.
In the presented yawn analysis algorithm, the two regions FMR
and FDR are monitored every 30 s. The analysis results produce
the status of yawning, as well as the period of yawning. Genuine
sequences of yawning from the SFF database video footage are used
for training, testing and evaluating the performance of the devel-
oped algorithms. Based on the conducted experiments and results
obtained, the overall developed system developed indicated a very
encouraging performance.

Acknowledgements

This work was supported by the Scottish Sensor Systems Centre

(SSSC), the Ministry for Higher Education of Malaysia and Universiti
Teknikal Malaysia, Melaka (UTeM). The SFF database development
was funded by a Bridging The Gap (BTG) grant from the University
of Strathclyde.
Fig. 20. Plots of yawning detection with mouth occluded.
The work which led to the development of the SFF database
was in collaboration with the school of Psychological and Health
Sciences, University of Strathclyde, the Glasgow Sleep Centre, Uni-
versity of Glasgow, Brooks Bell Ltd.

References

[1] D. f. T. UK, “Reported Road Casualties in Great Britain: 2010 Annual

Report” Department for Transport UK, United Kingdom,Statistical Release - 29
September 2011.
[2] P. Jackson, C. Hilditch, A. Holmes, N. Reed, N. Merat, L. Smith, Fatigue and
Road Safety: A Critical Analysis of Recent Evidence, Department for Transport,
London, 2011, pp. p88.
[3] R. Resendes, K.H. martin, Saving Lives Through Advanced Safety Technology,
Faderal Highway Administration, US, Washington, DC, 2003.
[4] Daimler, Drowsiness-Detection System Warns Drivers to Pre-
vent Them Falling Asleep Momentarily, Daimler, 2012, Available:
http://www.daimler.com/dccom/0-5-1210218-1-1210332-1-0-0-1210228-
0-0-135-0-0-0-0-0-0-0-0.html.
[5] M. Michael Littner, C.A. Kushida, D. Dennis Bailey, R.B. Berry, D.G. Davila, M.
Hirshkowitz, Practice parameters for the role of actigraphy in the study of sleep
and circadian rhythms: an update for 2002, Sleep 26 (2003) 337.
[6] N. Mabbott, ARRB Pro-active fatigue management system, Road Trans. Res. 12
(2003) 91–95.
[7] Y. Lu, Z. Wang, Detecting driver yawning in successive images, in: Bioinfor-
matics and Biomedical Engineering, 2007. ICBBE 2007. The 1st International
Conference on, 2007, pp. 581–583.
[8] M. Mohanty, A. Mishra, A. Routray, A non-rigid motion estimation algorithm for
yawn detection in human drivers, Int. J. Comput. Vision Rob. 1 (2009) 89–109.
[9] F. Xiao, Y. Bao-Cai, S. Yan-Feng, Yawning detection for monitoring driver
fatigue, in: Machine Learning and Cybernetics, 2007 International Conference
on, 2007, pp. 664–668.
[10] W. Yao, Y. Liang, M. Du, A real-time lip localization and tacking for lip reading,
in: Advanced Computer Theory and Engineering (ICACTE), 2010 Third Interna-
tional Conference on, 2010, V6-363-V6-366.
[11] M. Omidyeganeh, A. Javadtalab, S. Shirmohammadi, Intelligent driver drowsi-
ness detection through fusion of yawning and eye closure, in: Virtual
Environments Human–Computer Interfaces and Measurement Systems (VEC-
IMS), 2011 IEEE International Conference on, 2011, pp. 1–6.
[12] N. Alioua, A. Amine, M. Rziza, D. Aboutajdine, in: P. Real, D. Diaz-Pernil, H.
Fig. 21. Plots of yawning detection with mouth initially not occluded but occluded Molina-Abril, A. Berciano, W. Kropatsch (Eds.), Driver’s Fatigue and Drowsiness
subsequently. Detection to Reduce Traffic Accidents on Road Computer Analysis of Images and
Patterns, vol. 6855, Springer, Berlin/Heidelberg, 2011, pp. 397–404.
 M.M. Ibrahim et al. / Biomedical Signal Processing and Control 18 (2015) 360–369 369

[13] J. Jiménez-Pinto, M. Torres-Torriti, Driver alert state and fatigue detection by [23] P. Viola, M. Jones, Robust real-time face detection, in: Computer Vision, 2001.
salient points analysis, in: Systems, Man and Cybernetics, 2009. SMC 2009. IEEE ICCV 2001. Proceedings. Eighth IEEE International Conference on, 2001, p. 747.
International Conference on, 2009, pp. 455–461. [24] P. Viola, M.J. JONES, Robust real-time face detection, Int. J. Comput. Vision 57
[14] J. Shi, C. Tomasi, Good features to track, in: Computer Vision and Pattern Recog- (2) (2004) 137–154.
nition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, [25] P.I. Wilson, J. Fernandez, Facial feature detection using Haar classifiers, J. Com-
1994, pp. 593–600. put. Sci. Coll. 21 (2006) 127–133.
[15] W. Lirong, W. Xiaoli, X. Jing, Lip detection and tracking using variance based [26] M. Castrillón, O. Déniz, C. Guerra, M. Hernández, ENCARA2: real-time detection
Haar-like features and Kalman filter, in: Frontier of Computer Science and of multiple faces at different resolutions in video streams, J. Visual Commun.
Technology (FCST), 2010 Fifth International Conference on, 2010, pp. 608–612. Image Represent. 18 (2007) 130–140.
[16] L. Lingling, C. Yangzhou, L. Zhenlong, Yawning detection for monitoring driver [27] E.R. Davies, Machine Vision: Theory, Algorithms, Practicalities, Morgan Kauf-
fatigue based on two cameras, in: Intelligent Transportation Systems, 2009. mann, San Francisco, 2004.
ITSC’09. 12th International IEEE Conference on, 2009, pp. 1–6. [28] G. Zhao, M. Pietikainen, Dynamic texture recognition using local binary pat-
[17] Y. Yang, J. Sheng, W. Zhou, The monitoring method of driver’s fatigue based on terns with an application to facial expressions, in: Pattern Analysis and Machine
neural network, in: Mechatronics and Automation, 2007. ICMA 2007. Interna- Intelligence, IEEE Transactions on, 2007, pp. 915–928, 29.
tional Conference on, 2007, pp. 3555–3559. [29] T. Ojala, M. Pietikainen, T. Maenpaa, Multiresolution gray-scale and rota-
[18] H. García, A. Salazar, D. Alvarez, Á. Orozco, Driving fatigue detection using active tion invariant texture classification with local binary patterns, in: Pattern
shape models, Adv. Visual Comput. 6455 (2010) 171–180. Analysis and Machine Intelligence, IEEE Transactions on, 2002, pp. 971–987,
[19] S. Anumas, Kim Soo-chan, Driver fatigue monitoring system using video face 24.
images & physiological information, in: Biomedical Engineering International [30] R.O. Duda, P.E. Hart, Pattern Classification and Scene Analysis, vol. 3, Wiley,
Conference (BMEiCON), 2011, 2012, pp. 125–130. New York, NY, 1973.
[20] S. Hachisuka, K. Ishida, T. Enya, M. Kamijo, Facial expression measurement for [31] J.-Y. Zhang, Y. Chen, X.-x. Huang, Edge detection of images based on
detecting driver drowsiness, Eng. Psychol. Cogn. Ergon. 6781 (2011) 135–144. improved Sobel operator and genetic algorithms, in: Image Analysis and
[21] E. Vural, M. Cetin, A. Ercil, G. Littlewort, M. Bartlett, J. Movellan, Drowsy driver Signal Processing, 2009. IASP 2009. International Conference on, 2009,
detection through facial movement analysis, in: Human–Computer Interaction, pp. 31–35.
2007, pp. 6–18. [32] M. Mat Ibrahim, J. Soraghan, L. Petropoulakis, Mouth covered detection for
[22] X. Fan, B.-C. Yin, Y.-F. Sun, Dynamic human fatigue detection using feature- yawn, in: IEEE International Conference on Signal and Image Processing Appli-
level fusion, in: Springer (Ed.), Image and Signal Processing, Springer, 2008, pp. cations (ICSIPA), 2013, pp. 89–94.
94–102.