See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/319941041

Touchless Virtual Keyboard Controlled by Eye Blinking and EEG Signals

Conference Paper · January 2018

DOI: 10.1007/978-3-319-67792-7_6

CITATIONS READS

0 722

2 authors, including:

Krzysztof Dobosz
Silesian University of Technology
18 PUBLICATIONS   20 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Accessible text entry View project

Applications supporting the rehabilitation of people with cognitive disabilities View project

All content following this page was uploaded by Krzysztof Dobosz on 27 November 2017.

The user has requested enhancement of the downloaded file.

 Touchless virtual keyboard controlled by eye
blinking and EEG signals - DRAFT VERSION

Krzysztof Dobosz and Klaudiusz Stawski

Silesian University of Technology, Institute of Informatics,

Akademicka 16, Gliwice, Poland
[email protected],[email protected]

Abstract. The paper describes the idea of touchless virtual keyboard

designed for disabled people with tetraparesis. Each key can be selected
by three double eye blinks registered by EMG sensor. EEG signals are
used as a support that allows the user to change the input mode of sin-
gle characters to the mode of predicted words selection. The keyboard
was implemented and evaluated during provided experiments. Obtained
result (W P M = 1.11) partially confirms the calculated typing speed
(W P M = 1.23). It is also found that the keyboard efficiency can be im-
proved by using a list of predicted words, even the number of those words
is low and they are short (W P M = 1.27). Provided discuss presents that
improved keyboard used by experienced users could achieve the typing
efficiency even up to W P M = 3.15.

Keywords: virtual keyboard, text entry, brain-computer interface, BCI,

eye blinking

1 Introduction
Communication and the ability to interact with the environment, especially to
other people, are fundamental needs for human relationships. For people who
suffer from severe physical disabilities, because they are completely paralyzed,
ability to comply with this need is almost impossible. Their physical activities
very often are limited to eye blinking. Although it is sufficient to communicate
using Morse code - this method is very cumbersome. Human-Computer Interac-
tion (HCI) researchers explore opportunities of using as many sensors channels
as possible [24]. Among others, in support to impaired physical activity, bioelec-
trical brain signals can be used to provide an alternative communication channel.
A Brain-Computer Interface (BCI) is able to recognize changes in the ongoing
bioelectrical signals and to use them to appropriate commands in order to realize
the communication aids.
The aim of the project was to propose useful BCI, which allow users with
severe motor disabilities to use virtual keyboard. The electroencephalographic
(EEG) devices mostly measure fundamental human activity states as atten-
tion and relaxation (meditation). The devices sometimes integrates also an elec-
tromyographic sensor that can be used for eye blinks recognition. Such signals
2 Krzysztof Dobosz and Klaudiusz Stawski

can be sufficient to type characters on a virtual keyboard. The problem is to

design the appropriate method to move among virtual keys and select the char-
acters in order to achieve the efficient text entry.

2 Related Works

Eye blinking is a semi-autonomic rapid closing of the eyelid. Although it is the

essential function of the eye that helps spread tears across and remove irritants
from its surface, it can also play important role in the human-computer interac-
tion. Most of researches use eye blinks as an additional element in eye-tracking
systems [4]. Sometimes eye blink detection as a valuable supplement for faster
mouse click emulation [7]. There are also several ways to write by gaze [12].
In a typical setup, gaze direction is used to point and select letters on an on-
screen keyboard. Average efficiency of such solutions is about 5 W P M (Words
Per Minute). Alternatively, if the person cannot fixate and use traditional eye-
tracking system, the eyes can be used as switches using blinks to enter and
correct text [20]. Each kind of text entry, even using a single input as blinks, can
be improved (increase of the text entry speed) implementing words prediction
[21]. Most of the methods used to detect blinking are based on image processing
[17, 8].
In the recent years, a noticeable number of BCI systems have been developed
to provide an alternative communication tool for people with severe neuromus-
cular disorders. Hundreds of BCI research articles were published [5]. A lot of
implemented BCI applications are associated with different areas, i.e.: mental
speller [2] mouse control [14], robot arm control [3], game applications [13], navi-
gation [23], drowsiness detection system [9], brain-controlled smart home system
[10], cognitive ability assessment system [18], etc.. EEG applied to control a vir-
tual keyboard gave results from 0.85 [16] up to CP M = 3.38 (Characters Per
Minute) in error-free writing [19]. Some researchers noticed that the EEG and
EMG data can be used at the same time in order to improve the accuracy of
user input when using the biometric signal to the computer interface [6].
The aim of this work was to propose a new method of typing using eye
blinking with a support of EEG signals and words prediction.

3 Research Environment

3.1 EEG Device

The main element of the research environment is the EEG device. One of com-
monly available consumer of EEG signals is the NeuroSky MindWave Mobile
[15]. Devices developed by this producer are inexpensive, work well using a dry
electrode and have their own Software Development Kit, so software develop-
ers can easily create own applications. The NeuroSky Mindwave Mobile has also
good measurement accuracy, which can result in a wider group of potential users.
 Touchless Virtual Keyboard Controlled by Eye Blinking and EEG Signals 3

The device is composed of one dry electrode and a specially designed elec-
tronic circuit. It covers of a headset, an ear-clip, and a sensor arm. The headset’s
reference and ground electrodes are on the ear clip and the EEG electrode is on
the sensor arm, resting on the forehead above the eye. The MindWave Mobile
safely measures and outputs the EEG power spectrum (alpha waves, beta waves,
etc.). The sensor arm includes also EMG sensor that allows the user to measure
the blink strength. The headset transfers data via Bluetooth.
Typically, real time EEG signal processing and classification algorithms are
designed for powerful machines. Some of them use a weighted combination of
various classifiers. However, the producer’s firmware reduces the complexity of
managing the connection and handles parsing of the data stream from the EEG
headset. This comfortable software interface supplies both raw data and pre-
processed data. The software developer receives the value of attention and med-
itation normalized to the range 0-100%. Movement of muscles responsible for
blinking of the eye is normalized in the same way. It is very convenient and
helps avoid raw data analysis.

3.2 Software Tool

Developed research tool (https://platforma.polsl.pl/rau2/file.php/675/

NeuroKeyboard.zip) has a form of a desktop application. The computer requires
a Bluetooth connection to communicate with the EEG device. First, the EEG
device must be paired to the computer, and then the virtual keyboard application
can be started. Next, the user gets a graphical interface consisting of four parts
responsible for different elements of functionality (Fig.1).

Fig. 1. Graphical user interface of the research tool.

4 Krzysztof Dobosz and Klaudiusz Stawski

They are:
– text area presenting the typed text - selected characters are automatically
added at the end of the text,
– virtual keyboard containing five rows of keys and the list of predicted words,
– configuration panel - informs about the state of connection to the MindWave
device, covers following information: current values of attention, meditation
and blinks, configuration settings, and Calibrate button that is used to per-
sonalize the thresholds,
– simulation panel - covers two screen buttons: DoubleBlink and ChangeMode.
The virtual keyboard allows the user to personalize operating parameters.
Settings are: meditation threshold, attention threshold, time above the thresh-
olds, eye blinking strength, time period for eye double-blink, interval of switching
between the keyboard sections.

4 Method
The virtual keyboard can work in two modes: character selection, predicted word
selection. Expecting that all operations controlled by EEG signals will be real-
ized very slowly, we decided that the switch between the keyboard sections will
be executed automatically at specified intervals. Assuming the switch between
single keys takes place every second, then enter a ten-letter word would take
3-5 minutes, which is unsatisfactory result. In order to accelerate the selection
of the correct character, the Divide and Conquer algorithm was implemented.
Therefore, first a row of characters is selected. Highlighted row of keys changes
to the next one at a certain time until the selection. The change of rows pro-
ceeds automatically from top to bottom (Fig.2). Then the first row is highlighted
again.

Fig. 2. Selection of the key - step 1.

After the row selection, the line of six characters is divided into two parts,
which are alternately highlighted. After the part selection, current group of three
 Touchless Virtual Keyboard Controlled by Eye Blinking and EEG Signals 5

characters are divided into single keys (Fig.3). They are also alternately high-
lighted in certain time period. Finally, after the key selection, the corresponding
character is inserted at the end of the text area.

Fig. 3. Selection of the key - step 2.

The most common operation is the selection, so we decided to use an eye

double blink for that. Double blink is chosen to filter out the involuntary blinking.
The user must use double blink three times to select required key.
The switch to the panel of predicted words is done after exceeding the at-
tention of meditation threshold (depending on settings). This operation is per-
formed less frequently than the selection of a key, so it was possible to apply
the method, which requires more time and is more prone to errors. Appropriate
concentration or meditation time period above the threshold was determined for
the experiments needs.

5 Experiments

Provided experiments involved twelve Polish-spoken volunteers aged 20-47 years

of both sexes. They had not used that kind of BCI before. Texts used in stud-
ies should be moderate in length, easy to remember, and representative of the
target language [11]. Very often they are pangrams. For the user evaluation the
pangram in Polish language without diacritical characters was chosen:

ICH DALEKOPIS FALSZUJE GDY PROBY XQV NIE WYTRZYMUJE

The sentence consists of 8 words built of 44 characters separated by 7 white

spaces. The experiments used a dictionary containing the 5000 most common
Polish words. For simplicity, Polish diacritical characters had been removed.
The maximum time between blinks in a double blink was experimentally set
to 600ms. After preliminary test we had also determined that the period time
tp between change of keyboard section (time for the user’s decision and double
6 Krzysztof Dobosz and Klaudiusz Stawski

blink) should be set to 1.5s. Table 1 presents results of the experiment (time
- in seconds, errk - number of incorrectly selected keys), when the users typed
the pangram without support of predicted words. Calculated text entry factors
are: CP M = 5.52, what is about W P M = 1.11 (considering that 1 word is equal
to about 5 characters for English text), calculated MSD error rate [22] equals
7.03%.

Table 1. Typing without words prediction.

trial 1 trial 2 trial 3 trial 4 trial 5 avr

user
time errk time errk time errk time errk time errk time errk
1 550 6 687 11 617 7 532 4 525 4 582.2 6.4
2 532 3 580 6 518 2 646 4 510 3 557.2 3.6
3 565 5 523 3 510 2 494 1 501 3 526.6 2.8
4 619 7 600 8 678 6 550 7 554 4 580.2 6.4
5 580 3 567 4 545 2 530 2 543 1 553.0 2.4
6 524 2 520 3 523 2 533 5 510 2 522.0 3.0
7 623 7 608 7 611 6 592 6 560 3 598.8 5.8
8 578 4 533 3 537 2 549 3 543 2 548.0 2.8
9 534 3 521 1 509 0 511 1 521 1 519.2 1.2
10 593 8 550 3 534 2 533 3 540 2 550.0 3.6
11 542 3 529 0 521 0 529 1 525 0 529.2 0.8
12 576 6 551 4 552 4 538 2 542 5 551.9 4.2

Although the keyboard can be calibrated in any moment, for the experiment
including predicted words, we decided to set the attention and meditation thresh-
olds manually to study its influence. Their values were selected after preliminary
observations. Too low threshold caused accidental switches to the mode of pre-
dicted words. Too high value caused problems with maintaining a sufficiently
high level for a certain period of time. After initial tests we decided to set the
required period over the attention and meditation threshold should be equal to 3
seconds. Next tables (Table 2 and Table 3) presents results, when the key typing
is supported by the predicted words. The attention threshold thrA and medita-
tion threshold thrM range were set to 60-80. It was selected after preliminary
observations. The value of errk was very similar to the previously obtained, so it
is replaced by errs - number of incorrect switches to the list of predicted words.
Unfortunately, only three words of the pangram belong to the dictionary
(ICH, GDY, NIE), but they have taken first places on the lists, and consequently
they were quickly selected together with the whitespace at the end. The best
results were obtained for thrA equals 65 (W P M = 1.21) and thrM equals 80
(W P M = 1.27). The average number of unexpected switching erravr to the list
of predicted words was decreased when the thresholds were increased. In the case
of thrM = 80 only one of users had accidental switches to the mode of predicted
words.
Touchless Virtual Keyboard Controlled by Eye Blinking and EEG Signals 7

Table 2. Typing with words prediction and attention threshold.

thrA = 60 thrA = 65 thrA = 70 thrA = 75 thrA = 80

user
time errs time errs time errs time errs time errs
1 575 2 539 1 601 2 631 1 689 0
2 615 3 535 2 621 2 605 1 715 1
3 490 1 470 1 512 2 492 0 553 0
4 469 1 512 2 554 1 561 1 654 0
5 450 1 466 0 462 0 476 0 609 0
6 510 2 489 1 521 1 504 1 581 0
7 498 2 478 1 480 1 510 0 543 0
8 512 1 515 1 510 1 520 1 573 0
9 602 4 589 2 580 2 590 1 595 2
10 534 2 512 1 487 1 492 0 546 0
11 489 2 474 1 498 0 497 0 512 0
12 521 1 501 1 510 2 545 0 601 0
avr 521.1 1.83 506.7 1.17 528.0 1.25 535.3 0.5 597.6 0.25
W P M 1.17 - 1.21 - 1.16 - 1.14 - 1.02 -

Table 3. Typing with words prediction and meditation threshold.

thrA = 60 thrA = 65 thrA = 70 thrA = 75 thrA = 80

user
time errs time errs time errs time errs time errs
1 542 2 563 2 492 1 483 0 455 0
2 532 2 511 1 489 1 465 0 476 0
3 474 1 451 0 485 1 491 1 546 0
4 487 1 484 1 501 1 497 0 496 0
5 499 1 487 1 476 0 498 0 464 0
6 512 1 472 0 522 1 503 1 462 0
7 501 1 480 1 494 1 475 0 478 0
8 521 2 502 1 498 0 503 0 482 0
9 487 0 488 0 475 0 478 0 473 0
10 534 3 541 3 520 1 501 1 507 1
11 476 0 487 1 479 0 467 0 464 0
12 489 1 469 0 476 0 479 0 469 0
avr 504.5 1.25 494.6 0.92 492.3 0.58 486.7 0.25 481.0 0.08
W P M 1.21 - 1.24 - 1.24 - 1.26 - 1.27 -
8 Krzysztof Dobosz and Klaudiusz Stawski

6 Discuss
The time of the single key selection depends on number of the time periods
tp representing the interval of keyboard section change. Provided studies used
tp = 1500ms. This time period includes the time for the user’s decision about the
selection and the time of double blink tdb . Assuming the user needs full tp time
for decision and blinking, it gives i.e. 3tp to insert ”A”, and 7tp to insert ”R”,
because sequentially highlighted are: 3 rows, 2 groups (in the third row), 2 keys
(in the second group of the third row). Therefore, the arithmetic average time
of character insertion (including a space key) is 6.22 ∗ tp . However, it is 6.51 ∗ tp
for the pangram used for studies. After simple calculation we get the longest
faultless expected time of entry the pangram. It equals 498s (W P M = 1.23).
On the other hand the users can blink immediately when the required row,
group of keys or a key is highlighted. The sequence of keys is easy to remember
(in our research tool it is alphabetical order), the experienced user will know in
advance when he has to blink. Since each key is reached after three double blinks,
then three of tp periods can be shortened to 3tdb . Hence, i.e. the user needs 3tdb
to insert ”A”, and 4tp + 3tdb to insert ”R”. The duration of a single eye blink
is 100-400ms [1], double eye can be max. 800ms. During studies we assumed,
that 600ms will be sufficient tdb period for non-accidental double eye blink.
Repeating the calculation, this time the arithmetic average time of character
insertion is 3.22 ∗ tp + 3 ∗ tdb . However, for the pangram used for studies it is
3.51 ∗ tp + 3 ∗ tdb . Hence, we get the best possible time of the pangram entry
equals 360.3s, (W P M = 1.7).
Summarizing, expected users results of typing without words prediction should
be in the range 360.3-498 seconds. The only one test result (third user in forth
trial) belongs to this range (Table 1). However, we need to remember the users
were beginners without experience in such kind of HCI. They made many delays:
they selected incorrect keyboard sections by accidental double blink or thought
too long exceeding the time of highlighting. Considering this, the obtained re-
sults can be estimated as satisfactory. We can expect some improvement after
long term training.
Going further and having regard to experienced users, every tp period can
be set to tdb value. Then the text entry speed for the pangram can grow to
W P M = 3.15. It corresponds to 3.8 seconds per key selection. Such kind of
interaction does not need the EEG support in the form described in this paper.
However, such solution (with the very short time for user interaction) would
be hard to use resulting in many errors, and consequently would cause reduced
efficiency.

7 Conclusions
Proposed touchless keyboard uses eye double blinks. The obtained result during
evaluation is W P M = 1.11 with the MSD error rate equals 7.03%. In the area
of such kind of text entry systems, this result is good, although after theoretical
calculations we expected the value of non-error writing in the range of 1.23 − 1.7.
 Touchless Virtual Keyboard Controlled by Eye Blinking and EEG Signals 9

Controlling the keyboard with the blinks can be supported by EEG signals.
This is the novelty in the proposed approach. The results of experiments show
that the best mode is the mode using meditation threshold of 80. It achieves
efficiency of 1.27 W P M , but we should note that the words prediction was used.
Selecting the meditation as a parameter for the mode of predicted words, the
number of errors is lower in comparison to the experiment using attention. This
is due to the fact that when using the application, the user unwittingly focused
on the action, eg. the selection of the letters, which can inadvertently changes
the mode of application. Whereas a longer meditation above the threshold can
be easy achieved by closing eyes, which is done only for the intended purpose.
It is also possible to introduce improvements in the keyboard. The Divide and
Conquer algorithm could be used breaking down a set of keys into two subsets
every time. However, it would involve the reconfiguration of the number of rows
and columns, which could not be comfortable in use. Next, the order of keys
could be dependent on the frequency of corresponding characters in the national
language. Such HCI solution would require high activity of eyes, which could be
very uncomfortable for the user. During studies, after only first attempt, most
users complained of eye strain.

Acknowledgment

Publication financed by the Institute of Informatics at Silesian University of

Technology, statutory research no. 02/020/BK-16/0077

References
1. B10Numb3r5: Average duration of a single eye blink (Apr 2008), http://
bionumbers.hms.harvard.edu/bionumber.aspx?s=y&id=100706
2. Cecotti, H.: A self-paced and calibration-less ssvep-based brain–computer inter-
face speller. IEEE Transactions on Neural Systems and Rehabilitation Engineering
18(2), 127–133 (2010)
3. Chapin, J.K., Moxon, K.A., Markowitz, R.S., Nicolelis, M.A.: Real-time control of
a robot arm using simultaneously recorded neurons in the motor cortex. Nature
neuroscience 2(7), 664–670 (1999)
4. Duchowski, A.T.: Eye tracking methodology: Theory and practice. Springer (2017)
5. Hwang, H.J., Kim, S., Choi, S., Im, C.H.: Eeg-based brain-computer interfaces: a
thorough literature survey. International Journal of Human-Computer Interaction
29(12), 814–826 (2013)
6. Kang, K., Hong, J.: A framework for computer interface using eeg and emg. In:
Proceedings of the 2015 Conference on research in adaptive and convergent sys-
tems. pp. 472–473. ACM (2015)
7. Krapic, L., Lenac, K., Ljubic, S.: Integrating blink click interaction into a head
tracking system: implementation and usability issues. Universal Access in the In-
formation Society 14(2), 247–264 (2015)
8. Królak, A., Strumillo, P.: Eye-blink detection system for human–computer inter-
action. Universal Access in the Information Society pp. 1–11 (2012)
 10 Krzysztof Dobosz and Klaudiusz Stawski

9. Lin, C.T., Chen, Y.C., Huang, T.Y., Chiu, T.T., Ko, L.W., Liang, S.F., Hsieh,
H.Y., Hsu, S.H., Duann, J.R.: Development of wireless brain computer inter-
face with embedded multitask scheduling and its application on real-time driver’s
drowsiness detection and warning. IEEE Transactions on Biomedical Engineering
55(5), 1582–1591 (2008)
10. Lin, C.T., Lin, F.C., Chen, S.A., Lu, S.W., Chen, T.C., Ko, L.W.: Eeg-based
brain-computer interface for smart living environmental auto-adjustment. Journal
of Medical and Biological Engineering 30(4), 237–245 (2010)
11. MacKenzie, I.S., Soukoreff, R.W.: Phrase sets for evaluating text entry techniques.
In: CHI’03 extended abstracts on Human factors in computing systems. pp. 754–
755. ACM (2003)
12. Majaranta, P.: Communication and text entry by gaze. Gaze interaction and ap-
plications of eye tracking: Advances in assistive technologies pp. 63–77 (2012)
13. Marshall, D., Coyle, D., Wilson, S., Callaghan, M.: Games, gameplay, and bci:
the state of the art. IEEE Transactions on Computational Intelligence and AI in
Games 5(2), 82–99 (2013)
14. McFarland, D.J., Krusienski, D.J., Sarnacki, W.A., Wolpaw, J.R.: Emulation of
computer mouse control with a noninvasive brain–computer interface. Journal of
neural engineering 5(2), 101 (2008)
15. NeuroSky: Neurosky mindwave mobile eeg, http://store.neurosky.com/pages/
mindwave, (accessed: Dec 2016)
16. Obermaier, B., Muller, G.R., Pfurtscheller, G.: ” virtual keyboard” controlled by
spontaneous eeg activity. IEEE Transactions on Neural Systems and Rehabilitation
Engineering 11(4), 422–426 (2003)
17. Păsărică, A., Bozomitu, R.G., Cehan, V., Rotariu, C.: Eye blinking detection to
perform selection for an eye tracking system used in assistive technology. In: Design
and Technology in Electronic Packaging (SIITME), 2016 IEEE 22nd International
Symposium for. pp. 213–216. IEEE (2016)
18. Perego, P., Turconi, A., Andreoni, G., Gagliardi, C.: Cognitive ability assessment
by brain-computer interface ii: application of a bci-based assessment method for
cognitive abilities. Brain-Computer Interfaces 1(3-4), 170–180 (2014)
19. Scherer, R., Muller, G., Neuper, C., Graimann, B., Pfurtscheller, G.: An asyn-
chronously controlled eeg-based virtual keyboard: improvement of the spelling rate.
IEEE Transactions on Biomedical Engineering 51(6), 979–984 (2004)
20. Scott MacKenzie, I., Ashtiani, B.: Blinkwrite: efficient text entry using eye blinks.
Universal Access in the Information Society 10(1), 69–80 (2011)
21. Scott MacKenzie, I., Ashtiani, B.: Blinkwrite: efficient text entry using eye blinks.
Universal Access in the Information Society 10(1), 69–80 (2011)
22. Soukoreff, R.W., MacKenzie, I.S.: Measuring errors in text entry tasks: an appli-
cation of the levenshtein string distance statistic. In: CHI’01 extended abstracts
on Human factors in computing systems. pp. 319–320. ACM (2001)
23. Stamps, K., Hamam, Y.: Towards inexpensive bci control for wheelchair navigation
in the enabled environment-a hardware survey. Brain informatics 6334, 336–345
(2010)
24. TAN, D., Nijholt, A.: Brain-computer interaction: Applying our minds to human-
computer interaction (2010)

View publication stats