Seminar Report 2012

Brain Computer Interface

1. INTRODUCTION

Man machine interface has been one of the growing fields of research and development in recent years. Most of the effort has been dedicated to the design of userfriendly or ergonomic systems by means of innovative interfaces such as voice recognition, virtual reality. A direct brain-computer interface would add a new dimension to man-machine interaction. A brain-computer interface, sometimes called a direct neural interface or a brain machine interface, is a direct communication pathway between a human or animal brain(or brain cell culture) and an external device. In one BCIs, computers either accept commands from the brain or send signals to it but not both. Two way BCIs will allow brains and external devices to exchange information in both directions but have yet to be successfully implanted in animals or humans. In this definition, the word brain means the brain or nervous system of an organic life form rather than the mind. Computer means any processing or computational device, from simple circuits to silicon chips (including hypothetical future technologies like quantum computing). Research on BCIs has been going on for more than 30 years but from the mid 1990s there has been dramatic increase working experimental implants. The common thread throughout the research is the remarkable cortical-plasticity of the brain, which often adapts to BCIs treating prostheses controlled by implants and natural limbs. With recent advances in technology and knowledge, pioneering researches could now conceivably attempt to produce BCIs that augment human functions rather than simply restoring them, previously only the realm of science fiction.

1
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

Fig. 1.1: Schematic diagram of a BCI system

2
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

2. WORKING ARCHITECTURE
There are three types of BCI, these types are decided on the basis of the technique used for the interface. Each of these techniques has some advantages as well as some disadvantages. The three types of BCI are as follows with their features:

2.1 INVASIVE BCI: Invasive BCI are directly implanted into the grey matter of the brain during neurosurgery. They produce the highest quality signals of BCI devices . Invasive BCIs has targeted repairing damaged sight and providing new functionality to paralyzed people. But these BCIs are prone to building up of scar-tissue which causes the signal to become weaker and even lost as body reacts to a foreign object in the brain.

fig.2.1.1 : A man with acquired blindness using vision BCI

2.2 PARTIALLY INVASIVE BCI: Partially invasive BCI devices are implanted inside the skull but rest outside the brain rather than amidst the grey matter. They produce better resolution signals than noninvasive BCIs where the bone tissue of the cranium deflects and deforms signals and have a lower risk of forming scar-tissue in the brain than fully-invasive BCIs.

3
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

Electrocorticography(ECoG)

uses

the

same

technology

as

non-invasive

electroencephalography, but the electrodes are embedded in a thin plastic pad that is placed above the cortex, beneath the dura mater. ECoG technologies were first trialled in humans in 2004 by Eric Leuthardt and Daniel Moran from Washington University in St Louis. In a later trial, the researchers enabled a teenage boy to play Space Invaders using his ECoG implant. This research indicates that it is difficult to produce kinematic BCI devices with more than one dimension of control using ECoG.

2.3 NON INVASIVE BCI : As well as invasive experiments, there have also been experiments in humans using non-invasive neuroimaging technologies as interfaces. Signals recorded in this way have been used to power muscle implants and restore partial movement in an experimental volunteer. Although they are easy to wear, non-invasive implants produce poor signal resolution because the skull dampens signals, dispersing and blurring the electromagnetic waves created by the neurons. Although the waves can still be detected it is more difficult to determine the area of the brain that created them or the actions of individual neurons.

fig.2.3.1: Recordings of brainwaves produced by an electroencephalogram Electroencephalography(EEG) is the most studied potential non-invasive interface, mainly due to its fine temporal resolutions, ease of use, portability and low set-up cost. But as well as the technology's susceptibility to noise, another substantial barrier to using EEG as a brain-computer interface is the extensive training required before users can work the technology.In experiments beginning in the mid-1990s, the scientists used EEG recordings of slow cortical potential to give paralysed patients limited control over a computer cursor.The
4
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

EXPERIMENT saw ten patients trained to move a computer cursor by controlling their brainwaves. The process was slow, requiring more than an hour for patients to write 100 characters with the cursor, while training often took many months. Magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) have both been used successfully as non-invasive BCIs. In a widely reported experiment, fMRI allowed two users being scanned to play Pong in real-time by altering their haemodynamic response or brain blood flow through biofeedback techniques. fMRI measurements of haemodynamic responses in real time have also been used to control robot arms with a seven second delay between thought and movement.

5
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

3. THE CURRENT BCI TECHNIQUES

3.1 P300 Detection: Farwell [Farwell&Donchin 1988] of the Department of Psychology and Cognitive Psychophysiology Laboratory at the University of Illinois at Urbana-Champaign IL, describes a technique for detecting the P300 component of a subject's event-related brain potential (ERP) and using it to select from an array of 36 screen positions. The P300 component is a positivegoing ERP in the EEG with a latency of about 300ms following the onset of a rarely- occurring stimulus the subject has been instructed to detect. The EEG was recorded using electrodes placed at the Pz (parietal) site (10/20 International System), limited with band-pass filters to .02-35Hz and digitized at 50Hz. Electro-oculogram (EOG) data was also recorded from each subject via electrodes placed above and below the right eye. The "odd-ball" paradigm was used to elicit the P300, where a number of stimuli are presented to the experimental subject who is required to pay attention to a particular, rarely-occurring stimulus and respond to it in some non- motor way, such as by counting occurrences. Detecting the P300 response reliably requires averaging the EEG response over many presentations of the stimuli. The purpose of the current experiment was to discover the minimum number of presentations at two different inter-stimulus intervals (ISI) required to detect the P300 response. The experiment presented a 36-position array of letters, plus common typing characters and controls (e.g. space, backspace), made to flash in a random sequence first by rows and then columns. Each trial consisted of a complete set of six column or row flashes. Trials contaminated with muscular or EOG response were rejected and additional trials presented until data were collected from a block of 30 good trials, during which subjects were to fixate on a particular position, and count the number of times it flashed while a control message was elsewhere on the screen. After each block the fixated letter (one of B-R-A-I-N) was added to the screen so that subjects were conscious of slowly spelling out the word "BRAIN" through a succession of five blocks. A set

6
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

of five blocks was run at each ISI -- 125ms and 500ms. The two presentation rates were chosen to bracket a range of communication rates from a low of 30 averaged trials at 500ms ISI (93.6

seconds of presentation per character) to a high of one trial at 125ms (1.245 seconds of presentation per character), an effective communication rate range of .01 to .8 characters-persecond, respectively. The authors used four techniques to analyze the data for reliable P300 response detection stepwise descriminant analysis (SWDA), peak picking, area, and covariance, and identified SWDA as leading to the greatest accuracy at the fastest presentation rate. Results indicated that a character chosen from among 36 items can be detected with 95% accuracy within 26 seconds.

3.2. VEP Detection: This technique was reviewed by Sutter [Sutter 1992] at the Smith-Kettlewell Eye Research Institute in San Francisco CA, and Cilliers [Cilliers&VanDerKouwe 1993] and colleague at the Department of Electrical and Electronic Engineering, University of Pretoria, South Africa. Sutter describes presenting a 64-position block on a computer screen and detecting which block the subject looks at, while Cillier's work uses a series of four lights. In each case, several simultaneously presented stimuli are made to change rapidly in some controlled way(intensity, pattern, color-shift) and the subject has scalp electrodes placed over the visual cortex (back of the head) in a position to detect changes in the evoked potential(VEP) at that location. Sutter used a lengthy binary sequence to switch 64 screen positions between red and green, and in other trials to reverse a checkerboard pattern. Each screen position was shifted 20ms in the binary control sequence relative to its neighbors, and the entire sequence was auto correlated with the VEP in overlapping increments(the VEP response components last about 80ms) beginning 20ms apart, with the resultant vector stored in a 64-position array of registers. When a coefficient remains greater than all the others and above a threshold value for a certain amount of time, the corresponding stimulus is considered to have been selected. The 64 positions represent the letters of the alphabet and commonly used words in the English language. The subject can fixate on any word or letter. Whenever the subject fixates on a letter, the commonly used words change to words beginning with that letter, for quick selection of an entire word. Sutter suggests a need to optimize both electrode

7
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

placement and stimulation mode for each individual subject for good target discrimination. Seventy normal subjects evaluating a prototype system achieved adequate response times ranging from 1 to 3 seconds after an initial tuning process lasting 10-60 minutes. Sutter also tested his techniques on 20 severly disabled persons and describes an experimental version involving an ALS patient using intra-cranial electrodes implanted in the space between the dura and the skull. Cilliers' technique involves varying the intensity of four LED's modulated with a 10Hz sine wave in phase quadrature and detecting the signal in the subject's VEP using a pair of EEG surface electrodes placed on the occipital lobe. The four flashing LED's are arranged around the edge of a computer screen containing an image of a standard four-row keyboard with each row of keys in a different color. Each LED is associated with one of the colors. Fixating on one LED selects a key row, which is redisplayed in four colors for a more detailed selection. The subject can select any particular key in an average of three selections -about 15 seconds with the current setup. A short initial training period is required where subjects fixate on each LED for 5 seconds. Cilliers' paper describes work with a quadriplegic patient with a C2-level injury.

3.3. EEG Pattern Mapping: Several experimenters describe techniques for classifying, detecting and mapping EEG patterns. Pfurtscheller's technique used a neural net featuring learning-vector quantization (LVQ) to map EEG patterns during the 1-second interval before a signal the experimental subject was instructed to wait for. Hiraiwa [Hiraiwa et al 1993] used a backpropagation artificial neural network to study readiness potentials (RP's)-- patterns in the EEG immediately prior to the subject's uttering one of five different Japanese syllables or moving a joystick in one of four different directions. Twelve channels of EEG data taken >from scalpmounted electrodes at locations Fp1, Fp2,Fz, C3, C4, Pz, F5, F6, F7, F8, O1 and O2 (International 10/20 system) were used to train and then test two neural networks optimized for averaged data and for single-trial, real-time analysis, respectively. High recognition rates were obtained for the averaged data. Single- trial RP recognition, though less reliable, showed considerable promise in the experimenters' view. Keirn and Aunon [Keirn&Aunon 1990] recorded EEG data from scalp-mounted electrodes at five different locations during accomplishment of 5 different tasks during which subjects had their eyes open or closed, for 10 alternative responses. The tasks included:
8
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

(1) relaxing and trying to think of nothing, (2) a non-trivial multiplication problem, (3) a 30-second study of a drawing of a 3-dimensional object after which subjects were to visualize the object being rotated about an axis, (4) mental composition of a letter to a friend, and (5) visualize numbers being written on a blackboard sequentially, with the previous number being erased before the next was written.

3.4. Detecting lateral hemisphere differences: Drake [Drake 1993] studied induced lateral differences in relative brain hemisphere activation after subjects heard arguments through left, right or both earphones which they either strongly agreed with or strongly disagreed with, as determined by prior interviews. Subjects exhibited greater discounting of arguments they disagreed with during left hemisphere activation as measured by ratings of truth. Results supported previous work indicating asymmetries in lateral activation potential during processing persuasive arguments, however the study did not include measuring directly either activation levels or potentials in the cortex.

9
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

4. BRAIN GATE

fig.4.1:Dummy unit illustrating the design of a BrainGate interface BrainGate is a brain implant system developed by the bio-tech company Cyberkinetics in 2003 in conjunction with the Department of Neuroscience at Brown University. The device was designed to help those who have lost control of their limbs, or other bodily functions, such as patients with amyotrophic lateral sclerosis (ALS) or spinal cord injury. The computer chip, which is implanted into the brain, monitors brain activity in the patient and converts the intention of the user into computer commands. Currently the chip uses 100 hair-thin electrodes that sense the electro-magnetic signature of neurons firing in specific areas of the brain, for example, the area that controls arm movement. The activity is translated into electrically charged signals and are then sent and decoded using a program, which can move either a robotic arm or a computer cursor. In addition to real-time analysis of neuron patterns to relay movement, the Braingate array is also capable of recording electrical data for later analysis. A potential use of this feature would be for a neurologist to study seizure patterns in a patient with epilepsy. Existing technology stimulates muscle groups that can make an arm move. The problem Surgenor and his team faced was in creating an input or control signal. With the right control signal they found they could stimulate the right muscle groups to make arm movement.

10
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

4.1. DARPA: The Brown University group was partially funded by the Defence Advanced Research Projects Agency (DARPA), the central research and development organisation for the US Department of Defence (DoD). DARPA has been interested in Brain-MachineInterfaces (BMI) for a number of years for military applications like wiring fighter pilots directly to their planes to allow autonomous flight from the safety of the ground. Future developments are also envisaged in which humans could 'download' memory implants for skill enhancement, allowing actions to be performed that have not been learned directly.

11
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

5. BCI Applications
5.1. Introduction : After we go through the various techniques of BCI the first question that comes to our mind is, what does BCI do to us and what are its applications. So BCI in todays time turns useful to us in many ways. Whether it be any medical field or a field leading to enhancement of human environment. Some of the BCI applications are discussed below. 5.2. The Mental Typewriter: The initial project demonstrates how a paralysed patient could communicate by using a mental typewriter alone without touching the keyboard. In the case of serious accident or illness, a patients limbs can be paralyzed, severely restricting communication with the outside world. The interface is already showing how it can help these patients to write texts and thus communicate with their environment. Theres also a PONG game (computer tennis) used to demonstrate how the interface can be used. Brain Pong involves two BBCI users playing a game of teletennis in which the rackets are controlled by imagining movements and predictably the general media has focussed the majority of its attention on computer gaming applications but BCCI could equally be used in safety technologies (e.g. in automobiles for monitoring cognitive driver stress), in controlling prostheses, wheelchairs, instruments and even machinery. The Berlin Brain-Computer Interface (BBCI which uses the electrical activity of the brain in the form of an electroencephalogram (EEG) has electrodes attached to the scalp measure the brains electrical signals. These are then amplified and transmitted to the computer, which converts them into technical control signals. The principle behind the BBCI is that the activity of the brain already reflects the purely mental conception of a particular behaviour, e.g. the idea of moving a hand or foot.

12
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

The BBCI recognizes the corresponding changes in brain activity and uses them, say, to choose between two alternatives: one involves imagining that the left hand is moved; the other that the right hand is moved. This enables a cursor, for example, to be moved to the left or right. The person operating the mental typewriter uses the cursor to select a letters field. The next step reduces the choice, and after a few more steps we arrive at the individual letters, which can be used to write words. This process enables simple sentences to be constructed within minutes. A first prototype of the mental typewriter is currently available. In a series of experiments, different spelling methods are tested in terms of their usability and are adapted to the BBCI. It will be some years, though, before the mental typewriter can be used in everyday applications. Further research is needed, in particular to refine the EEG sensors. 5.3. BCI offers paralyzed patients improved quality of life: A braincomputer interface installed early enough in patients with neurondestroying diseases can enable them to be taught to communicate through an electronic device and slow destruction of the nervous system. Fundamental theories regarding consciousness, emotion and quality of life in sufferers of paralysis from Amyotrophic Lateral Sclerosis (ALS) are being challenged based on new research on brain-computer interaction. ALS is a progressive disease that destroys neurons affecting movement. The research focuses on a condition called the completely locked-in state (CLIS, a total lack of muscle control). In a CLIS situation, intentional thoughts and imagery can rarely be acted upon physically and, therefore, are rarely followed by a stimulus. The research suggests that as the disease progresses and the probability for an external event to function as a link between response and consequence becomes progressively smaller it may eventually vanish altogether. Researchers have found that by implementing a brain-computer interface before the completely locked-in state occurs, a patient can be taught to communicate through an electronic device with great regularity. The continued interaction between thought, response and consequence is believed to slow the destruction of the nervous system.

13
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

The findings are also raising a number of new questions about the quality of life amongst paralysis sufferers. Patients surveyed were found to be much healthier mentally than psychiatrically depressed patients without any life-threatening bodily disease. Only 9% of ALS patients showed long episodes of depression and most were during the period following diagnosis and a period of weeks after tracheotomy.

5.4. Adaptive BCI for Augmented Cognition and Action : The goal of this project is to demonstrate improved human/computer performance for specific tasks through detection of task-relevant cognitive events with real-time EEG. For example, in tasks for which there is a direct tradeoff between reaction time and error rate, (such as typing or visual search) it may be beneficial to correct a users errors without interrupting the pace of the primary task. Such a user interface is possible through the direct detection of EEG signatures associated with the perception of an error, often referred to as Error Related Negativity. In general such signatures may be used to dynamically adjust the behavior of human-computer interfaces and information displays. This project advances signal analysis techniques for high density EEG to detect discrete events associated with cognitive processing. Corresponding real-time adaptive interfaces with sub-second latency are being designed to evaluate this concept of an adaptive brain-computer interface in three specific applications: 5.4.1 Error and conflict perception: Error related negativity (ERN) in EEG has been linked to perceived response errors and conflicts in decision-making. In this project we have developed single trial ERN detection to predict task-related errors. The system can be used as an automated real-time decision checker for time-sensitive control tasks. In the first phase of this project we demonstrated improved human/computer performance at a rapid forced choice discrimination task with an average 23% reduction of human errors. This open-loop error correction paradigm represents the first application of real-time cognitive event detection and demonstrates the utility of real-time EEG brain monitoring within the Augmented Cognition program. We will evaluate video game scenarios with closed-loop feedback at latencies of less

14
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

than 150 ms where detected errors are corrected or application parameters such as speed are varied according to the measured or "gauged" conflict perception. 5.4.2 Working memory encoding: Transient modulation of oscillations in the theta (4-8 Hz) and gamma (20-30 Hz) bands, recorded using EEG and magneto encephalography (MEG), have been implicated in the encoding and retrieval of semantic information in working memory. In this project we will exploit these neural correlates of semantic processing to detect problems with semantic information processing. This memory gauge could be used to detect memory recall deficits, and repeat or enhance the presented information and thus better prime memory recall. 5.4.3 Rapid visual recognition: We are exploring the signals elicited by visual target detection, which were recently observed in rapid sequential visual presentation (RSVP) experiments. We have demonstrated that the detection of these signals on a single trial basis can be used to replace the slow manual response of a human operator, thereby significantly increasing the throughput of image search tasks. This paradigm has the potential to improve the performance of Image Analysts who need to routinely survey large volumes of aerial imagery within short periods of time. In addition, the approach looks to measure the "bottleneck" between constant delay perceptual processing and more variable delay cognitive processing. Thus the detected signatures can be used to "gauge" if cognitive systems are capable/incapable of assimilating perceptual input for fast decision making. In the first phase of this project a fully automated real-time signal analysis system and hardware infrastructure has been developed that can give short latency feedback to the user within 50ms of the recorded activity. The signal processing system adaptively learns to detect evoked responses from the real-time streaming EEG signal. The current system, which is used for some tasks can be configured for single trial detection for any number of cognitive events such ERN, rapid visual recognition, readiness potential, response to oddball stimulus (P300), as well as conventional visual, auditory, or somato-sensory responses. We are in the progress of applying this system to event detection in the Warship Commander - a common task set proposed for integration and evaluation by the Augmented Cognition Program.

15
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

fig.5.4.1: Real-time brain computer interface system for augmented cognition and action.

The information delivery to the human is adapted to a user's cognitive processing based on cognitive events detected in real-time high-density EEG. Applications include automatic correction of perceived errors, prediction of memory performance, and rapid visual search. In the experimental system a pipelined modular processing architecture is used to collect EEG data, increase the signal-to-noise ratio (SNR), and generate a control signal that is fed back to the subject via a display. As an example consider the task of fast image search. A rapid sequence of images is presented on the display. The subject views the images with the goal of detecting a target image. The EEG signal from the high-density sensor net is sampled and processed in real-time using algorithms for artifact removal, and noise reduction. The signal is analyzed in real-time to identify the cognitive activity associated with visual target detection. The augmentation manager records the images associated with recognition events. This information is used to prioritize the large volumes of imagery that has to be analyzed. The
16
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

selected images can subsequently be presented for more careful analysis without interrupting the fast visual processing of the human subjects in the initial scan. In Phase 1 of the project it has been demonstrated that improved prioritization performance is obtained as compared to selecting the image with a manual button push.

17
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

6. Experimental Brain Computer Interface Software for the Modular EEG (The ABI software)
6.1. Introduction: ABI is a simple software for the Modular EEG that implements an experimental Brain Computer Interface (BCI). Nowadays, BCI research is a highly active field, but the existing technology is still immature for its use outside of a lab's settings. The ABI software tries to provide a simple tool for hobbyists to do experiments on its own with BCI technology. 6.2. Work of the software: The ABI is a BCI based on trials. A trial is a time interval where the user generates brainwaves to perform an action. The BCI tries to process this signal and to associate it to a given class. The association is done by feeding a neural net with the preprocessed EEG data. The neural net's output is then further processed and this final output corresponds to the given class. The neural net should be trained in order to learn the association. The ABI software allows you to 1) Do simple Biofeedback. You can display raw EEG channels, narrow band frequency amplitudes and classes. 2) Simulate trials. 3) Record trials for a number of choices of different classes. 4) Train the interface. ABI has three operating modes: SIMULATION, RECORDING and TRAINING. You can switch between operating modes by pressing F1, F2 or F3 respectively (the software doesn't change its mode instantly, because a trial shouldn't be interrupted in the middle). The operation is quite simple. The user records several trials for the different classes (RECORDING mode). Each class is associated to a different mental task. After recording a reasonable amount of trials (more than 50 trials for each class), the user can train the system to learn a way to discriminate between the different classes (TRAINING mode). This process

18
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

can be repeated in order to improve the quality of the recognition. The system can be tested under the SIMULATION mode.

Simulation and Recording modes perform single trials. The SIMULATION mode is used to test the BCI. RECORDING is the same as SIMULATION, with the difference that the EEG data is recorded and used as training examples. Pressing the F3 key, the system starts to train the neural net with the available data (Training mode). The training set used for this purpose is the set of the last Trial Buffer recorded trials' features.

19
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

CONCLUSION
Brain-Computer Interface (BCI) is a method of communication based on voluntary neural activity generated by the brain and independent of its normal output pathways of peripheral nerves and muscles. The neural activity used in BCI can be recorded using invasive or noninvasive techniques. We can say as detection techniques and experimental designs improve, the BCI will improve as well and would provide wealth alternatives for individuals to interact with their environment.

20
GEC MCA Thrissur

Seminar Report 2012

Brain Computer Interface

REFERENCES
1) Toward a P300-based Computer Interface James B. Polikoff, H. Timothy Bunnell, & Winslow J. Borkowski Jr. 2) Pearlmutter B.A, Ward T.E, Matthews.F and Markham Hemodynamics for Brain-Computer Interfaces. Signal Processing Magazine,IEEE, 2008. 3) Synchronous neural interactions assessed by magnetoencephalography: A functional biomarker for brain disorders. Signal Processing Magazine,IEEE, Dec 2007. 4) Various papers from PASCAL 2004 5) The future of brain-controlled devices. ACM, April 2010.

21
GEC MCA Thrissur