Brain Computer Interfaces (BCI) are devices that use brain signals for control or communication. Since they don’t require movement of any part of the body, BCI are the natural choice for assisted communication when a person is unable to move.
In this article, BCI based communicator for persons in locked-in state is described. It is based on P300 brain response of the user, thus does not require prior training, movement or imagination of movement. Auditory paradigm is selected in order to apply the communicator in cases where visual ability is also impaired. The communicator was designed to prove also whether low cost hardware with reduced electrode set could be used efficiently in everyday environment, without the need for expert personnel.
The design of the communicator is described first, followed by detailed analyses of the performance when used by either healthy or disabled subjects. It is shown that auditory paradigm is the primary factor that limits the accuracy of communication. Hardware characteristics and reduced electrode set influence the accuracy in a negative way as well, while different questions and answer types produce no major differences.
Locked-in syndrome (LIS) is the state in which a person can not consciously control his or her own body, nor it can communicate in any way with others (Wolpaw
Brain Computer Interfaces (BCI) use neurological signals originating in the brain to control external devices or computers (Wolpaw Tobii AB, Karlsrovägen 2D, S-182, 53 Danderyd, Sweden.
Researches of BCIs based on EEG signals are primarily using slow cortical potentials (SCPs) (Birbaumer
Farwell and Donchin (
In this paper, a BCI based communicator for persons with LIS is described. It is an auditory based P300 design and therefore it can be used in most severe cases of LIS or complete LIS. The design of the communicator is described in the next section, while analyses of system performance is provided in Section
In our design of EEG based BCI for person with LIS, we use auditory based P300 “speller”, where the user responds to 5 different words. We have 2 types of questions: “yes/no” questions, and questions with names of the persons, cities, etc. In “yes/no” questions, “yes” or “no” could be desired (“target”) answer, while 3 more “dummy” (“non-target”) words are added in order to resemble oddball paradigm. The words were chosen with an intention to keep the duration of the words short and similar. In questions with names, one name is desired or the “target” name, while the others are “non-target” names. More than two answers were used since it was established that P300 amplitude is inversely related to the relative probability of the evoking stimulus, and directly related to its task relevance (Kachenoura
During the experiment the possible answers or stimuli were normalized and played at a moderate sound level, with an inter stimulus interval of 1 second, and the stimuli were randomly shuffled in a segment. This inter stimulus interval is much longer than the interval usually used in visual experiments (around 300 ms). This is an obvious disadvantage of auditory paradigm, since long inter stimulus interval increases the time needed to obtain the answer. However, in the case of persons with LIS, the primary scope of the communicator is the establishment of elementary communication or even the determination of the state of consciousness, while the speed of communication is of secondary importance. Total time for reproducing answers for one question was around 3 minutes.
“Yes/no” questions are obviously the simplest ones, providing for binary information only. Questions with names are added for two reasons. First, it allows the multiple choice (e.g. questions like “Which part of your body hurts” or “What would you like to drink/eat?” could be asked in the future). Secondly, in cases when it is necessary to establish the state of consciousness of the person, familiar names could elicit response more easily than non-familiar names (Schnakers
The EEG signals were obtained with Emotiv’s Epoc EMOTIV, Inc., San Francisco, CA 94102 USA.
Emotiv Epoc has 14 bit resolution, EEG data is internally sampled at 2048 Hz and then down sampled to 128 Hz, signal bandwidth is 0.2–43 Hz, and digital notch filters at 50 Hz and 60 Hz are used. The EEG headset consists of 14 gold plated electrodes which make direct contact with the detachable electrode tips. These electrode tips have foam which is soaked in saline solution and sits on the scalp, adapted to the scalp topology. The saline solution acts as a good conductive medium for small voltage fluctuations. The electrodes are pre-attached to the headset with fixed arms to point at the locations: AF3, F3, F7, FC5, T7, P7, O1, O2, P8, T8, FC6, F8, F4, AF4, Common Mode Sense (CMS) at the left mastoid M1 and Driven Right Leg (DRL) at right mastoid M2. Since some electrode positions typically used in P300 detection (e.g. Cz, electrodes in the parietal region) are not available, tests are conducted in order to verify whether the present configuration of electrodes could produce reliable results.
Data acquisition and processing is done in OpenVIBE software platform (Renard
Linear Discriminant Analysis (LDA) classifier (Hoffmann
However, this BCI communicator uses auditory paradigm which is less reliable than visual speller paradigm (Lopez-Gordo
Only the basic idea behind the algorithm is explained, while more detailed explanations can be found in Rivet
The subjects were briefed about the experiment and asked to keep the movements at minimum to avoid artifacts. They were asked to focus on a fixed point, to make sure that there are minimum eye movement artifacts.
The experiment has a training phase and an operational phase. During the training phase, the subject was asked a question the answer of which was known a priori, and such data were used to train both the spatial filter and the classifier. The training phase was conducted only once at the beginning of the session. During the operational phase the responses were classified in real time. The subject was asked questions and the classifier would classify the data based on the subject’s selection. At the end of each trail healthy subjects were asked to verify the answer, while the disabled subjects were asked only those questions that had as a priori known answers.
A series of 5 sessions was conducted with the same subject on different days, and each session comprised of 5 questions. Once a question was asked, the 5 answers were played by the system in the form of 30 random segments, in such a way that the last stimulus of the previous segment was never the same as the first stimulus of the next segment. This and shuffling of stimuli in the segment ensured that the subject was not able to predict the played stimuli. As there were 30 audio events for each kind of answers, there were a total of 150 stimuli per question which means 150 epochs were considered for classification.
The system was tested on 39 participants, 7 females and 32 males, aged between 22 and 29. 25 participants were healthy subjects that did not suffer any past physical, neurological or psychiatric disorder, while 14 (12 male and 2 female) participants had some form of disability, varying from highly disabled with no motoric control, to locked in (multiple sclerosis, ALS, traumatic brain injury, dystrophy). On average 6 trials (questions) were recorded per each subject (Table
Tests performed for each subject. Number of tests performed with different modes of attention “enhancers” (mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt)), type of EEG device used and type of questions used are given. Conditions: Traumatic Brain Injury (TBI), Amyotrophic Lateral Sclerosis (ALS), Neuronal Ceroid Lipofuscinosis 2 (CLN2), Muscular Dystrophy (MD), Locked-In Syndrome (LIS).
Subject | Condition | No. of rec. | ma | mc | fc | tt | mt | Epoc/Quickamp | “yes-no”/“names” |
H-LBZ | – | 2 | 0 | 2 | 0 | 0 | 0 | Quickamp | 2/0 |
H-LBN | – | 5 | 0 | 5 | 0 | 0 | 0 | Epoc | 5/0 |
H-IM | – | 2 | 0 | 2 | 0 | 0 | 0 | Epoc | 2/0 |
H-DM | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-DS | – | 10 | 2 | 2 | 2 | 2 | 2 | Epoc | 5/5 |
H-DM | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-FS | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
D-IO | TBI | 4 | 4 | 0 | 0 | 0 | 0 | Epoc | 2/2 |
H-IP | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-KJ | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-KB | – | 8 | 2 | 2 | 0 | 2 | 2 | Epoc | 4/4 |
H-LB | – | 10 | 2 | 2 | 2 | 2 | 2 | Epoc | 5/5 |
H-LR | – | 10 | 2 | 2 | 2 | 2 | 2 | Epoc | 5/5 |
H-ZD | – | 9 | 1 | 2 | 2 | 2 | 2 | Epoc | 5/4 |
D-ZF | ALS | 4 | 2 | 2 | 0 | 0 | 0 | Epoc | 2/2 |
H-VJ | – | 10 | 2 | 2 | 2 | 2 | 2 | Epoc | 5/5 |
D-SD | CLN2 | 5 | 5 | 0 | 0 | 0 | 0 | Epoc | 2/3 |
H-ST | – | 10 | 2 | 2 | 2 | 2 | 2 | Epoc | 5/5 |
H-SK | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-NA | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-RN | – | 10 | 3 | 2 | 2 | 2 | 1 | Epoc | 6/4 |
H-MJ | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-MK | – | 8 | 2 | 2 | 0 | 2 | 2 | Epoc | 4/4 |
H-MP | – | 8 | 2 | 2 | 2 | 2 | 0 | Epoc | 4/4 |
H-AGZ | – | 4 | 0 | 4 | 0 | 0 | 0 | Quickamp | 3/1 |
H-CZ | – | 5 | 2 | 3 | 0 | 0 | 0 | Quickamp | 3/2 |
H-FZ | – | 6 | 3 | 3 | 0 | 0 | 0 | Quickamp | 4/2 |
H-MZ | – | 6 | 3 | 3 | 0 | 0 | 0 | Quickamp | 4/2 |
D-IO2 | TBI | 5 | 5 | 0 | 0 | 0 | 0 | Epoc | 3/2 |
D-LC | MD | 2 | 2 | 0 | 0 | 0 | 0 | Epoc | 1/1 |
D-MK | MS | 4 | 2 | 2 | 0 | 0 | 0 | Epoc | 2/2 |
D-ML | LIS | 4 | 2 | 2 | 0 | 0 | 0 | Epoc | 2/2 |
D-MZ | TBI | 4 | 0 | 4 | 0 | 0 | 0 | Epoc | 1/3 |
D-MJ | MS | 4 | 0 | 4 | 0 | 0 | 0 | Epoc | 2/2 |
D-ND | MS | 4 | 0 | 4 | 0 | 0 | 0 | Epoc | 2/2 |
D-PC | MD | 3 | 0 | 3 | 0 | 0 | 0 | Epoc | 2/1 |
D-SC | MD | 4 | 0 | 4 | 0 | 0 | 0 | Epoc | 2/2 |
D-VS | LIS | 3 | 0 | 3 | 0 | 0 | 0 | Epoc | 2/1 |
D-MS | LIS | 5 | 0 | 5 | 0 | 0 | 0 | Epoc | 2/3 |
Disabled subjects had various types of motor impairments. Not all of them were in LIS, but our primary goal was to test proposed communicator in real world situations (in not ideal situations, e.g. subjects unable to control movements, not fully accessible head positions, lying positions, etc.).
In the following, True Positive rate (TP), False Positive rate (FP) and Accuracy are given. The results are obtained from Open WIBE classifier. TP rate is calculated as a ratio of the number of positive answers correctly classified and the total number of positive answers. FP rate is similarly calculated as a ratio of the negative answers incorrectly classified as positive and the total number of negative answers. Accuracy is calculated as a ratio of the sum of true answers (True Positive and True Negative) and the total sum (Positive and Negative).
Overall average accuracy of classification during the operational phase was 75%. This average accuracy is lower than one reported for visual P300 speller (Sellers and Donchin,
Table
Low ITR are obtained, but that was expected since our primary goal is to develop a BCI system that can be used to establish basic communication in cases were more efficient communicator types cannot be used.
Number of recordings, minimum, average, maximum accuracy, standard deviation of the accuracy and information transfe rate (ITR) for each subject during testing. Prefix H in subject code indicates healthy subjects, while D indicates disabled ones.
Subject | No. of recordings | min acc | max acc | avg | stdev | ITR |
H-LBZ | 2 | 77.0 | 80.5 | 78.8 | 2.5 | 1.2 |
H-LBN | 5 | 73.6 | 83.0 | 76.1 | 3.9 | 1.0 |
H-IM | 2 | 75.4 | 76.0 | 75.7 | 0.5 | 1.0 |
H-DM | 8 | 74.7 | 90.4 | 80.7 | 4.8 | 1.2 |
H-DS | 10 | 69.6 | 77.6 | 73.7 | 2.9 | 1.0 |
H-DM | 8 | 71.6 | 80.1 | 76.1 | 2.4 | 1.1 |
H-FS | 8 | 70.1 | 77.1 | 74.3 | 2.0 | 1.0 |
D-IO | 4 | 72.0 | 76.0 | 74.8 | 1.9 | 1.0 |
H-IP | 8 | 73.2 | 78.1 | 76.2 | 1.7 | 1.1 |
H-KJ | 8 | 70.0 | 79.6 | 74.9 | 3.5 | 1.0 |
H-KB | 8 | 65.1 | 78.4 | 75.4 | 4.3 | 1.0 |
H-LB | 10 | 68.8 | 78.5 | 74.0 | 2.8 | 1.0 |
H-LR | 10 | 69.6 | 78.8 | 73.3 | 3.0 | 0.9 |
H-ZD | 9 | 72.3 | 78.8 | 75.2 | 2.2 | 1.0 |
D-ZF | 4 | 70.1 | 78.3 | 74.7 | 4.0 | 1.0 |
H-VJ | 10 | 69.9 | 82.0 | 75.9 | 3.0 | 1.0 |
D-SD | 5 | 71.2 | 76.1 | 74.3 | 2.1 | 1.0 |
H-ST | 10 | 64.7 | 79.3 | 73.9 | 4.4 | 1.0 |
H-SK | 8 | 71.6 | 83.1 | 77.8 | 4.2 | 1.1 |
H-NA | 8 | 73.9 | 79.1 | 75.4 | 1.6 | 1.0 |
H-RN | 10 | 68.5 | 80.1 | 74.5 | 3.4 | 1.0 |
H-MJ | 8 | 69.7 | 78.7 | 73.9 | 2.8 | 1.0 |
H-MK | 8 | 70.0 | 80.9 | 75.4 | 3.5 | 1.0 |
H-MP | 8 | 73.2 | 78.8 | 75.0 | 2.0 | 1.0 |
H-AGZ | 4 | 74.1 | 83.9 | 79.8 | 4.4 | 1.2 |
H-CZ | 5 | 76.3 | 89.3 | 80.7 | 5.0 | 1.2 |
H-FZ | 6 | 78.1 | 85.1 | 80.7 | 2.4 | 1.2 |
H-MZ | 6 | 76.5 | 80.5 | 78.9 | 1.4 | 1.2 |
D-IO2 | 5 | 71.9 | 78.2 | 74.7 | 2.7 | 1.0 |
D-LC | 2 | 72.3 | 75.5 | 73.9 | 2.3 | 1.0 |
D-MK | 4 | 70.7 | 74.9 | 72.7 | 2.0 | 0.9 |
D-ML | 4 | 68.4 | 78.1 | 72.6 | 4.1 | 0.9 |
D-MZ | 4 | 69.7 | 75.5 | 74.0 | 2.8 | 1.0 |
D-MJ | 4 | 72.4 | 79.6 | 75.7 | 3.0 | 1.0 |
D-ND | 4 | 68.7 | 75.8 | 72.4 | 3.0 | 0.9 |
D-PC | 3 | 72.3 | 78.0 | 75.4 | 2.9 | 1.0 |
D-PC | 4 | 68.7 | 77.9 | 74.8 | 4.2 | 1.0 |
D-VS | 3 | 70.6 | 72.9 | 72.0 | 1.2 | 0.9 |
D-MS | 5 | 71.3 | 78.8 | 75.2 | 2.9 | 1.0 |
Average | 71.5 | 79.3 | 75.5 | 2.9 | 1.0 |
As already mentioned in Section
Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for Emotiv Epoc and Quickamp.
Epoc | Quickamp | |
FP (%) | 12.6 ± 2.9 | 8.0 ± 3.1 |
TP (%) | 25.4 ± 6.1 | 31.7 ± 7.1 |
Accuracy (%) | 75.0 | 79.9 |
As it can be seen, Quickamp device with more electrodes, especially at central and parietal regions has higher accuracy by more than 4%. More detailed results of comparison between the two hardware amplifiers and electrode sets are given in Fig.
False positives versus true positives of the results acquired during classifier training in auditory P300 experiment with Emotiv Epoc (x) and BrainProducts QuickAmp devices (◇). The dotted line represents a random guess, a 50-50 guess-miss. Measurements above the line have a greater percentage of guess and measurements below it have a greater percentage of miss.
In order to perform comparison between audio and visual stimuli performance, seven additional healthy subjects were tested with visual P300 speller and EPOC device. Table
Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for auditory and visual experiments.
Auditory | Visual | |
FP (%) | 12.2 ± 3.2 | 2.4 ± 0.3 |
TP (%) | 26.0 ± 6.5 | 15.8 ± 4.4 |
Accuracy (%) | 74.4 | 84.0 |
As expected, visual paradigm has better accuracy, by more than 9%. There is however an interesting characteristic of auditory stimuli; the percentage of true positives is significantly greater than the same measure for visual stimuli. It means that in auditory case, the system is better to correctly classify P300 when it occurs. On the other side, during visual stimuli, misclassification of P300 in its absence occur on average in only 2.4% of cases. This is more clearly visible in Fig.
False positives versus true positives of the measurements acquired during classifier training in auditory (x) and visual (◇) P300 experiment. One trial in auditory experiment is data acquired for one question, and one trial in visual experiment is data acquired for 1 character.
Fig.
ROC curves for auditory (AUD) and visual (VIS) P300 experiment. Area under ROC curve is 0.61 for auditory experiment and 0.73 for visual experiment.
The BCI communicator was firstly tested on healthy subjects, and later the system was tested on 14 disabled subjects. Since primarily it was designed to help the disabled persons communicate with their caregivers, it was important to test its applicability in realistic situations. Table
The values presented in Table
Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for healthy and disabled persons.
Healthy | Disabled | |
FP (%) | 11.8 ± 3.3 | 13.3 ± 2.5 |
TP (%) | 26.5 ± 6.8 | 24.3 ± 5.1 |
Accuracy (%) | 75.8 | 74.2 |
False positives versus true positives of the measurements acquired during classifier training in auditory (x) and visual (◇) P300 experiment. One trial in auditory experiment is data acquired for one question, and one trial in visual experiment is data acquired for 10 letters.
In Section
Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for “yes/no” and “names” questions.
“yes/no” | “names” | |
FP (%) | 11.9 ± 3.3 | 12.4 ± 3.1 |
TP (%) | 26.3 ± 7.2 | 25.6 ± 5.5 |
Accuracy (%) | 75.7 | 75.2 |
False positives versus true positives of the measurements acquired during classifier training in auditory P300 experiment for “yes/no” (x) and “names” (◇) questions.
Finally, different modes of attention “enhancers” were tested in order to gain more insights in possible improvements of system performance. Mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt) were tested. Results given in Table
Mean values and standard deviations for false positives (FP), true positives (TP), and classification accuracy for different modes of attention “enhancers” (mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt)).
ma | mc | fc | tt | mt | |
FP (%) | 12.3 ± 2.6 | 12.0 ± 3.5 | 12.0 ± 3.8 | 12.5 ± 3.2 | 12.0 ± 1.9 |
TP (%) | 25.4 ± 5.4 | 25.9 ± 6.9 | 27.2 ± 8.8 | 25.7 ± 5.7 | 27.1 ± 4.0 |
Accuracy (%) | 75.3 | 75.5 | 75.8 | 75.1 | 75.9 |
False positives versus true positives of the measurements acquired during classifier training in auditory P300 experiment for different modes of attention “enhancers” (mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt)).
In this paper, a BCI communicator for persons with LIS is described. It is based on P300 ERPs and auditory paradigm which are suitable for most severe cases of LIS when even visual system cannot be used. The system is also based on low cost, portable, mobile and easy to mount hardware, in order to obtain a communicator for everyday use. With the present design choices, the system has several drawbacks, the major ones being the use of auditory paradigm which is slower and less reliable than the visual one, a smaller number of electrodes and the lack of electrodes in standard positions.
In order to test whether the performance of the system is satisfactory, several comparisons were performed and results were given. It was shown that the auditory paradigm and the selected hardware give lower scores when compared with the visual experiment and with a better hardware, but nonetheless the system still can be used. There is no major difference in the results provided by either healthy or disabled persons. Finally, different choices of questions and answer types have similar results, therefore either choice can be used.
Future improvements include the use of other signal processing techniques for blind source separation and possibly the inclusion of other types of EEG signals, in order to enhance the classification accuracy.