In our design of EEG based BCI for person with LIS, we use auditory based P300 “speller”, where the user responds to 5 different words. We have 2 types of questions: “yes/no” questions, and questions with names of the persons, cities, etc. In “yes/no” questions, “yes” or “no” could be desired (“target”) answer, while 3 more “dummy” (“non-target”) words are added in order to resemble oddball paradigm. The words were chosen with an intention to keep the duration of the words short and similar. In questions with names, one name is desired or the “target” name, while the others are “non-target” names. More than two answers were used since it was established that P300 amplitude is inversely related to the relative probability of the evoking stimulus, and directly related to its task relevance (Kachenoura et al., 2008).

During the experiment the possible answers or stimuli were normalized and played at a moderate sound level, with an inter stimulus interval of 1 second, and the stimuli were randomly shuffled in a segment. This inter stimulus interval is much longer than the interval usually used in visual experiments (around 300 ms). This is an obvious disadvantage of auditory paradigm, since long inter stimulus interval increases the time needed to obtain the answer. However, in the case of persons with LIS, the primary scope of the communicator is the establishment of elementary communication or even the determination of the state of consciousness, while the speed of communication is of secondary importance. Total time for reproducing answers for one question was around 3 minutes.

“Yes/no” questions are obviously the simplest ones, providing for binary information only. Questions with names are added for two reasons. First, it allows the multiple choice (e.g. questions like “Which part of your body hurts” or “What would you like to drink/eat?” could be asked in the future). Secondly, in cases when it is necessary to establish the state of consciousness of the person, familiar names could elicit response more easily than non-familiar names (Schnakers et al., 2008) and therefore they can serve in the initial phase of using the communicator by the person with complete LIS. Also, several types of attention enhancement were tried. In Schnakers et al. (2008) it was established that in cases when persons were instructed to mentally count the number of occurrence of “target”, the P300 response was more pronounced with respect to cases where subjects were instructed to give a “mental counting”. In order to test other options, finger counting, tapping and mental tapping have been tested as well. In finger counting type of response the subject was asked to finger count the correct answers, and in tapping the subject was supposed to press the key (on regular PC keyboard), using only one finger, every time he heard the correct answer. In mental tapping type of response, the subject was asked to imagine him tapping the key on a keyboard. Although the options involving the movement of hand or fingers are not options for persons with LIS, testing was done in order to establish whether these types of response have any potential for improvement of P300 detection.

The EEG signals were obtained with Emotiv’s Epoc2 EEG headset. This EEG device was selected due to its low cost and mobility but also because of simple and fast electrodes placement suitable to everyday use, while maintaining the 10–20 international standards of electrode placement. The data recorded using the headset are of good EEG quality and comparable to the data obtained with research grade devices (Badcock et al., 2013). Nonetheless, the comparison between the results obtained with Emotiv Epoc and the research grade device has been conducted, and provided in Section 4.

Emotiv Epoc has 14 bit resolution, EEG data is internally sampled at 2048 Hz and then down sampled to 128 Hz, signal bandwidth is 0.2–43 Hz, and digital notch filters at 50 Hz and 60 Hz are used. The EEG headset consists of 14 gold plated electrodes which make direct contact with the detachable electrode tips. These electrode tips have foam which is soaked in saline solution and sits on the scalp, adapted to the scalp topology. The saline solution acts as a good conductive medium for small voltage fluctuations. The electrodes are pre-attached to the headset with fixed arms to point at the locations: AF3, F3, F7, FC5, T7, P7, O1, O2, P8, T8, FC6, F8, F4, AF4, Common Mode Sense (CMS) at the left mastoid M1 and Driven Right Leg (DRL) at right mastoid M2. Since some electrode positions typically used in P300 detection (e.g. Cz, electrodes in the parietal region) are not available, tests are conducted in order to verify whether the present configuration of electrodes could produce reliable results.

Data acquisition and processing is done in OpenVIBE software platform (Renard et al., 2010). The OpenVIBE’s acquisition client is used to obtain raw EEG data and the algorithm for EEG data processing was built using the OpenVIBE Designer. Apart from being a free open source software, OpenViBE allows us to run the classifier and the whole BCI software in real time as the data is getting acquired from the EEG headset. Data preprocessing includes additional filtering using a 4th order Butterworth band-pass filter with a lower cutoff frequency of 1 Hz and a higher cutoff frequency of 20 Hz, with a pass band ripple of 0.5 dB, and down sampling by a factor of 4. An epoch, a slice of 1 second of data after stimulus onset, is considered in following analyses. No artifact removal was done in this stage, although it can improve the overall performance of the system (Minguillon et al., 2017). Manual artifact removal was not done since the proposed communicator should work in everyday situations where data analysis experts are not available. Automatic artifact removal was not applied since all the data processing needs to be done in real time. Also, answers are repeated 30 times in order to be less affected by discarding some of the segments due to artefacts.

Linear Discriminant Analysis (LDA) classifier (Hoffmann et al., 2008) is used to correctly classify responses of the user. LDA classifier computes a discriminant vector that separates the “target” and “non-target” classes. LDA classifier is simple to use because it is nonparametric. Also, it was established that LDA classifier gives good results for P300 classification (Krusienski et al., 2008).

However, this BCI communicator uses auditory paradigm which is less reliable than visual speller paradigm (Lopez-Gordo et al., 2012; Klobassa et al., 2009) and the used headset has a reduced number of electrodes. Therefore, in order to obtain an acceptable accuracy of classification, it is needed to further enhance and preprocess the data prior to the classification. There are numerous contributions dealing with spatial filtering (Rivet et al., 2009; Rivet and Souloumiac, 2013) and blind source separation (Wolpaw et al., 2002; Albera et al., 2008; Xu et al., 2004; Bayliss and Ballard, 1999; Hill et al., 2004; Wang and James., 2006; James and Hesse, 2005) for P300 speller designs, and in this case spatial filtering based on xDAWN algorithm was used (Rivet and Souloumiac, 2013). xDAWN algorithm is specially designed for P300 speller paradigm.

Only the basic idea behind the algorithm is explained, while more detailed explanations can be found in Rivet et al. (2009), Rivet and Souloumiac (2013). The basic idea is to automatically estimate P300 subspace from raw EEG signals. P300 evoked potentials are then enhanced by projecting raw EEG on the estimated P300 subspace. The algorithm derives its name from the following model for the recorded data X: where $X\in {\mathcal{R}^{({N_{t}}\times {N_{s}})}}$ is the matrix of recorded EEG signals, ${N_{s}}$ is the number of sensors, ${N_{t}}$ is the number of temporal samples, $D\in {\mathcal{R}^{({N_{t}}\times {N_{e}})}}$ is the Toeplitz matrix defined from the set of stimuli onsets and estimated durations of the ERP, ${N_{e}}$ is the number of temporal samples of the ERP (1 s in our case), and matrix N is the on-going activity of the user’s brain as well as artifacts. DA represents the synchronous response with target stimuli. The least square estimation of response A, $\hat{A}={({D^{T}}D)^{(-1)}}{D^{T}}X$, is different than classical epoching of matrix X when synchronous response A extents over several consecutive stimuli. That is generally the case for visual P300 spellers, but not in our case, where inter stimulus interval is 1 s, which excludes the possibility of overlapping responses. xDAWN algorithm estimates ${N_{f}}$ spatial filters (with ${N_{f}}<{N_{s}}$) in order to enhance the synchronous response: where $U\in {\mathcal{R}^{({N_{s}}+{N_{f}})}}$ is the spatial filter matrix. The idea is similar to principal component analysis (PCA) of $\hat{A}$, where recorded signals are projected on the ${N_{f}}$ main components associated with the ${N_{f}}$ largest principal values. In xDAWN algorithm, the spatial filters U are designed in order to maximize signal to signal plus noise ratio. Finally, the model (1) can be rewritten as: where ${A^{\prime }}$ is synchronous response of reduced dimension, W is its spatial distribution over sensors, and ${N^{\prime }}$ is the noise term. The enhanced signals are then computed by: ${N^{\prime\prime }}$ is the matrix that includes noise and artifacts, transformed by spatial filtering. Spatial filters can be computed by two QR factorizations and one singular value decomposition, or by using generalized eigenvalue decomposition of pair of covariance matrices (Rivet and Souloumiac, 2013).

As already mentioned in Section 2.2, Emotiv Epoc was selected due to its low cost, portability and ease of use, while its major disadvantage was the reduced number of electrodes. Therefore, the same system was tested with a different hardware and electrode set, Quick amp with 32 gel electrode set, and 24 bit A/D converter. In this case, there were more electrodes in the central and the parietal region, which could be beneficial to P300 detection. Table 3 gives the classification accuracy for the two devices, as well as the percentage of true positives (TP) and false positives (FP). Both mean values and standard deviations are given for TP and FP.

As it can be seen, Quickamp device with more electrodes, especially at central and parietal regions has higher accuracy by more than 4%. More detailed results of comparison between the two hardware amplifiers and electrode sets are given in Fig. 1. It shows the false positives versus true positives scores of measurements acquired during classifier training in auditory P300 experiment with Emotiv Epoc and BrainProducts QuickAmp devices. For each trial or question, 5 fold cross validation was done and the average values of all cross validation results for each question are given in Fig. 1. Ideally, the classification should give results located at the left upper position, corresponding to TP of 1 (or 100%) and FP of 0 (0%).

In order to perform comparison between audio and visual stimuli performance, seven additional healthy subjects were tested with visual P300 speller and EPOC device. Table 4 gives classification accuracy for auditory paradigm and visual P300 speller, as well as the percentage of true positives (TP) and false positives (FP). Both mean values and standard deviations are given for TP and FP. For visual P300 speller, $6\times 6$ matrix was used in standard set-up: a single row or column was randomly illuminated for 200 ms, and inter stimulus interval was 300 ms. Visual P300 speller is much faster than the proposed auditory paradigm and therefore the number of epochs is greater. All other parameters for preprocessing, xDAWN algorithm and classifier have not changed. Visual P300 speller usually has higher accuracy than one obtained here, but to obtain higher accuracy a greater number of electrodes has to be employed.

As expected, visual paradigm has better accuracy, by more than 9%. There is however an interesting characteristic of auditory stimuli; the percentage of true positives is significantly greater than the same measure for visual stimuli. It means that in auditory case, the system is better to correctly classify P300 when it occurs. On the other side, during visual stimuli, misclassification of P300 in its absence occur on average in only 2.4% of cases. This is more clearly visible in Fig. 2. This was expected, since it was reported by the examined persons that subjectively it was much easier to spatially focus attention in the visual experiment than in the auditory experiment.

Fig. 3 gives Receiver Operating Characteristic curves for targets, for auditory (left) and visual (right) P300 experiment. Upper left corner represents a perfect classifier with no false positives (100% specificity) and no false negatives (100% sensitivity). Again, it is visible that ROC curve for visual experiment is closer to upper left corner of the ROC space, thus indicating better performance for the visual experiment.

The BCI communicator was firstly tested on healthy subjects, and later the system was tested on 14 disabled subjects. Since primarily it was designed to help the disabled persons communicate with their caregivers, it was important to test its applicability in realistic situations. Table 5 gives the comparison between healthy and disabled subjects.

The values presented in Table 5 were obtained in same way as in previous comparisons. The accuracy was slightly smaller for disabled subjects, but often the testing conditions couldn’t be arranged in such an optimal way as for healthy subjects. Therefore these results are suggesting that the approach taken in the system presented was suitable for the disabled subjects. This conclusion is confirmed in Fig 4, showing a clear overlapping of results from healthy and disabled persons.

In Section 2.1. the reasons were given why two types of questions were used, “yes/no” questions, and “names” questions. It was important to see whether the different types of questions have different classification accuracy. Since training “name” questions include familiar names, it was possible that this type of questions result in a more pronounced and easier to detect P300 response. This may be desirable in the training phase, although it might have provided overoptimistic results. Generally, a longer word could have also caused more variability and more difficulty in the classification. Once again, the comparison of “yes/no” questions and “names” questions, as provided in Table 6 and Fig. 5, show that the system performs in a similar manner for different type of questions.

Finally, different modes of attention “enhancers” were tested in order to gain more insights in possible improvements of system performance. Mental counting (mc), mental attention (ma), finger counting (fc), tapping (tt) and mental tapping (mt) were tested. Results given in Table 7 and Fig. 6 were tested with analysis of variance (ANOVA) test which confirmed that there were no significant differences between different forms of attention enhancers (p value of 0.92). Mental counting was easy to perform and an effective way of attention enhancer.