Towards BCI-Based Interfaces for Augmented Reality: Feasibility, Design and Evaluation

Abstract

Brain-Computer Interfaces (BCIs) enable users to interact with computers without any dedicated movement, bringing new hands-free interaction paradigms. In this paper we study the combination of BCI and Augmented Reality (AR). We first tested the feasibility of using BCI in AR settings based on Optical See-Through Head-Mounted Displays (OST-HMDs). Experimental results showed that a BCI and an OST-HMD equipment (EEG headset and Hololens in our case) are well compatible and that small movements of the head can be tolerated when using the BCI. Second, we introduced a design space for command display strategies based on BCI in AR, when exploiting a famous brain pattern called Steady-State Visually Evoked Potential (SSVEP). Our design space relies on five dimensions concerning the visual layout of the BCI menu ; namely: orientation, frame-of-reference, anchorage, size and explicitness. We implemented various BCI-based display strategies and tested them within the context of mobile robot control in AR. Our findings were finally integrated within an operational prototype based on a real mobile robot that is controlled in AR using a BCI and a HoloLens headset. Taken together our results (4 user studies) and our methodology could pave the way to future interaction schemes in Augmented Reality exploiting 3D User Interfaces based on brain activity and BCIs.

SSVEP Stimulus Layout Effect on Accuracy of Brain-Computer Interfaces in Augmented Reality Glasses

“…When the stimulus duration is less than 3s, AR-Pos2 achieved the highest classification accuracy compared to the other 3 positions, and 89.0% and 94.6% average accuracy could be obtained under the stimulus duration of 2s and 3s. Compared with the existing AR-BCI study [28], this study obtained a higher recognition accuracy by a shorter stimulus duration.…”

“…Displaying repetitive visual stimulus via OST-AR technology could overcome the limitations of the vision of PC screen, and improve the portability and flexibility of SSVEP-BCI simultaneously. In 2018, Hakim Si-Mohammed et al verified that it is feasible of combining OST-HMD with BCI through designing and evaluating the application prototype of AR-BCI system [28]. Ming et al designed SSVEP-BCI with OST-AR and achieved 8 commands robotic arm control, their results showed that longer stimulation time is required to guarantee higher accuracy for OST-AR compared with PC screen due to the weakened SSVEP responses evoked by the transparency of stimulation [29].…”

Augmented Reality for Robotics: A Review

“…The obtained results were the following: amount of trigger commands necessary to complete the task reduced and amount of errors during the lifting process decreased more than 50% in comparison to the trials when a vision system without AR feedback was utilized. Si-Mohammed et al [109] introduced a BCI system with AR feedback to simplify the control of a mobile robot. Microsoft HoloLens MR goggles were used for visualization of the augmented environment.…”

Arquitectura para la integración de motores de videojuegos en aplicaciones basadas en interfaces cerebro-computador

“…Esta misma impresión, sobre el doble papel de los sistemas BCI en videojuegos y entornos de realiad virtual se obtiene de una revisión de 2012 [4] En el campo de la realidad aumentada, se están empezando a publicar estudios ahora. Por ejemplo, el estudio de 2018 [5] realiza un análisis interesante y sistemático sobre la utilización de un sistema BCI basado en SSEVP (potencial evocado de estado estable) con realidad aumentada de 3 comandos para dirigir un robot móvil. Dicho estudio preliminar muestra que un interfaz basado en realidad aumentada con los targets situados en el plano del robot y moviéndose conél, son preferidos por los usuarios debido a que se ahorran la necesidad de hacer operaciones mentales para mapear los signos visuales con la orientación del robot en cada momento.…”