Superman-like X-ray vision: Towards brain-computer interfaces for medical augmented reality

Abstract

This paper describes first steps towards a Superman-like X-ray vision where a brain-computer interface (BCI) device and a gazetracker are used to allow the user controlling the augmented reality (AR) visualization. A BCI device is integrated into two medical AR systems. To assess the potential of this technology first feedback from medical doctors is gathered. While in this pilot study not the full range of available signals but only electromyographic signals are used, the medical doctors provided very positive feedback on the use of BCI for medical AR.

Recent Developments and Future Challenges in Medical Mixed Reality

“…These methods usually involve marker-based (external fiducial markers), featurebased (internal anatomical landmarks) or intensity-based methods that find a geometric transformation that brings the projection of a 3D image into the best possible spatial correspondence with the 2D images by optimizing a registration criterion. [75] Veins Localization Optical Vuzix STAR 1200XL(modified) Manually Aligned Chang et al (2015) [76] Remote Surgical Assistance Video VUNIX iWear VR920 Optical Tracker + KLT * Hsieh et al (2015) [77] Head CT Visulization Video Vuzix Wrap 1200DXAR KLT + ICP ** Wang et al (2015) [78] Screw Placement Navigation Optical NVIS nVisor ST60 Optical Tracker Vigh et al (2014) [79] Oral Implantology Video NVIS nVisor SX60 Optical Tracker Hu et al (2013) [80] Surgery Guidance Video eMagin Z800 3D Visor Marker Abe et al (2013) [81] Percutaneous Vertebroplasty Video HMD by Epson (Model Unknow) Marker Azimi et al (2012) [82] Navigation in Neurosurgery Optical Goggles by Juxtopia Marker Blum et al (2012) [83] Anatomy Visulization Video Not Mentioned Gaze-tracker Toshiaki (2010) [84] Cognitive [86] Treatment of Cockroach Phobia Video HMD by 5DT (Model Unknow) Marker Alamri et al (2010) [87] Poststroke-Patient Rehabilitation Video VUNIX iWear VR920 Marker * Kanade-Lucas-Tomasi algorithm [88]. ** Iterative Closest Point algorithm [89].…”

Virtual Surgery, Applications and Limitations

“…Blum et al [111] have proposed a system that allows the user to control the augmented reality visualization by using brain signal and gaze through a braincomputer interface (BCI) and a gaze-tracker. Data and feedback from medical doctors (MD) are gathered to access the potential of this technology.…”

“…Collision detection techniques are also being applied and computed in real-time to avoid collision between the teleoperator arms and their Fig. 8.7 Setup using a HMD, a monitor-based visualization, a gaze-tracker and a BCI [111] corresponding ports. By implementing the triangle-triangle intersection detection, which is applied on the patient's polygonal model [121], collision between the arms and the anatomy can be avoided.…”

DEMO: Eva, uma mascote virtual em Realidade Aumentada

“…A Realidade Aumentada (RA) está sendo utilizada em diversas áreas [1]- [3] e, principalmente, no entretenimento [4]- [6], onde as previsões preveem ser um mercado crescente [7]- [9], e onde os dispositivos móveis (principalmente os smartphones) têm sido bastante utilizados como ferramentas que permitem aos usuários interagirem com objetos virtuais inseridos em um ambiente real. Jogos como AR Dragon e Jurassic World destacam-se nessa emocionante interação digital.…”