Performance assessment in brain-computer interface-based augmentative and alternative communication

Abstract

A large number of incommensurable metrics are currently used to report the performance of brain-computer interfaces (BCI) used for augmentative and alterative communication (AAC). The lack of standard metrics precludes the comparison of different BCI-based AAC systems, hindering rapid growth and development of this technology. This paper presents a review of the metrics that have been used to report performance of BCIs used for AAC from January 2005 to January 2012. We distinguish between Level 1 metrics used to report performance at the output of the BCI Control Module, which translates brain signals into logical control output, and Level 2 metrics at the Selection Enhancement Module, which translates logical control to semantic control. We recommend that: (1) the commensurate metrics Mutual Information or Information Transfer Rate (ITR) be used to report Level 1 BCI performance, as these metrics represent information throughput, which is of interest in BCIs for AAC; 2) the BCI-Utility metric be used to report Level 2 BCI performance, as it is capable of handling all current methods of improving BCI performance; (3) these metrics should be supplemented by information specific to each unique BCI configuration; and (4) studies involving Selection Enhancement Modules should report performance at both Level 1 and Level 2 in the BCI system. Following these recommendations will enable efficient comparison between both BCI Control and Selection Enhancement Modules, accelerating research and development of BCI-based AAC systems.

A concept-environment for computer-based augmentative and alternative communication founded on a systematic review

“…Paper Category (Bermúdez et al, 2010), (Cabrera et al, 2008), (Chin et al, 2010), (Deepa et al, 2010), (Okasaka and Hoshino, 2012), (Schalk et al, 2008), (Sorger et al, 2009), (Sun et al, 2010, (Thomas et al, 2008), (Thompson et al, 2013), (Ortner et al, 2011), (Al-Abdullatif et al, 2013), , , (Ming et al, 2009) BCI (Hanson et al, 2010) Speech generator device (Keegan et al, 2009) EOG (Joubert et al, 2011) Exploratory study (Ann and Theng, 2011b) Facial expression (Park et al, 2012), (Cipresso et al, 2011) Eyetracking (AlJa 'am et al, 2008), (Ann and Theng, 2011b), (Friginal et al, 2013), , (Sorna et al, 2009), (Hsieh et al, 2009) Software modeling , , , (Silva and Pereira, 2011), (Tsai, 2013) Communication board (Belani, 2012), (Arnott and Alm, 2013), (Deepa et al, 2010), (Hill, 2010), (McCoy et al, 2013), (Mak and Wolpaw, 2009), Conceptual review (Alwell and Cobb, 2009), (Finke et al, 2008) Systematic review , (Prabhu and Prasad, 2011), Alternative virtual keyboard…”

“…Use an existing low cost device (McCoy et al, 2013), (Hill, 2010) Lack of training and technical support from suppliers Develop open source Solution (Arnott and Alm, 2013), (Ann and Theng, 2011b), , (Ming et al, 2009), (Keegan et al, 2009), (Schalk et al, 2008), (Sun et al, 2010), (Thompson et al, 2013) Low frequency of data input…”

What is the optimal task difficulty for reinforcement learning of brain self-regulation?

“…the average of TPR and TNR, is the most common measure for assessing the performance of classification algorithms (Thompson et al 2013). For classical assistive BRIs, which follow the goal to replace lost functions and control external devices, maximizing classification accuracy is considered to be optimal, and the threshold (θ) is selected accordingly (Theodoridis and Koutroumbas 2009).…”

“…At present, the performance of classification algorithms is mainly based on communication and control criteria such as classification accuracy (Thompson et al 2013). As for the restorative context, only limited research has been carried out on performance measures on the basis of learning principles such as instructional efficiency (Georges 1931;Clark 2006).…”

“…Linear and synchronous classifiers allow the straightforward calculation of performance metrics such as the true positive rate (TPR) or true negative rate (TNR), which equal the sensitivity and specificity of the classifier, respectively. Common performance metrics (Sokolova and Lapalme 2009;Thomas et al 2013; Thompson et al 2013) suggest that accuracy and speed of the classifier are important goals. However, restorative BRIs can also be understood as a form of neurofeedback training, i.e.…”