Performance measurement for brain–computer or brain–machine interfaces: a tutorial

Abstract

Objective Brain-Computer Interfaces (BCIs) have the potential to be valuable clinical tools. However, the varied nature of BCIs, combined with the large number of laboratories participating in BCI research, makes uniform performance reporting difficult. To address this situation, we present a tutorial on performance measurement in BCI research. Approach A workshop on this topic was held at the 2013 International BCI Meeting at Asilomar Conference Center in Pacific Grove, California. This manuscript contains the consensus opinion of the workshop members, refined through discussion in the following months and the input of authors who were unable to attend the workshop. Main Results Checklists for methods reporting were developed for both discrete and continuous BCIs. Relevant metrics are reviewed for different types of BCI research, with notes on their application to encourage uniform application between laboratories. Significance Graduate students and other researchers new to BCI research may find this tutorial a helpful introduction to performance measurement in the field.

Neural control of finger movement via intracortical brain–machine interface

“…This metric is fairly robust to variations in tasks (Thompson et al, 2014), and therefore can be compared across algorithms and lab procedures. The index of difficulty of this behavioral task compares well to previous whole-arm tasks, both in physical and brain control.…”

“…To evaluate the performance of the online movement decoding during brain control, we computed several metrics including trial success rate, time to target acquisition, and bit rate via Fitts’s law (Thompson et al, 2014), and compared them to those computed during physical task control.…”

BNCI Horizon 2020: towards a roadmap for the BCI community

“…have no clear answers. There are no gold standards on how to objectively evaluate BCI performance or which key facts should be reported in scientific publications, [18] and there are no guidelines on how to properly deal with ethical, legal, and societal issues. In addition, comparing signal processing and machine learning algorithms across different groups is practically impossible, because there is no central open database with curated benchmark data sets.…”

Identifying engineering, clinical and patient's metrics for evaluating and quantifying performance of brain-machine interface (BMI) systems

“…ITR (bits per sec) is a general evaluation metric devised for brain-computer interface systems (BCIs) for restoring linguistic communication such as P300 BCI spellers (see [19] for a review) or to evaluate performance in BMI systems for 2D cursor control [10], [20]. It also allows comparison of the performance of BCI systems, which have a different number of tasks [6], [7], [21]. Speier et al [19] summarize several limitations of the current use of ITR as a BMI metric: a) conditional probabilities for selection sequences have not been reported, b) information about types of errors in BCI for communication are not used to improve their selection (errors are either ignored or deleted; time outs in 2D BCIs limit quantification of performance), c) task constraints or 'shared control' are usually not factored in the quantification of BMI performance, and d) it is unclear how low the ITR would need to be in order to understand the BCI output.…”

“…In this regard, public efforts have been recently made, including the Workshop on BCI metrics at the Asilomar meeting held on June 3–7, 2013 [5], which is summarized in [6]. A recent study have also addressed some challenges and limitations in the development and selection of BCI performance metrics [7], including developing efficient measurement techniques that adapt rapidly and reliably to capture a wide range of performance levels and the identification of BCI subsystems that may potentially restrict the maximum systems level performance, which is a critical factor for considerations of device interoperability.…”