The Brain-Computer Interface (BCI) was envisioned as an assistive
technology option for people with severe movement impairments. The traditional
synchronous event-related potential (ERP) BCI design uses a fixed communication
speed and is vulnerable to variations in attention. Recent ERP BCI designs have
added asynchronous features, including abstention and dynamic stopping, but it
remains a open question of how to evaluate asynchronous BCI performance. In this
work, we build on the BCI-Utility metric to create the first evaluation metric
with special consideration of the asynchronous features of self-paced BCIs. This
metric considers accuracy as all of the following three – probability of
a correct selection when a selection was intended, probability of making a
selection when a selection was intended, and probability of an abstention when
an abstention was intended. Further, it considers the average time required for
a selection when using dynamic stopping and the proportion of intended
selections versus abstentions. We establish the validity of the derived metric
via extensive simulations, and illustrate and discuss its practical usage on
real-world BCI data. We describe the relative contribution of different inputs
with plots of BCI-Utility curves under different parameter settings. Generally,
the BCI-Utility metric increases as any of the accuracy values increase and
decreases as the expected time for an intended selection increases. Furthermore,
in many situations, we find shortening the expected time of an intended
selection is the most effective way to improve the BCI-Utility, which
necessitates the advancement of asynchronous BCI systems capable of accurate
abstention and dynamic stopping.