Closing the loop in cortically-coupled computer vision: a brain–computer interface for searching image databases

Abstract

We describe a closed-loop brain-computer interface that re-ranks an image database by iterating between user generated 'interest' scores and computer vision generated visual similarity measures. The interest scores are based on decoding the electroencephalographic (EEG) correlates of target detection, attentional shifts and self-monitoring processes, which result from the user paying attention to target images interspersed in rapid serial visual presentation (RSVP) sequences. The highest scored images are passed to a semi-supervised computer vision system that reorganizes the image database accordingly, using a graph-based representation that captures visual similarity between images. The system can either query the user for more information, by adaptively resampling the database to create additional RSVP sequences, or it can converge to a 'done' state. The done state includes a final ranking of the image database and also a 'guess' of the user's chosen category of interest. We find that the closed-loop system's re-rankings can substantially expedite database searches for target image categories chosen by the subjects. Furthermore, better reorganizations are achieved than by relying on EEG interest rankings alone, or if the system were simply run in an open loop format without adaptive resampling.

Online and Offline Domain Adaptation for Reducing BCI Calibration Effort

“…The target images are much less frequent than the non-target ones, so that they can elicit P300 ERPs in the oddball paradigm. The P300 ERPs can be detected by a BCI system [35], and the corresponding images are then triaged for further inspection. Research [33], [39], [54] has shown that these BCI systems enable the subject to detect targets in large aerial photographs faster and more accurately than traditional standard searches.…”

Single-trial classification of neural responses evoked in rapid serial visual presentation: Effects of stimulus onset asynchrony and stimulus repetition

“…The error based on the rank for sorting blocks of 10 images, err rank , was 2.86 for C 2,1 , 1.93 for C 4,2 , 1.70 for C 6,3 , and 1.33 for C 10,5 , showing that it possible to find the target among the two highest ranked responses. The accuracy by considering the n first best response is presented in Figure 4.…”

An overview of brain computer interface

“…It has been used to develop virtual on-screen keyboards [17], [43], control devices such as wheelchair [29] or play virtual games [62]. P300s are also elicited in target-distractor paradigms such as Rapid Serial Visual Presentation (RSVP) and is used in image search applications [21], [51]. Another EP, called interaction Error-related Potentials (ErrP), are generated when an external device commit an erroneous action [19], [18].…”