Exploring EEG for Object Detection and Retrieval

Abstract

This paper explores the potential for using Brain Computer Interfaces (BCI) as a relevance feedback mechanism in contentbased image retrieval. Several experiments are performed using a rapid serial visual presentation (RSVP) of images at different rates (5Hz and 10Hz) on 8 users with different degrees of familiarization with BCI and the dataset. We compare the feedback from the BCI and mouse-based interfaces in a subset of TRECVid images, finding that, when users have limited time to annotate the images, both interfaces are comparable in performance. Comparing our best users in a retrieval task, we found that EEG-based relevance feedback can outperform mouse-based feedback. Categories and Subject Descriptors MOTIVATIONThe exponential growth of visual content and its huge diversity has motivated considerable research on how documents can be retrieved according to user intentions when formulating a query.Advances in image processing and computer vision have provided tools for a perceptual and semantic interpretation of both the query and the indexed content. This has allowed the development of retrieval systems capable of processing queries by example and concepts.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. The role of a human user during visual retrieval is critical, and his judgment about the correctness of the retrieved results can greatly speed up the search processes. This kind of relevance feedback has been demonstrated to significantly improve retrieval performance in image [10] and video [1] retrieval. Manually annotating images using a mouse, especially in a visual retrieval context can be tedious and mentally exhausting. In that such a scenario, EEG-based brain computer interfaces offer a potential solution as a mechanism to quickly annotate images. RELATED WORKEEG signals have been used for object detection in [2], where authors aim to detect airplanes in a dataset of satellite images from the city of London. The work in [3] expands the catalog of objects in very simple images, where the object on a black background occupies the whole image.EEG signals have been also used for image retrieval in [9], where authors used EEG relevance annotations to retrieve specific concepts in a complex dataset of keyframes from TRECVid 2005. However, while that work aimed at detecting concepts depicted by the whole image, we focus on the more challenging task of detecting a local object in a complex scenario. Another similar work [8], addresses the usage of EEG for image retrieval by formu...

Context based healthcare informatics system to detect gallstones using deep learning methods

“…Findings show that the target identification correctness of Fast R-CNN is 91.6 percent in the picture collection with a resolution of 866×652 (pixels), and AdaBoost as the remaining detector improves the precision to 96.76 percent but the time cost is too high. Mohedano et al [19] suggested the Electroencephalogram (EEG) tool for relevance feedback and contrasted it to the conventional "click -based" mechanism for an item extraction assignment. It is discovered that the EEGbased approach achieved comparable accuracy.…”

Synthetic-Neuroscore: Using a neuro-AI interface for evaluating generative adversarial networks

“…In outline, Neuroscore is calculated via measurement of the P300, an eventrelated potential (ERP) present in EEG, via a rapid serial visual presentation (RSVP) paradigm. The P300 and RSVP paradigm are mature techniques in the brain-computer interface (BCI) community and have been applied in a wide variety of tasks such as image search [38], information retrieval [39], and others. The unique benefit of Neuroscore is that it more directly reflects human perceptual judgment of images, which is intuitively more reliable compared to conventional metrics in the literature [14].…”

Synthetic-Neuroscore: Using A Neuro-AI Interface for Evaluating Generative Adversarial Networks

“…In detail, Neuroscore is calculated via measurement of the P300, an event-related potential (ERP) present in EEG, via a rapid serial visual presentation (RSVP) paradigm. P300 and RSVP paradigm are mature techniques in the brain-computer interface (BCI) community and have been applied in a wider variety of tasks such as image search [Gerson et al, 2006], information retrieval [Mohedano et al, 2015], and etc. The unique benefit of Neuroscore is that it more directly reflects the human perceptual judgment of images, which is intuitively more reliable compared to the conventional metrics in the literature [Borji, 2018].…”