Towards a real-world brain-computer interface for image retrieval

Abstract

Brain decoding -the process of inferring a person's momentary cognitive state from their brain activity -has enormous potential in the field of human-computer interaction. In this study we propose a zero-shot EEG-to-image brain decoding approach which makes use of state-of-the-art EEG preprocessing and feature selection methods, and which maps EEG activity to biologically inspired computer vision and linguistic models. We apply this approach to solve the problem of identifying viewed images from recorded brain activity in a reliable and scalable way. We demonstrate competitive decoding accuracies across two EEG datasets, using a zero-shot learning framework more applicable to real-world image retrieval than traditional classification techniques. Introduction 1Research in the field of Brain-Computer Interfaces (BCI) began in the 1970s [1] with 2 the aim of providing a new, intuitive, and rich method of communication between 3 computer systems and their users. Typically, these methods involve measuring some 4 aspect of neural activity and inferring or decoding an intended action or particular 5 characteristic of the user's cognitive state. Although BCI is still in its infancy, there are 6 already practical applications in assistive technology as well as disease diagnosis [2,3]. 7Brain-controlled prosthetics [4] and spellers [5] have shown their potential to enable 8 natural interaction in comparison with more traditional methods, such as mechanical 9 prosthetics or eye-movement-based spellers. Other relevant applications include 10 identifying the image that a user is viewing, usually referred as image retrieval, of 11 particular interest in the field of visual attention applied to advertising and marketing, 12 searching and organising large collections of images, or reducing distractions during 13 driving, to name a few.14 Although brain decoding technology has immense potential in diverse applications, it 15 faces multiple challenges related to speed and accuracy that must be overcome before it 16 emerges as a disruptive technology. The complexity of BCI stems from the naturally low 17 signal-to-noise ratio (SNR) and high dimensionality of raw brain data, which often 18 complicates automated analysis and can force researchers to manually analyse 19 previously recorded neural activation data. This is typically done either by examining 20 the frequency domain or by plotting Event-Related Potentials (ERPs). In an ERP 21 experiment the participant will be presented several times with a similar stimulus and 22 their neural response each time can be recorded and averaged. These ERPs can be 23 analysed against well-known response patterns or, alternatively, characteristics such as 24 February 27, 2019 1/18 the strength and timing of signal peaks can be quantified and analysed automatically. 25ERP analysis is well established and has strong applications in medical diagnosis [6] and 26 in cognitive neuroscience research [7,8]; however, the broad characterisation of brain 27 response used in traditional ERP methods is not ...

EEG Signals Classification related to Visual Objects using Long Short-Term Memory Network and Nonlinear Interval Type-2 Fuzzy Regression

“…The dataset used in this paper was collected by Stanford University (Kaneshiro et al 2015) and used in (McCartney et al 2019) to classify visual images. Six groups of images including the human body, animal body, human face, animal face, fruits/vegetables and tools (12 images in each group) were randomly shown to the participants.…”

“…As described in Section 3.1, this paper uses the data set used in (Kaneshiro et al 2015). According to our knowledge, only three articles (Ahmadieh et al 2023;McCartney, Devereux, and Martinez-del-Rincon 2022;McCartney et al 2019) have used this database to classify visual images. In (McCartney et al 2019), classification accuracy for each participant separately, when all participants are considered together and the average classification accuracy for all participants for the three feature extraction approaches (including visual features, semantic features, and a combination of visual and semantic) has been reported.…”

“…According to our knowledge, only three articles (Ahmadieh et al 2023;McCartney, Devereux, and Martinez-del-Rincon 2022;McCartney et al 2019) have used this database to classify visual images. In (McCartney et al 2019), classification accuracy for each participant separately, when all participants are considered together and the average classification accuracy for all participants for the three feature extraction approaches (including visual features, semantic features, and a combination of visual and semantic) has been reported. The mean classification accuracy for all individuals and for the three approaches are 58.08, 60.88, and 62.02, as reported in Table 4 (the last three columns of the last row) and Fig.…”

“…Fig.14Classification accuracy of all participants (subject-independent) and mean of all participants to compare with a classification accuracy of the three reported methods in(Ahmadieh et al 2023;McCartney et al 2019) with the same dataset.…”

“…Classification accuracy of all participants (subject-independent) and mean of all participants for comparing the recommended methods in this paper with methods of the latest research with this dataset(McCartney et al 2019). …”