Brain decoding -the process of inferring a person's momentary cognitive state from their brain activity -has enormous potential in the field of human-computer interaction. In this study we propose a zero-shot EEG-to-image brain decoding approach which makes use of state-of-the-art EEG preprocessing and feature selection methods, and which maps EEG activity to biologically inspired computer vision and linguistic models. We apply this approach to solve the problem of identifying viewed images from recorded brain activity in a reliable and scalable way. We demonstrate competitive decoding accuracies across two EEG datasets, using a zero-shot learning framework more applicable to real-world image retrieval than traditional classification techniques.
Introduction 1Research in the field of Brain-Computer Interfaces (BCI) began in the 1970s [1] with 2 the aim of providing a new, intuitive, and rich method of communication between 3 computer systems and their users. Typically, these methods involve measuring some 4 aspect of neural activity and inferring or decoding an intended action or particular 5 characteristic of the user's cognitive state. Although BCI is still in its infancy, there are 6 already practical applications in assistive technology as well as disease diagnosis [2,3]. 7Brain-controlled prosthetics [4] and spellers [5] have shown their potential to enable 8 natural interaction in comparison with more traditional methods, such as mechanical 9 prosthetics or eye-movement-based spellers. Other relevant applications include 10 identifying the image that a user is viewing, usually referred as image retrieval, of 11 particular interest in the field of visual attention applied to advertising and marketing, 12 searching and organising large collections of images, or reducing distractions during 13 driving, to name a few.14 Although brain decoding technology has immense potential in diverse applications, it 15 faces multiple challenges related to speed and accuracy that must be overcome before it 16 emerges as a disruptive technology. The complexity of BCI stems from the naturally low 17 signal-to-noise ratio (SNR) and high dimensionality of raw brain data, which often 18 complicates automated analysis and can force researchers to manually analyse 19 previously recorded neural activation data. This is typically done either by examining 20 the frequency domain or by plotting Event-Related Potentials (ERPs). In an ERP 21 experiment the participant will be presented several times with a similar stimulus and 22 their neural response each time can be recorded and averaged. These ERPs can be 23 analysed against well-known response patterns or, alternatively, characteristics such as 24 February 27, 2019 1/18 the strength and timing of signal peaks can be quantified and analysed automatically.
25ERP analysis is well established and has strong applications in medical diagnosis [6] and 26 in cognitive neuroscience research [7,8]; however, the broad characterisation of brain 27 response used in traditional ERP methods is not ...