Object classification from randomized EEG trials

Ahmed, Hamad; Wilbur, Ronnie Β.; Bharadwaj, Hari; Siskind, Jeffrey Mark

doi:10.1109/cvpr46437.2021.00384

Cited by 15 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dataset. We verify the proposed approach using the largest publicly-accessible single-subject EEG dataset that measures brain activity evoked by visual stimuli (Ahmed et al 2021). The subject is shown 40,000 images and recorded 40,000 EEG segments in response to those visual cues.…”

Section: Empirical Studies Experimental Setupmentioning

confidence: 99%

“…Third, it enables EEG encodings to extend beyond a limited set of visual classes, since cross-modal relationships are learned under the supervision of naturally formed EEG-image pairs, other than manual class annotations. Empirical studies on (Ahmed et al 2021), the largest single-subject EEG dataset regarding visually-evoked tasks, shot that EEGVis-CMR performs competitively against state-of-the-arts. In summary:…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

Ye¹,

Yao²,

Gustin³

2022

Preprint

View full text Add to dashboard Cite

Recent studies demonstrate the use of a two-stage supervised framework to generate images that depict human perception to visual stimuli from Electroencephalography (EEG), referring to EEG-visual reconstruction. They are, however, unable to "reproduce" the exact visual stimulus, since it is the human-specified annotation of images, not their data, that determines what the synthesized images are. Moreover, synthesized images often suffer from noisy EEG encodings and unstable training of generative models, making them hard to recognize. Instead, we present a single-stage EEG-visual retrieval paradigm where the data of two modalities are correlated, as opposed to their annotations, allowing us to recover the exact visual stimulus for an EEG clip. Specifically, we maximize the mutual information between the EEG encoding and associated visual stimulus through optimization of a contrastive self-supervised objective, leading to two additional benefits. One, it enables EEG encodings to handle visual classes beyond seen ones during training, since learning is not directed at class annotations. In addition, the model is no longer required to generate every detail of the visual stimulus, but rather focuses on cross-modal alignment and retrieves images at the instance level, ensuring distinguishable model output. Empirical studies are conducted on the largest singlesubject EEG dataset that measures brain activities evoked by image stimuli. We demonstrate the proposed approach completes an instance-level EEG-visual retrieval task, i.e., to report the exact visual stimulus which existing methods cannot. We also examine the implications of a range of EEG and visual encoder structures. Furthermore, for a mostly studied semantic-level EEG-visual classification task, despite not using class annotations, the proposed method outperforms stateof-the-art supervised EEG-visual reconstruction approaches, particularly on the capability of open class recognition.

show abstract

Section: Empirical Studies Experimental Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

Ye¹,

Yao²,

Gustin³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Since the visual cortex occupies a large portion of the brain, brain signals carry a vast amount of information such as objects, scenery, and characters. In recent years, a number of studies [9], [14], [22], [24], [25] have been conducted to recognize the human imagination based on EEG data induced by visual stimuli. For instance, some work tries to directly decode the own memory of humans [14], [22], [24], while others focus on controlling devices by recognizing the visual stimulus [25].…”

Section: A Brain Activities With Visual Perceptionsmentioning

confidence: 99%

“…In recent years, a number of studies [9], [14], [22], [24], [25] have been conducted to recognize the human imagination based on EEG data induced by visual stimuli. For instance, some work tries to directly decode the own memory of humans [14], [22], [24], while others focus on controlling devices by recognizing the visual stimulus [25]. On the other hand, with the recent advance of image generation models, recent research attempts to reconstruct visual stimuli from the brain signals [26]- [28] and enables the users to readily edit the reconstructed images by thinking [13].…”

Section: A Brain Activities With Visual Perceptionsmentioning

confidence: 99%

Subject Adaptive EEG-Based Visual Recognition

Lee

Hwang²,

Jeon

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper focuses on subject adaptation for EEGbased visual recognition. It aims at building a visual stimuli recognition system customized for the target subject whose EEG samples are limited, by transferring knowledge from abundant data of source subjects. Existing approaches consider the scenario that samples of source subjects are accessible during training. However, it is often infeasible and problematic to access personal biological data like EEG signals due to privacy issues. In this paper, we introduce a novel and practical problem setup, namely source-free subject adaptation, where the source subject data are unavailable and only the pre-trained model parameters are provided for subject adaptation. To tackle this challenging problem, we propose classifier-based data generation to simulate EEG samples from source subjects using classifier responses. Using the generated samples and target subject data, we perform subjectindependent feature learning to exploit the common knowledge shared across different subjects. Notably, our framework is generalizable and can adopt any subject-independent learning method. In the experiments on the EEG-ImageNet40 benchmark, our model brings consistent improvements regardless of the choice of subject-independent learning. Also, our method shows promising performance, recording top-1 test accuracy of 74.6% under the 5-shot setting even without relying on source data. Our code can be found at https://github.com/DeepBCI/Deep-BCI.

show abstract

“…Although recent EEG-based visual recognition methods have shown great performance, most existing methods are developed in a subject-dependent way [14], [17]- [21]. Therefore, in order to utilize the existing methods for a new user, they necessitate the calibration process, i.e., re-training (or tuning) the model with an additionally collected dataset from the new user.…”

Section: Introductionmentioning

confidence: 99%

Inter-subject Contrastive Learning for Subject Adaptive EEG-based Visual Recognition

Lee¹,

Hwang²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper tackles the problem of subject adaptive EEG-based visual recognition. Its goal is to accurately predict the categories of visual stimuli based on EEG signals with only a handful of samples for the target subject during training. The key challenge is how to appropriately transfer the knowledge obtained from abundant data of source subjects to the subject of interest. To this end, we introduce a novel method that allows for learning subject-independent representation by increasing the similarity of features sharing the same class but coming from different subjects. With the dedicated sampling principle, our model effectively captures the common knowledge shared across different subjects, thereby achieving promising performance for the target subject even under harsh problem settings with limited data. Specifically, on the EEG-ImageNet40 benchmark, our model records the top-1 / top-3 test accuracy of 72.6% / 91.6% when using only five EEG samples per class for the target subject.

show abstract

Object classification from randomized EEG trials

Cited by 15 publications

References 35 publications

See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

Subject Adaptive EEG-Based Visual Recognition

Inter-subject Contrastive Learning for Subject Adaptive EEG-based Visual Recognition

Contact Info

Product

Resources

About