Cross-Modal Cloze Task: A New Task to Brain-to-Word Decoding

Zou, Shuxian; Wang, Shaonan; Zhang, Jiajun; Zong, Chengqing

doi:10.18653/v1/2022.findings-acl.54

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fortunately, Brain-computer interfaces (BCIs) can restore language abilities to such patients by decoding neural activities into the natural language (Brain-to-Text), which can drastically improve their quality of life [4]. To pursue this goal, various Brain-to-Text works are proposed, building upon either invasive brain recordings, such as electrocorticography (ECoG) [2], [24], [25], or non-invasive brain recordings, such as functional magnetic resonance imaging (fMRI) [37] and electroencephalography (EEG) [34]. Amongst, EEG shows its superior benefits in portability and cost-effectiveness in real-world applications, thus EEG-to-Text generation gains a lot of research interest recently [10], [34].…”

Section: Introductionmentioning

confidence: 99%

Aligning Semantic in Brain and Language: A Curriculum Contrastive Method for Electroencephalography-to-Text Generation

Feng,

Qin

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Electroencephalography-to-Text generation (EEG-to-Text), which aims to directly generate natural text from EEG signals has drawn increasing attention in recent years due to the enormous potential for Brain-computer interfaces. However, the remarkable discrepancy between the subject-dependent EEG representation and the semantic-dependent text representation poses a great challenge to this task. To mitigate this, we devise a Curriculum Semantic-aware Contrastive Learning strategy (C-SCL), which effectively recalibrates the subject-dependent EEG representation to the semantic-dependent EEG representation, thereby reducing the discrepancy. Specifically, our C-SCL pulls semantically similar EEG representations together while pushing apart dissimilar ones. Besides, in order to introduce more meaningful contrastive pairs, we carefully employ curriculum learning to not only craft meaningful contrastive pairs but also make the learning progressively. We conduct extensive experiments on the ZuCo benchmark and our method combined with diverse models and architectures shows stable improvements across three types of metrics while achieving the new state-of-the-art. Further investigation proves not only its superiority in both the single-subject and low-resource settings but also its robust generalizability in the zero-shot setting 1 .

show abstract