Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network With Multiscale Feature Fusion

Luo, Jie; Cui, Weigang; Liu, Jingyu; Li, Yang; Guo, Yuzhu; Xu, Shengyuan; Wang, Lina

doi:10.1109/tcds.2022.3181469

Cited by 3 publications

(2 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the field of natural image reconstruction from brain activity, Miyawaki et al [ 5 ] reconstructed the arbitrary binary contrast patterns by separately predicting on predefined multi–scale local image bases. Luo et al [ 40 ] proposed DA–HLGN–MSFF, which combines the hierarchical feature extraction and multi–scale feature fusion block to improve the reconstruction performance. Meng et al [ 18 ] exploited a similar multi–scale encoder–decoder architecture to achieve promising natural image reconstruction.…”

Section: Related Workmentioning

confidence: 99%

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

Kuang,

Zhan,

Gao

2024

Brain Sciences

View full text Add to dashboard Cite

Reconstructing natural stimulus images using functional magnetic resonance imaging (fMRI) is one of the most challenging problems in brain decoding and is also the crucial component of a brain–computer interface. Previous methods cannot fully exploit the information about interactions among brain regions. In this paper, we propose a natural image reconstruction method based on node–edge interaction and a multi-scale constraint. Inspired by the extensive information interactions in the brain, a novel graph neural network block with node–edge interaction (NEI–GNN block) is presented, which can adequately model the information exchange between brain areas via alternatively updating the nodes and edges. Additionally, to enhance the quality of reconstructed images in terms of both global structure and local detail, we employ a multi-stage reconstruction network that restricts the reconstructed images in a coarse-to-fine manner across multiple scales. Qualitative experiments on the generic object decoding (GOD) dataset demonstrate that the reconstructed images contain accurate structural information and rich texture details. Furthermore, the proposed method surpasses the existing state-of-the-art methods in terms of accuracy in the commonly used n-way evaluation. Our approach achieves 82.00%, 59.40%, 45.20% in n-way mean squared error (MSE) evaluation and 83.50%, 61.80%, 46.00% in n-way structural similarity index measure (SSIM) evaluation, respectively. Our experiments reveal the importance of information interaction among brain areas and also demonstrate the potential for developing visual-decoding brain–computer interfaces.

show abstract

Section: Related Workmentioning

confidence: 99%

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

Kuang,

Zhan,

Gao

2024

Brain Sciences

View full text Add to dashboard Cite

show abstract

“…As the hyperparameters of deep learning models have a significant impact on performance, it is crucial to set appropriate values for optimal model [40][41][42][43][44]. In this study, we employed a simple yet effective grid search strategy to optimize hyperparameters such as the number of scales S for multi-scale temporal features and the dimension of domain-specific representation L. Performance comparison results of the proposed model under different parameter settings are presented in Fig.…”

Section: Analysis Of Hyperparameter Settingsmentioning

confidence: 99%

A Cross-Scale Transformer and Triple-View Attention Based Domain-Rectified Transfer Learning for EEG Classification in RSVP Tasks

Luo,

Cui,

et al. 2024

IEEE Trans. Neural Syst. Rehabil. Eng.

Self Cite

View full text Add to dashboard Cite

Rapid serial visual presentation (RSVP)-based brain-computer interface (BCI) is a promising target detection technique by using electroencephalogram (EEG) signals. However, existing deep learning approaches seldom considered dependencies of multi-scale temporal features and discriminative multi-view spectral features simultaneously, which limits the representation learning ability of the model and undermine the EEG classification performance. In addition, recent transfer learning-based methods generally failed to obtain transferable cross-subject invariant representations and commonly ignore the individualspecific information, leading to the poor cross-subject transfer performance. In response to these limitations, we propose a cross-scale Transformer and triple-view attention based domain-rectified transfer learning (CST-TVA-DRTL) for the RSVP classification. Specially, we first develop a cross-scale Transformer (CST) to extract multi-scale temporal features and exploit the dependencies of different scales features. Then, a triple-view attention (TVA) is designed to capture spectral features from triple views of multi-channel timefrequency images. Finally, a domain-rectified transfer learning (DRTL) framework is proposed to simultaneously obtain transferable domain-invariant representations and untransferable domain-specific representations, then utilize domain-specific information to rectify domain-invariant representations to adapt to target data. Experimental results on two public RSVP datasets suggests that our CST-TVA-DRTL outperforms the state-of-the-art methods in the RSVP classification task. The source code of our model is publicly available in https://github.com/ljbuaa/CST_TVA_DRTL.

show abstract

A Dual-Branch Spatio-Temporal-Spectral Transformer Feature Fusion Network for EEG-Based Visual Recognition

Luo

Cui

Xu³

et al. 2024

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network With Multiscale Feature Fusion

Cited by 3 publications

References 35 publications

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

A Cross-Scale Transformer and Triple-View Attention Based Domain-Rectified Transfer Learning for EEG Classification in RSVP Tasks

A Dual-Branch Spatio-Temporal-Spectral Transformer Feature Fusion Network for EEG-Based Visual Recognition

Contact Info

Product

Resources

About