Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation

Tseng, Kuan-Lun; Lin, Yen-Liang; Hsu, Winston H.; Huang, Chung-Yang

doi:10.1109/cvpr.2017.398

Cited by 156 publications

(105 citation statements)

References 22 publications

(41 reference statements)

Supporting

Mentioning

100

Contrasting

Order By: Relevance

“…Two-dimensional methods [6,12] that are based on inplane segmentation does not utilize contextual information along the slice direction. Three-dimensional convolutional neural networks (CNN) [4,13] have been proposed to capture image dependencies between consecutive slices. However, by trading-off between the convolution kernel size and the number of pooling layers, these methods can only capture limited local receptive field and short-range dependencies.…”

Section: Introductionmentioning

confidence: 99%

“…The contributions of our RSANet are three folds. First, unlike methods [3,13] using RNN or LSTM to capture the slice-wise dependencies, where RNN and LSTM have inherent drawbacks [2,10,14] of capturing long-range dependencies, we propose a novel slice-wise attention module, called SA Block (see Fig. 2) to compute the response at a slice as a weighted sum of the features from all slices along the same direction.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

RSANet: Recurrent Slice-Wise Attention Network for Multiple Sclerosis Lesion Segmentation

Zhang

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Brain lesion volume measured on T2 weighted MRI images is a clinically important disease marker in multiple sclerosis (MS). Manual delineation of MS lesions is a time-consuming and highly operatordependent task, which is influenced by lesion size, shape and conspicuity. Recently, automated lesion segmentation algorithms based on deep neural networks have been developed with promising results. In this paper, we propose a novel recurrent slice-wise attention network (RSANet), which models 3D MRI images as sequences of slices and captures longrange dependencies through a recurrent manner to utilize contextual information of MS lesions. Experiments on a dataset with 43 patients show that the proposed method outperforms the state-of-the-art approaches. Our implementation is available online at https://github. com/tinymilky/RSANet Keywords: magnetic resonance imaging · convolutional neural networks · long-range dependencies · multiple sclerosis lesion segmentation.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

RSANet: Recurrent Slice-Wise Attention Network for Multiple Sclerosis Lesion Segmentation

Zhang

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In medical images, combining multi-modality images such as CT, T1 MRI, T2 MRI, etc. to realize multi-organ segmentation [17] and lesion segmentation [18,19] is widely adopted due to distinct responses of different modalities datasets for different tissues. According to the review [20] of deep learning for medical image segmentation using multi-modality fusion, multi-modal segmentation network architectures can be categorized into input-level fusion network, layer-level fusion network and decision fusion network.…”

Section: Multi-modal Fusionmentioning

confidence: 99%

“…In decision-level fusion segmentation networks, multiple pathways are set to process separately multimodal images and the final features [25] or results [26] are combined for decision making. The layer-level fusion network [17,18,23,24] fuses multi-source features in mediate layers to obtain complementary and interdependent features.…”

Section: Multi-modal Fusionmentioning

confidence: 99%

“…We concentrate three volumes of MRI as different input channel, the architectures of encoder and decoder are as described in section 3.1, while we set the channel number of encoder as three times of original encoder to keep the number of low-level features is the same as MMFNet. 6) Merging encoders' features [18], setting three modality-specific encoders to capture low-level features and a decoder to fuse low-level and highlevel features. 7) Linking features across multi-path [23], which builds multiple streams for different modalities of MRI and links features across these streams.…”

Section: Comparative Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

MMFNet: A multi-modality MRI fusion network for segmentation of nasopharyngeal carcinoma

Huai

Yin

et al. 2020

Neurocomputing

View full text Add to dashboard Cite

Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic ResonanceImages (MRI) is a crucial prerequisite for NPC radiotherapy. However, manually segmenting of NPC is time-consuming and labor-intensive. Additionally, single-modality MRI generally cannot provide enough information for its accurate delineation. Therefore, a multi-modality MRI fusion network (MMFNet), which is a novel framework to fuse information from multi-modality medical images, is proposed to utilize MRI of T1, T2 and contrast-enhanced T1 to complete accurate segmentation of NPC. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders to capture modality-specific features and one decoder to obtain fused features for NPC segmentation. A fusion block is presented to effectively fuse multi-source features.It contains a 3D Convolutional Block Attention Module (3D-CBAM), recalibrating low-level features captured from modality-specific encoders to highlight both informative features and regions of interest (ROIs), and a residual fusion block (RFBlock), which fuses re-weighted features to keep balance between fused ones and high-level features from decoder. Moreover, in order to make full mining of individual information from multi-modality MRI, a training strategy named self- * transfer is proposed to utilize pre-trained modality-specific encoders to initialize multi-encoder-based network. The proposed method based on multi-modality MRI can effectively segment NPC and its advantages are validated by extensive experiments.

show abstract

Brain tumor segmentation by combining MultiEncoder UNet with wavelet fusion

Pan,

Yong,

et al. 2024

J Applied Clin Med Phys

View full text Add to dashboard Cite

Background and objectiveAccurate segmentation of brain tumors from multimodal magnetic resonance imaging (MRI) holds significant importance in clinical diagnosis and surgical intervention, while current deep learning methods cope with situations of multimodal MRI by an early fusion strategy that implicitly assumes that the modal relationships are linear, which tends to ignore the complementary information between modalities, negatively impacting the model's performance. Meanwhile, long‐range relationships between voxels cannot be captured due to the localized character of the convolution procedure.MethodAiming at this problem, we propose a multimodal segmentation network based on a late fusion strategy that employs multiple encoders and a decoder for the segmentation of brain tumors. Each encoder is specialized for processing distinct modalities. Notably, our framework includes a feature fusion module based on a 3D discrete wavelet transform aimed at extracting complementary features among the encoders. Additionally, a 3D global context‐aware module was introduced to capture the long‐range dependencies of tumor voxels at a high level of features. The decoder combines fused and global features to enhance the network's segmentation performance.ResultOur proposed model is experimented on the publicly available BraTS2018 and BraTS2021 datasets. The experimental results show competitiveness with state‐of‐the‐art methods.ConclusionThe results demonstrate that our approach applies a novel concept for multimodal fusion within deep neural networks and delivers more accurate and promising brain tumor segmentation, with the potential to assist physicians in diagnosis.

show abstract

Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation

Cited by 156 publications

References 22 publications

RSANet: Recurrent Slice-Wise Attention Network for Multiple Sclerosis Lesion Segmentation

RSANet: Recurrent Slice-Wise Attention Network for Multiple Sclerosis Lesion Segmentation

MMFNet: A multi-modality MRI fusion network for segmentation of nasopharyngeal carcinoma

Brain tumor segmentation by combining MultiEncoder UNet with wavelet fusion

Contact Info

Product

Resources

About