IAS‐NET: Joint intraclassly adaptive GAN and segmentation network for unsupervised cross‐domain in neonatal brain MRI segmentation

Abstract

In neonatal brain magnetic resonance image (MRI) segmentation, the model we trained on the training set (source domain) often performs poorly in clinical practice (target domain). As the label of target-domain images is unavailable, this cross-domain segmentation needs unsupervised domain adaptation (UDA) to make the model adapt to the target domain. However, the shape and intensity distribution of neonatal brain MRI images across the domains are largely different from adults'. Current UDA methods aim to make synthesized images similar to the target domain as a whole. But it is impossible to synthesize images with intraclass similarity because of the regional misalignment caused by the cross-domain difference. This will result in generating intraclassly incorrect intensity information from target-domain images. To address this issue, we propose an IAS-NET (joint intraclassly adaptive generative adversarial network (GAN) (IA-NET) and segmentation) framework to bridge the gap between the two domains for intraclass alignment. Methods: Our proposed IAS-NET is an elegant learning framework that transfers the appearance of images across the domains from both image and feature perspectives. It consists of the proposed IA-NET and a segmentation network (S-NET). The proposed IA-NET is a GAN-based adaptive network that contains one generator (including two encoders and one shared decoder) and four discriminators for cross-domain transfer. The two encoders are implemented to extract original image, mean, and variance features from source and target domains. The proposed local adaptive instance normalization algorithm is used to perform intraclass feature alignment to the target domain in the feature-map level. S-NET is a U-net structure network that is used to provide semantic constraint by a segmentation loss for the training of IA-NET. Meanwhile, it offers pseudo-label images for calculating intraclass features of the target domain. Source code (in Tensorflow) is available at https://github.com/lb-whu/RAS-NET/. Results: Extensive experiments are carried out on two different data sets (Neo-BrainS12 and dHCP), respectively. There exist great differences in the shape, size, and intensity distribution of magnetic resonance (MR) images in the two databases. Compared to baseline, we improve the average dice score of all tissues on NeoBrains12 by 6% through adaptive training with unlabeled dHCP images. Besides, we also conduct experiments on dHCP and improved the average dice score by 4%. The quantitative analysis of the mean and variance of the synthesized images shows that the synthesized image by the proposed is closer to the target domain both in the full brain or within each class than that of the compared methods.

Lung lesion segmentation of CT scans after SARS-CoV-2 infection: combining nonhuman primate with human data for inter-species transfer learning

“…In the realm of unsupervised learning and generative AI, future work could focus on developing models that learn how imaging abnormalities after various respiratory infections present in animal models and then generate/simulate scans with these infections for humans. Such work is well-suited for generative adversarial networks, which were successfully applied experimentally in brain magnetic resonance imaging (MRI) 14 . Additionally, DL/AI work leverages and adapts existing methods or data to new modalities, as demonstrated by successful implementation of unsupervised learning and selfsupervised learning for domain adaption of different disease modalities as well as various organs and imaging capabilities [15][16][17][18][19][20][21] .…”

Exploring approaches to tackle cross-domain challenges in brain medical image segmentation: a systematic review

“…Moreover, the application of advanced architectures, such as 3D-CNNs (Ji et al, 2013 ), Transformers (Vaswani et al, 2017 ), and UNets, has further enhanced the performance of brain image segmentation across different domains (Dolz et al, 2018 ; Goubran et al, 2020 ; Huang et al, 2020 ; Liu Y. et al, 2020 ; Basak et al, 2021 ; Li et al, 2021 ; Meyer et al, 2021 ; Sun et al, 2021 ; Zhao et al, 2021 ). These models have been applied to various brain structures and conditions, including white matter, brain tumors, multiple sclerosis, and stroke (Erus et al, 2018 ; Knight et al, 2018 ; Ravnik et al, 2018 ; Reiche et al, 2019 ; Basak et al, 2021 ; Jiang et al, 2021 ; Kruger et al, 2021 ; Li et al, 2021 ; Sun et al, 2021 ; Kaffenberger et al, 2022 ; Zhou et al, 2022 ; Liu D. et al, 2023 ; Yu et al, 2023b ; Zhang et al, 2023 ).…”