PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network at Unpaired Cross-Modality Cardiac Segmentation

Abstract

Deep convolutional networks have demonstrated the state-of-the-art performance on various medical image computing tasks. Leveraging images from different modalities for the same analysis task holds clinical benefits. However, the generalization capability of deep models on test data with different distributions remain as a major challenge. In this paper, we propose the PnP-AdaNet (plug-and-play adversarial domain adaptation network) for adapting segmentation networks between different modalities of medical images, e.g., MRI and CT. We propose to tackle the significant domain shift by aligning the feature spaces of source and target domains in an unsupervised manner. Specifically, a domain adaptation module flexibly replaces the early encoder layers of the source network, and the higher layers are shared between domains. With adversarial learning, we build two discriminators whose inputs are respectively multi-level features and predicted segmentation masks. We have validated our domain adaptation method on cardiac structure segmentation in unpaired MRI and CT. The experimental results with comprehensive ablation studies demonstrate the excellent efficacy of our proposed PnP-AdaNet. Moreover, we introduce a novel benchmark on the cardiac dataset for the task of unsupervised cross-modality domain adaptation. We will make our code and database publicly available, aiming to promote future studies on this challenging yet important research topic in medical imaging.

Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation

“…However, for the abdominal images, the adaptation performance for both directions are equivalently high and close to the supervised training upper bound. This indicates that the difficulty of domain adaptation across modalities might depend more on the task than the adaptation direction, which adds new findings over the previous work [44]. Potential future studies in terms of different segmentation tasks will help further analyze this issue.…”

“…To validate the effectiveness of our segmentation backbone, we compare the supervised training performance of our segmentation model on cardiac dataset with Payer et al [54], which obtained the first ranking in the MMWHS Challenge 2017. Table III shows that our model can achieve comparable performance to Payer et al But unlike [7], [44] which prefers network architectures without skip connections, we consider that other network architectures could also be used in our framework, such as the Unet model [1], which is the most common networks for medical image segmentation. The generator can be directly implemented as a Unet model, which has been demonstrated to be effective and stable during training for unpaired image-to-image transformation 1 .…”

“…A higher Dice value and a lower ASD value indicate better segmentation results. The evaluation is performed on the subject-level segmentation volume to be consistent with the MMWHS and CHAOS challenges as well as previous works [37], [44].…”

“…In [17], [42], [43], the discriminator is adopted to differentiate the semantic predictions generated from the extracted features, demonstrating more effective feature alignment. For cross-modality cardiac segmentation in [7], [44], the adversarial learning is employed to only adapt the early-layer feature distributions while the higher-layer features are fixed. However, their method needs comprehensive empirical studies to determine the optimal adaptation depth.…”

Deep Learning for Cardiac Image Segmentation: A Review

“…Several researchers have recently started to investigate the use of unsupervised domain adaptation techniques that aim at optimizing the model performance on unseen datasets without additional labeling costs. Several works have successfully applied adversarial training for cross-modality segmentation tasks, adapting a cardiac segmentation model learned from MR images to CT images and vice versa (Dou et al, 2018(Dou et al, , 2019Ouyang et al, 2019;Chen et al, 2019c). These type of approaches can also be adopted for semi-supervised learning, where the target domain is a new set of unlabeled data of the same modality (Chen et al, 2019d).…”