Pairwise domain adaptation module for CNN-based 2-D/3-D registration

Abstract

Accurate two-dimensional to three-dimensional (2-D/3-D) registration of preoperative 3-D data and intraoperative 2-D x-ray images is a key enabler for image-guided therapy. Recent advances in 2-D/3-D registration formulate the problem as a learning-based approach and exploit the modeling power of convolutional neural networks (CNN) to significantly improve the accuracy and efficiency of 2-D/3-D registration. However, for surgery-related applications, collecting a large clinical dataset with accurate annotations for training can be very challenging or impractical. Therefore, deep learning-based 2-D/3-D registration methods are often trained with synthetically generated data, and a performance gap is often observed when testing the trained model on clinical data. We propose a pairwise domain adaptation (PDA) module to adapt the model trained on source domain (i.e., synthetic data) to target domain (i.e., clinical data) by learning domain invariant features with only a few paired real and synthetic data. The PDA module is designed to be flexible for different deep learning-based 2-D/3-D registration frameworks, and it can be plugged into any pretrained CNN model such as a simple Batch-Norm layer. The proposed PDA module has been quantitatively evaluated on two clinical applications using different frameworks of deep networks, demonstrating its significant advantages of generalizability and flexibility for 2-D/3-D medical image registration when a small number of paired real-synthetic data can be obtained.

Deep learning in medical image registration: a review

“…They claimed that the learnt similarity metric outperformed MI and its variant MIND [63]. [92] Spine, Cardiac 3D-3D CT-CBCT Rigid [101] Chest, Abdomen 2D-2D CT-Depth Image Rigid [108] Spine 3D-2D 3DCT-Xray Rigid [183] Spine 3D-2D 3DCT-Xray Rigid [144] Nasopharyngeal 2D-2D MR-CT Rigid with scaling…”

“…Adequate data augmentation could be performed to mitigate the second limitation. Domain adaption [41,183] could be used to account for the domain difference between the artificially-generated and the true images. Image registration is an ill-posed problem, the ground truth transformation could help to constrain the final transformation prediction.…”

“…They demonstrated that the network could rapidly and accurately quantify registration performance. Multi-grid Inference [163] Brain 3D-3D MR-US Rigid LSTM [8] Brain 2D-2D CT, T1, T2, PD Rigid Manifold Learning [165] Brain, Abdomen 2D-3D CT-PET, CT-MRI Deformable CAE, DSCNN [178] Spine 3D-2D 3DCT-Xray Rigid FasterRCNN [54] Brain 2D-2D T1-T2, T1-PD Deformable FCN [183] Spine 3D-2D 3DCT-Xray Rigid Domain adaptation…”

Deep learning in medical image registration: a survey

“…Additionally, Zheng et al [119] proposed the integration of a pairwise domain adaptation module (PDA) into a pre-trained CNN that performs the rigid registration of pre-operative 3D X-Ray images and intraoperative 2D X-ray images using a limited amount of training data. Domain adaptation was used to address the discrepancy between synthetic data that was used to train the deep model and real data.…”

Communal Domain Learning for Registration in Drifted Image Spaces

“…Typical approaches to handle drifts use either information theoretic similarity metrics such as mutual information or normalized cross correlation that maximize the transferability of knowledge between domains [1]. In addition, significant research effort has been invested in devising methods for the effective representations of registering domains, more recently using the deep learning approaches [2][3][4]. These representation transformation approaches known as domain adaptation or transfer learning methods range from simplistic techniques such as intensity standardization to more sophisticated feature mapping approaches [5].…”