Deep deformable registration: Enhancing accuracy by fully convolutional neural net

Abstract

Deformable registration is ubiquitous in medical image analysis. Many deformable registration methods minimize sum of squared difference (SSD) as the registration cost with respect to deformable model parameters. In this work, we construct a tight upper bound of the SSD registration cost by using a fully convolutional neural network (FCNN) in the registration pipeline. The upper bound SSD (UB-SSD) enhances the original deformable model parameter space by adding a heatmap output from FCNN. Next, we minimize this UB-SSD by adjusting both the parameters of the FCNN and the parameters of the deformable model in coordinate descent. Our coordinate descent framework is end-to-end and can work with any deformable registration method that uses SSD. We demonstrate experimentally that our method enhances the accuracy of deformable registration algorithms significantly on two publicly available 3D brain MRI data sets.

Solving inverse problems using data-driven models

“…For the latter, note that the framework can handle any task that is given by a trainable neural network. This includes a wide range of tasks, such as semantic segmentation (Thoma 2016, Guo, Liu, Georgiou andLew 2018), caption generation (Karpathy andFei-Fei 2017, Li, Liang, Hu andXing 2018a), in-painting (Xie, Xu and Chen 2012), depixelization/super-resolution (Romano, Isidoro and Milanfar 2017b), demosaicing (Syu, Chen and Chuang 2018), image translation (Wolterink et al 2017), object recognition (Sermanet et al 2013, He et al 2016, Farabet, Couprie, Najman and LeCun 2013 and non-rigid image registration (Yang, Kwitt, Styner and Niethammer 2017, Ghosal and Ray 2017, Dalca, Balakrishnan, Guttag and Sabuncu 2018, Balakrishnan et al 2019. Section 7.6 shows the performance of task-adapted reconstruction for joint tomographic image reconstruction and segmentation of white brain matter.…”

“…2013, He et al. 2016, Farabet, Couprie, Najman and LeCun 2013) and non-rigid image registration (Yang, Kwitt, Styner and Niethammer 2017, Ghosal and Ray 2017, Dalca, Balakrishnan, Guttag and Sabuncu 2018, Balakrishnan et al. 2019).…”

Accelerated white matter lesion analysis based on simultaneous T1 and T2∗ quantification using magnetic resonance fingerprinting and deep learning

“…Furthermore, image synthesis has gained attention, which transforms a set of input images to a new set of image contrasts 29,30 . These image transformations can also contain deformable registration and artifact correction which showed good accuracy using CNN’s 31‐33 …”

“…29,30 These image transformations can also contain deformable registration and artifact correction which showed good accuracy using CNN's. [31][32][33] Especially for MRF, several models using fully connected neuronal network, 34,35 recurrent and convolutional neuronal network (CNN) 18,[36][37][38] were analyzed showing promising results regarding the speed and accuracy of the reconstruction. 39 A deep learning reconstruction on MRF data using the spatiotemporal relationship between neighboring signal evolutions was proposed, 40,41 which showed an improvement in the reconstruction especially for undersampled complex MRF data.…”

ProsRegNet: A deep learning framework for registration of MRI and histopathology images of the prostate

“…A deep learning based registration network can be considered as a function that takes two images, a fixed image and a moving image, as the input and directly outputs a unique transformation without requiring additional optimization. Many deep learning approaches ( Balakrishnan et al, 2018 , 2019 ); Dalca et al (2018) ; Ghosal and Ray (2017) ; Krebs et al (2019) ; Yang et al (2017) ; Zhang (2018) assume that the fixed and moving images have already been aligned by affine registration and only focus on the deformable registration. However, the affine registration of MRI and histopathology images of the prostate is challenging since they are considerably different modalities while having different contents.…”