A Deep Metric for Multimodal Registration

Simonovsky, Martin; Gutiérrez-Becker, Benjamín; Mateus, Diana; Navab, Nassir; Komodakis, Nikos

doi:10.1007/978-3-319-46726-9_2

Cited by 150 publications

(131 citation statements)

References 12 publications

(23 reference statements)

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…They employed latent Dirichlet allocation (LDA), a type of stochastic model that generates a distribution over a vocabulary of topics based on words in a document. In a later work, Shin et al (2016a) proposed a sys- RBM fMRI blind source separation; RBM for both internal and functional interaction-induced latent sources detection Simonovsky et al (2016) CNN Similarity measurement; 3D CNN estimating similarity between reference and moving images stacked in the input Wu et al (2013) ISA Correspondence detection in deformable registration; stacked convolutional ISA for unsupervised feature learning Yang et al (2016d) CNN Image registration; Conv. encoder-decoder net.…”

Section: Combining Image Data With Reportsmentioning

confidence: 99%

“…Broadly speaking, two strategies are prevalent in current literature: (1) using deep-learning networks to estimate a similarity measure for two images to drive an iterative optimization strategy, and (2) to directly predict transformation parameters using deep regression networks. Wu et al (2013), Simonovsky et al (2016), and Cheng et al (2015) used the first strategy to try to optimize registration algorithms. Cheng et al (2015) used two types of stacked auto-encoders to assess the local similarity between CT and MRI images of the head.…”

Section: Registrationmentioning

confidence: 99%

See 1 more Smart Citation

A survey on deep learning in medical image analysis

Litjens

Kooi

Bejnordi

et al. 2017

Medical Image Analysis

9,175

5,414

View full text Add to dashboard Cite

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks. Concise overviews are provided of studies per application area: neuro, retinal, pulmonary, digital pathology, breast, cardiac, abdominal, musculoskeletal. We end with a summary of the current state-of-the-art, a critical discussion of open challenges and directions for future research.

show abstract

Section: Combining Image Data With Reportsmentioning

confidence: 99%

Section: Registrationmentioning

confidence: 99%

A survey on deep learning in medical image analysis

Litjens

Kooi

Bejnordi

et al. 2017

Medical Image Analysis

9,175

5,414

View full text Add to dashboard Cite

show abstract

“…In synergy with using a CT synth bridge, improvements in mono‐modality registration would also lead to better multimodal registration in the head‐and‐neck. Currently, research using neural networks offers some exciting new avenues in this regard, including completely learning‐based unsupervised DVF generation . However, the performance of these methods depends on the availability and quality of training sets, which are particularly challenging for multimodel registration.…”

Section: Discussionmentioning

confidence: 99%

Multimodality image registration in the head‐and‐neck using a deep learning‐derived synthetic CT as a bridge

et al. 2020

View full text Add to dashboard Cite

Purpose To develop and demonstrate the efficacy of a novel head‐and‐neck multimodality image registration technique using deep‐learning‐based cross‐modality synthesis. Methods and Materials Twenty‐five head‐and‐neck patients received magnetic resonance (MR) and computed tomography (CT) (CTaligned) scans on the same day with the same immobilization. Fivefold cross validation was used with all of the MR‐CT pairs to train a neural network to generate synthetic CTs from MR images. Twenty‐four of 25 patients also had a separate CT without immobilization (CTnon‐aligned) and were used for testing. CTnon‐aligned's were deformed to the synthetic CT, and compared to CTnon‐aligned registered to MR. The same registrations were performed from MR to CTnon‐aligned and from synthetic CT to CTnon‐aligned. All registrations used B‐splines for modeling the deformation, and mutual information for the objective. Results were evaluated using the 95% Hausdorff distance among spinal cord contours, landmark error, inverse consistency, and Jacobian determinant of the estimated deformation fields. Results When large initial rigid misalignment is present, registering CT to MRI‐derived synthetic CT aligns the cord better than a direct registration. The average landmark error decreased from 9.8 ± 3.1 mm in MR→CTnon‐aligned to 6.0 ± 2.1 mm in CTsynth→CTnon‐aligned deformable registrations. In the CT to MR direction, the landmark error decreased from 10.0 ± 4.3 mm in CTnon‐aligned→MR deformable registrations to 6.6 ± 2.0 mm in CTnon‐aligned→CTsynth deformable registrations. The Jacobian determinant had an average value of 0.98. The proposed method also demonstrated improved inverse consistency over the direct method. Conclusions We showed that using a deep learning‐derived synthetic CT in lieu of an MR for MR→CT and CT→MR deformable registration offers superior results to direct multimodal registration.

show abstract

“…However, those dissimilarity metrics were not able to accurately reflect the dissimilarity distances between image patches that are subject to significant noise and irregular deformations. Various metrics have been proposed using deep‐learning methods in order to improve the accuracy of the dissimilarity measures . Such learning‐based dissimilarity measures have outperformed the conventional hand‐crafted dissimilarity measures.…”

Section: Methodsmentioning

confidence: 99%

Automatic large quantity landmark pairs detection in 4DCT lung images

Thomas

et al. 2019

Medical Physics

View full text Add to dashboard Cite

Purpose To automatically and precisely detect a large quantity of landmark pairs between two lung computed tomography (CT) images to support evaluation of deformable image registration (DIR). We expect that the generated landmark pairs will significantly augment the current lung CT benchmark datasets in both quantity and positional accuracy. Methods A large number of landmark pairs were detected within the lung between the end‐exhalation (EE) and end‐inhalation (EI) phases of the lung four‐dimensional computed tomography (4DCT) datasets. Thousands of landmarks were detected by applying the Harris‐Stephens corner detection algorithm on the probability maps of the lung vasculature tree. A parametric image registration method (pTVreg) was used to establish initial landmark correspondence by registering the images at EE and EI phases. A multi‐stream pseudo‐siamese (MSPS) network was then developed to further improve the landmark pair positional accuracy by directly predicting three‐dimensional (3D) shifts to optimally align the landmarks in EE to their counterparts in EI. Positional accuracies of the detected landmark pairs were evaluated using both digital phantoms and publicly available landmark pairs. Results Dense sets of landmark pairs were detected for 10 4DCT lung datasets, with an average of 1886 landmark pairs per case. The mean and standard deviation of target registration error (TRE) were 0.47 ± 0.45 mm with 98% of landmark pairs having a TRE smaller than 2 mm for 10 digital phantom cases. Tests using 300 manually labeled landmark pairs in 10 lung 4DCT benchmark datasets (DIRLAB) produced TRE results of 0.73 ± 0.53 mm with 97% of landmark pairs having a TRE smaller than 2 mm. Conclusion A new method was developed to automatically and precisely detect a large quantity of landmark pairs between lung CT image pairs. The detected landmark pairs could be used as benchmark datasets for more accurate and informative quantitative evaluation of DIR algorithms.

show abstract

A Deep Metric for Multimodal Registration

Cited by 150 publications

References 12 publications

A survey on deep learning in medical image analysis

A survey on deep learning in medical image analysis

Multimodality image registration in the head‐and‐neck using a deep learning‐derived synthetic CT as a bridge

Automatic large quantity landmark pairs detection in 4DCT lung images

Contact Info

Product

Resources

About