Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration

Gao, Cong; Liu, Xingtong; Gu, Wenhao; Killeen, Benjamin D.; Armand, Mehran; Taylor, Russell H.; Unberath, Mathias

doi:10.1007/978-3-030-59716-0_32

Cited by 34 publications

(38 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…is most pronounced. 2 Indeed, among the seven studies that trained deep CNNs on synthetic data and evaluated in a way that allowed for comparisons between synthetic and real data performance, we found quite substantial performance drops (Miao et al, 2018;Bier et al, 2019;Gao et al, 2020c;Doerr et al, 2020;Gu et al, 2020;Guan et al, 2020;Li et al, 2020). Worse, three studies used different evaluation metrics in synthetic and real experiments so that comparison was not possible (Miao et al, 2016a;Toth et al, 2018;Esfandiari et al, 2021), and perhaps worst, ten studies that trained on synthetic data never even tested (meaningfully) on real data (Hou et al, 2017(Hou et al, , 2018Pei et al, 2017;Xie et al, 2017;Bier et al, 2018;Foote et al, 2019;Guan et al, 2019;Yang and Chen, 2019;Neumann et al, 2020;Zhang et al, 2020).…”

Section: Preserving Improvements Under Domain Shift From Training To Deploymentmentioning

confidence: 92%

“…In many other-from a research perspective perhaps more exciting-cases, this retrospective data collection paradigm is infeasible because the task to be performed with a machine learning algorithm is not currently performed in clinical practice. The more obvious examples are visual servoing of novel robotic surgery platforms (Gao et al, 2019) or robotic imaging paradigms that alter how data is acquired (Zaech et al, 2019;Thies et al, 2020). Despite the fact that most studies included in this review address use-cases that fall under the "opportunity cost" category, we found that only 16 out of the 48 studies used real clinical or cadaveric data to train the machine learning algorithms.…”

Section: Preserving Improvements Under Domain Shift From Training To Deploymentmentioning

confidence: 97%

“…It is well known that most commonly used metrics fail to accurately represent the distances in geometric parameter space that generate the mismatch between the current observations. It is thus not surprising that ten studies describe methods to better model and quantify the similarity between the source and target images to increase the capture range, and thus, the likelihood of registration success (Tang and Scalzo, 2016;Liao et al, 2019;Schaffert et al, 2019Schaffert et al, , 2020bSchaffert et al, 2020a;Gao et al, 2020c;Grupp et al, 2020c;Francois et al, 2020;Gu et al, 2020;Neumann et al, 2020). Some studies propose novel image similarity functions S θS (•, •) that, analogous to traditional similarity metrics, accept as input the source and target image and return a scalar or vector that is related to the mismatch in parameter space (Francois et al, 2020;Gu et al, 2020;Tang and Scalzo, 2016;Neumann et al, 2020;Grupp et al, 2020c;Gao et al, 2020b).…”

Section: Similarity Modelingmentioning

confidence: 99%

“…Certainly, some of the above ideas will become increasingly important in the quest to accelerate and improve 2D/3D registration pipelines. This is because image-based navigation techniques (Sugano, 2003;Hummel et al, 2008;Tucker et al, 2018) as well as visual servoing of surgical robots (Yi et al, 2018;Gao et al, 2019;Unberath et al, 2019) will also require highprecision 2D/3D registration at video frame-rates.…”

Section: Perspectivementioning

confidence: 99%

Section: Systematic Reviewmentioning

confidence: 99%

See 4 more Smart Citations

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

show abstract

Section: Preserving Improvements Under Domain Shift From Training To Deploymentmentioning

confidence: 92%

Section: Preserving Improvements Under Domain Shift From Training To Deploymentmentioning

confidence: 97%

Section: Similarity Modelingmentioning

confidence: 99%

Section: Perspectivementioning

confidence: 99%

Section: Systematic Reviewmentioning

confidence: 99%

See 3 more Smart Citations

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

show abstract

Deformable motion compensation in interventional cone‐beam CT with a context‐aware learned autofocus metric

Huang,

Liu,

Siewerdsen

et al. 2024

Medical Physics

Self Cite

View full text Add to dashboard Cite

PurposeInterventional Cone‐Beam CT (CBCT) offers 3D visualization of soft‐tissue and vascular anatomy, enabling 3D guidance of abdominal interventions. However, its long acquisition time makes CBCT susceptible to patient motion. Image‐based autofocus offers a suitable platform for compensation of deformable motion in CBCT, but it relies on handcrafted motion metrics based on first‐order image properties and that lack awareness of the underlying anatomy. This work proposes a data‐driven approach to motion quantification via a learned, context‐aware, deformable metric, , that quantifies the amount of motion degradation as well as the realism of the structural anatomical content in the image.MethodsThe proposed was modeled as a deep convolutional neural network (CNN) trained to recreate a reference‐based structural similarity metric—visual information fidelity (VIF). The deep CNN acted on motion‐corrupted images, providing an estimation of the spatial VIF map that would be obtained against a motion‐free reference, capturing motion distortion, and anatomic plausibility. The deep CNN featured a multi‐branch architecture with a high‐resolution branch for estimation of voxel‐wise VIF on a small volume of interest. A second contextual, low‐resolution branch provided features associated to anatomical context for disentanglement of motion effects and anatomical appearance. The deep CNN was trained on paired motion‐free and motion‐corrupted data obtained with a high‐fidelity forward projection model for a protocol involving 120 kV and 9.90 mGy. The performance of was evaluated via metrics of correlation with ground truth and with the underlying deformable motion field in simulated data with deformable motion fields with amplitude ranging from 5 to 20 mm and frequency from 2.4 up to 4 cycles/scan. Robustness to variation in tissue contrast and noise levels was assessed in simulation studies with varying beam energy (90–120 kV) and dose (1.19–39.59 mGy). Further validation was obtained on experimental studies with a deformable phantom. Final validation was obtained via integration of on an autofocus compensation framework, applied to motion compensation on experimental datasets and evaluated via metric of spatial resolution on soft‐tissue boundaries and sharpness of contrast‐enhanced vascularity.ResultsThe magnitude and spatial map of showed consistent and high correlation levels with the ground truth in both simulation and real data, yielding average normalized cross correlation (NCC) values of 0.95 and 0.88, respectively. Similarly, achieved good correlation values with the underlying motion field, with average NCC of 0.90. In experimental phantom studies, properly reflects the change in motion amplitudes and frequencies: voxel‐wise averaging of the local across the full reconstructed volume yielded an average value of 0.69 for the case with mild motion (2 mm, 12 cycles/scan) and 0.29 for the case with severe motion (12 mm, 6 cycles/scan). Autofocus motion compensation using resulted in noticeable mitigation of motion artifacts and improved spatial resolution of soft tissue and high‐contrast structures, resulting in reduction of edge spread function width of 8.78% and 9.20%, respectively. Motion compensation also increased the conspicuity of contrast‐enhanced vascularity, reflected in an increase of 9.64% in vessel sharpness.ConclusionThe proposed , featuring a novel context‐aware architecture, demonstrated its capacity as a reference‐free surrogate of structural similarity to quantify motion‐induced degradation of image quality and anatomical plausibility of image content. The validation studies showed robust performance across motion patterns, x‐ray techniques, and anatomical instances. The proposed anatomy‐ and context‐aware metric poses a powerful alternative to conventional motion estimation metrics, and a step forward for application of deep autofocus motion compensation for guidance in clinical interventional procedures.

show abstract

3D-2D Registration Using X-Ray Simulation and CMA-ES

Wen

Mihail

Vidal

2021

Applications of Evolutionary Computation

View full text Add to dashboard Cite

Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration

Cited by 34 publications

References 21 publications

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Deformable motion compensation in interventional cone‐beam CT with a context‐aware learned autofocus metric

3D-2D Registration Using X-Ray Simulation and CMA-ES

Contact Info

Product

Resources

About