Data-driven detection and registration of spine surgery instrumentation in intraoperative images

Doerr, Sophia A.; Uneri, Ali; Huang, Yixuan; Jones, Craig K.; Zhang, Xiaoxuan; Ketcha, Michael D.; Helm, Patrick A.; Siewerdsen, Jeffrey H.

doi:10.1117/12.2550052

Cited by 6 publications

(16 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…is most pronounced. 2 Indeed, among the seven studies that trained deep CNNs on synthetic data and evaluated in a way that allowed for comparisons between synthetic and real data performance, we found quite substantial performance drops (Miao et al, 2018;Bier et al, 2019;Gao et al, 2020c;Doerr et al, 2020;Gu et al, 2020;Guan et al, 2020;Li et al, 2020). Worse, three studies used different evaluation metrics in synthetic and real experiments so that comparison was not possible (Miao et al, 2016a;Toth et al, 2018;Esfandiari et al, 2021), and perhaps worst, ten studies that trained on synthetic data never even tested (meaningfully) on real data (Hou et al, 2017(Hou et al, , 2018Pei et al, 2017;Xie et al, 2017;Bier et al, 2018;Foote et al, 2019;Guan et al, 2019;Yang and Chen, 2019;Neumann et al, 2020;Zhang et al, 2020).…”

Section: Preserving Improvements Under Domain Shift From Training To Deploymentmentioning

confidence: 92%

“…Studies summarized in this theme use machine learning techniques to increase the information available to the 2D/3D registration problem by extracting semantic information from the 2D or 3D data (Lin and Winey, 2012;Varnavas et al, 2013Varnavas et al, , 2015bBier et al, 2018;Chen et al, 2018;Bier et al, 2019;Luo et al, 2019;Yang and Chen, 2019;Grupp et al, 2020c;Doerr et al, 2020; TABLE 1 | Parameters defining the registration problems described in the studies included for review. Registration purpose refers to the registration stage being addressed, such as initialization (init.…”

Section: Contextualizationmentioning

confidence: 99%

“…Perhaps the most striking observation is that all but three (45) included studies only consider the registration of a single object. Two other studies that deal with multiple objects, however, are limited to object detection ( Doerr et al, 2020 ) and inpainting ( Esfandiari et al, 2021 ), respectively, and do not report registration results. The remaining study ( Grupp et al, 2020c ) performs a 2D segmentation of multiple bones, but does not apply any additional learning to perform the registration.…”

Section: Systematic Reviewmentioning

confidence: 99%

“…Studies summarized in this theme use machine learning techniques to increase the information available to the 2D/3D registration problem by extracting semantic information from the 2D or 3D data ( Lin and Winey, 2012 ; Varnavas et al, 2013 , 2015b ; Bier et al, 2018 ; Chen et al, 2018 ; Bier et al, 2019 ; Luo et al, 2019 ; Yang and Chen, 2019 ; Grupp et al, 2020c ; Doerr et al, 2020 ; Francois et al, 2020 ; Karner et al, 2020 ; Wang et al, 2020 ; Esfandiari et al, 2021 ).…”

Section: Systematic Reviewmentioning

confidence: 99%

“…Perhaps the most prevalent approach here is the detection of anatomical landmarks on 2D images ( Bier et al, 2018 , 2019 ; Yang and Chen, 2019 ; Grupp et al, 2020c ; Wang et al, 2020 ) to define correspondences with the respective 3D locations, which allows for either, explicit determination of

using PnP ( Bier et al, 2019 , 2018 ) or stereo-reconstruction following 3D-3D matching ( Yang and Chen, 2019 ; Wang et al, 2020 ), or for the introduction of soft re-projection constraints as a regularizing term ( Grupp et al, 2020c ). Another way of benefiting initialization through contextualization is object detection ( Lin and Winey, 2012 ; Varnavas et al, 2013 , 2015b ; Chen et al, 2018 ; Doerr et al, 2020 ). While all methods for landmark detection rely on deep CNNs, object detection achieved satisfactory results already using less complex learning models, i.e., templates in ( Lin and Winey, 2012 ), PCA over object contours in ( Chen et al, 2018 ), and the Generalized Hough Transform in ( Varnavas et al, 2013 ; 2015b ).…”

Section: Systematic Reviewmentioning

confidence: 99%

See 4 more Smart Citations

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

View full text Add to dashboard Cite

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

show abstract

Section: Preserving Improvements Under Domain Shift From Training To Deploymentmentioning

confidence: 92%

Section: Contextualizationmentioning

confidence: 99%

Section: Systematic Reviewmentioning

confidence: 99%

“…Studies summarized in this theme use machine learning techniques to increase the information available to the 2D/3D registration problem by extracting semantic information from the 2D or 3D data ( Lin and Winey, 2012 ; Varnavas et al, 2013 , 2015b ; Bier et al, 2018 ; Chen et al, 2018 ; Bier et al, 2019 ; Luo et al, 2019 ; Yang and Chen, 2019 ; Grupp et al, 2020c ; Doerr et al, 2020 ; Francois et al, 2020 ; Karner et al, 2020 ; Wang et al, 2020 ; Esfandiari et al, 2021 ).…”

Section: Systematic Reviewmentioning

confidence: 99%

Section: Systematic Reviewmentioning

confidence: 99%

See 3 more Smart Citations

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

Long-length tomosynthesis and 3D-2D registration for intraoperative assessment of spine instrumentation

Zhang¹,

Uneri²,

Wu³

et al. 2021

Phys. Med. Biol.

Self Cite

View full text Add to dashboard Cite

Purpose. A system for long-length intraoperative imaging is reported based on longitudinal motion of an O-arm gantry featuring a multi-slot collimator. We assess the utility of long-length tomosynthesis and the geometric accuracy of 3D image registration for surgical guidance and evaluation of long spinal constructs. Methods. A multi-slot collimator with tilted apertures was integrated into an O-arm system for long-length imaging. The multi-slot projective geometry leads to slight view disparity in both long-length projection images (referred to as ‘line scans’) and tomosynthesis ‘slot reconstructions’ produced using a weighted-backprojection method. The radiation dose for long-length imaging was measured, and the utility of long-length, intraoperative tomosynthesis was evaluated in phantom and cadaver studies. Leveraging the depth resolution provided by parallax views, an algorithm for 3D-2D registration of the patient and surgical devices was adapted for registration with line scans and slot reconstructions. Registration performance using single-plane or dual-plane long-length images was evaluated and compared to registration accuracy achieved using standard dual-plane radiographs. Results. Longitudinal coverage of ∼50–64 cm was achieved with a single long-length slot scan, providing a field-of-view (FOV) up to (40 × 64) cm2, depending on patient positioning. The dose-area product (reference point air kerma × x-ray field area) for a slot scan ranged from ∼702–1757 mGy·cm2, equivalent to ∼2.5 s of fluoroscopy and comparable to other long-length imaging systems. Long-length scanning produced high-resolution tomosynthesis reconstructions, covering ∼12–16 vertebral levels. 3D image registration using dual-plane slot reconstructions achieved median target registration error (TRE) of 1.2 mm and 0.6° in cadaver studies, outperforming registration to dual-plane line scans (TRE = 2.8 mm and 2.2°) and radiographs (TRE = 2.5 mm and 1.1°). 3D registration using single-plane slot reconstructions leveraged the ∼7–14° angular separation between slots to achieve median TRE ∼2 mm and <2° from a single scan. Conclusion. The multi-slot configuration provided intraoperative visualization of long spine segments, facilitating target localization, assessment of global spinal alignment, and evaluation of long surgical constructs. 3D-2D registration to long-length tomosynthesis reconstructions yielded a promising means of guidance and verification with accuracy exceeding that of 3D-2D registration to conventional radiographs.

show abstract

Deformable 3D-2D registration for high-precision guidance and verification of neuroelectrode placement

Uneri¹,

Wu²,

Jones³

et al. 2021

Phys. Med. Biol.

View full text Add to dashboard Cite

Purpose. Accurate neuroelectrode placement is essential to effective monitoring or stimulation of neurosurgery targets. This work presents and evaluates a method that combines deep learning and model-based deformable 3D-2D registration to guide and verify neuroelectrode placement using intraoperative imaging. Methods. The registration method consists of three stages: (1) detection of neuroelectrodes in a pair of fluoroscopy images using a deep learning approach; (2) determination of correspondence and initial 3D localization among neuroelectrode detections in the two projection images; and (3) deformable 3D-2D registration of neuroelectrodes according to a physical device model. The method was evaluated in phantom, cadaver, and clinical studies in terms of (a) the accuracy of neuroelectrode registration and (b) the quality of metal artifact reduction (MAR) in cone-beam CT (CBCT) in which the deformably registered neuroelectrode models are taken as input to the MAR. Results. The combined deep learning and model-based deformable 3D-2D registration approach achieved 0.2 ± 0.1 mm accuracy in cadaver studies and 0.6 ± 0.3 mm accuracy in clinical studies. The detection network and 3D correspondence provided initialization of 3D-2D registration within 2 mm, which facilitated end-to-end registration runtime within 10 s. Metal artifacts, quantified as the standard deviation in voxel values in tissue adjacent to neuroelectrodes, were reduced by 72% in phantom studies and by 60% in first clinical studies. Conclusions. The method combines the speed and generalizability of deep learning (for initialization) with the precision and reliability of physical model-based registration to achieve accurate deformable 3D-2D registration and MAR in functional neurosurgery. Accurate 3D-2D guidance from fluoroscopy could overcome limitations associated with deformation in conventional navigation, and improved MAR could improve CBCT verification of neuroelectrode placement.

show abstract

Data-driven detection and registration of spine surgery instrumentation in intraoperative images

Cited by 6 publications

References 8 publications

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Long-length tomosynthesis and 3D-2D registration for intraoperative assessment of spine instrumentation

Deformable 3D-2D registration for high-precision guidance and verification of neuroelectrode placement

Contact Info

Product

Resources

About