Deep learning‐based X‐ray inpainting for improving spinal 2D‐3D registration

Abstract

Background Two‐dimensional (2D)‐3D registration is challenging in the presence of implant projections on intraoperative images, which can limit the registration capture range. Here, we investigate the use of deep‐learning‐based inpainting for removing implant projections from the X‐rays to improve the registration performance. Methods We trained deep‐learning‐based inpainting models that can fill in the implant projections on X‐rays. Clinical datasets were collected to evaluate the inpainting based on six image similarity measures. The effect of X‐ray inpainting on capture range of 2D‐3D registration was also evaluated. Results The X‐ray inpainting significantly improved the similarity between the inpainted images and the ground truth. When applying inpainting before the 2D‐3D registration process, we demonstrated significant recovery of the capture range by up to 85%. Conclusion Applying deep‐learning‐based inpainting on X‐ray images masked by implants can markedly improve the capture range of the associated 2D‐3D registration task.

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

“…Perhaps the most striking observation is that all but three (45) included studies only consider the registration of a single object. Two other studies that deal with multiple objects, however, are limited to object detection ( Doerr et al, 2020 ) and inpainting ( Esfandiari et al, 2021 ), respectively, and do not report registration results. The remaining study ( Grupp et al, 2020c ) performs a 2D segmentation of multiple bones, but does not apply any additional learning to perform the registration.…”

“…Studies summarized in this theme use machine learning techniques to increase the information available to the 2D/3D registration problem by extracting semantic information from the 2D or 3D data ( Lin and Winey, 2012 ; Varnavas et al, 2013 , 2015b ; Bier et al, 2018 ; Chen et al, 2018 ; Bier et al, 2019 ; Luo et al, 2019 ; Yang and Chen, 2019 ; Grupp et al, 2020c ; Doerr et al, 2020 ; Francois et al, 2020 ; Karner et al, 2020 ; Wang et al, 2020 ; Esfandiari et al, 2021 ).…”

“…A complementary trend is the identification of image regions ( Francois et al, 2020 ; Esfandiari et al, 2021 ) or whole images ( Luo et al, 2019 ) that should not contribute to the optimization problem because of inconsistency. Francois et al (2020) use a U-net-like fully convolutional NN (FCN) to segment occluding contours of the uterus to reject regions while Luo et al (2019) identify and reject poor quality frames in bronchoscopy.…”

“… Francois et al (2020) use a U-net-like fully convolutional NN (FCN) to segment occluding contours of the uterus to reject regions while Luo et al (2019) identify and reject poor quality frames in bronchoscopy. Esfandiari et al (2021) consider mismatch introduced by intra-operative instrumentation. They contribute a non-blind image inpainting method using a FCN that seeks to restore the background anatomy after image regions corresponding to instruments were identified.…”

“…2 Indeed, among the seven studies that trained deep CNNs on synthetic data and evaluated in a way that allowed for comparisons between synthetic and real data performance, we found quite substantial performance drops ( Miao et al, 2018 ; Bier et al, 2019 ; Gao et al, 2020c ; Doerr et al, 2020 ; Gu et al, 2020 ; Guan et al, 2020 ; Li et al, 2020 ). Worse, three studies used different evaluation metrics in synthetic and real experiments so that comparison was not possible ( Miao et al, 2016a ; Toth et al, 2018 ; Esfandiari et al, 2021 ), and perhaps worst, ten studies that trained on synthetic data never even tested (meaningfully) on real data ( Hou et al, 2017 , 2018 ; Pei et al, 2017 ; Xie et al, 2017 ; Bier et al, 2018 ; Foote et al, 2019 ; Guan et al, 2019 ; Yang and Chen, 2019 ; Neumann et al, 2020 ; Zhang et al, 2020 ). To mitigate the negative impact of domain shift, there is growing interest in domain adaptation and generalization techniques.…”