Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration

Abstract

Purpose Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. In this paper we propose a method for fully automatic registration using anatomical annotations produced by a neural network.Methods Neural networks are trained to simultaneously segment anatomy and identify landmarks in fluoroscopy. Training data is obtained using a computationallyintensive, intraoperatively incompatible, 2D/3D registration of the pelvis and each femur. Ground truth 2D segmentation labels and anatomical landmark locations are established using projected 3D annotations. Intraoperative registration couples a traditional intensitybased strategy with annotations inferred by the network and requires no human assistance.

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

“…Studies summarized in this theme use machine learning techniques to increase the information available to the 2D/3D registration problem by extracting semantic information from the 2D or 3D data (Lin and Winey, 2012;Varnavas et al, 2013Varnavas et al, , 2015bBier et al, 2018;Chen et al, 2018;Bier et al, 2019;Luo et al, 2019;Yang and Chen, 2019;Grupp et al, 2020c;Doerr et al, 2020; TABLE 1 | Parameters defining the registration problems described in the studies included for review. Registration purpose refers to the registration stage being addressed, such as initialization (init.…”

“…It is well known that most commonly used metrics fail to accurately represent the distances in geometric parameter space that generate the mismatch between the current observations. It is thus not surprising that ten studies describe methods to better model and quantify the similarity between the source and target images to increase the capture range, and thus, the likelihood of registration success (Tang and Scalzo, 2016;Liao et al, 2019;Schaffert et al, 2019Schaffert et al, , 2020bSchaffert et al, 2020a;Gao et al, 2020c;Grupp et al, 2020c;Francois et al, 2020;Gu et al, 2020;Neumann et al, 2020). Some studies propose novel image similarity functions S θS (•, •) that, analogous to traditional similarity metrics, accept as input the source and target image and return a scalar or vector that is related to the mismatch in parameter space (Francois et al, 2020;Gu et al, 2020;Tang and Scalzo, 2016;Neumann et al, 2020;Grupp et al, 2020c;Gao et al, 2020b).…”

“…It is thus not surprising that ten studies describe methods to better model and quantify the similarity between the source and target images to increase the capture range, and thus, the likelihood of registration success (Tang and Scalzo, 2016;Liao et al, 2019;Schaffert et al, 2019Schaffert et al, , 2020bSchaffert et al, 2020a;Gao et al, 2020c;Grupp et al, 2020c;Francois et al, 2020;Gu et al, 2020;Neumann et al, 2020). Some studies propose novel image similarity functions S θS (•, •) that, analogous to traditional similarity metrics, accept as input the source and target image and return a scalar or vector that is related to the mismatch in parameter space (Francois et al, 2020;Gu et al, 2020;Tang and Scalzo, 2016;Neumann et al, 2020;Grupp et al, 2020c;Gao et al, 2020b). Among those, two methods rely on regularization: Grupp et al (2020c) detect anatomical landmarks to expand an analytic similarity function with landmark-reprojection constraints to enhance the capture range of an intensity-based strategy, while Francois et al (2020) segment occluding contours to constrain similarity evaluation to salient regions.…”

“…Two other studies that deal with multiple objects, however, are limited to object detection ( Doerr et al, 2020 ) and inpainting ( Esfandiari et al, 2021 ), respectively, and do not report registration results. The remaining study ( Grupp et al, 2020c ) performs a 2D segmentation of multiple bones, but does not apply any additional learning to perform the registration. While there are three methods that do, in fact, describe the 2D/3D registration of multiple objects, i.e., vertebral bodies ( Varnavas et al, 2013 ; 2015b ) and knee anatomy ( Wang et al, 2020 ), the individual registrations are solved independently which inhibits information sharing to ease the optimization problem.…”

A new 2D‐3D registration gold‐standard dataset for the hip joint based on uncertainty modeling

“…Some works proposed more realistic DRR images but were highly specific to body areas (e.g., chest 31 ), although recent work leveraging Deep Learning advances look very promising (e.g., DeepDRR 32 ). To the authors' knowledge, only one validation dataset of the hip joint 33 with real fluoroscopic images currently exists.…”