Model-to-Image Registration via Deep Learning towards Image-Guided Endovascular Interventions

Abstract

Cardiologists highlight the need for an intraoperative 3D visualization to assist interventions. The intraoperative 2D X-ray/Digital Subtraction Angiography (DSA) images in the standard clinical workflow limit cardiologists' views significantly. Compared with image-to-image registration, model-to-image registration is an essential approach taking advantage of the reuse of pre-operative 3D models reconstructed from Computed Tomography Angiography (CTA) images. Traditional optimized-based registration methods suffer severely from high computational complexity. Moreover, the consequence of lacking ground truth for learning-based registration approaches should not be neglected. To overcome these challenges, we introduce a model-to-image registration framework via deep learning for image-guided endovascular catheterization. This work performs autonomous vessel segmentation from intra-operative fluoroscopy images via a deep residual U-net and a model-to-image matching via a convolutional neural network. For this study, image data were collected from 10 patients who performed Transcatheter Aortic Valve Implantation (TAVI) procedures. It was found that vessel segmentation of test data results in median values of Dice Similarity Coefficient, Precision, and Recall of (0.75, 0.58, 0.67) for femoral artery, and (0.71, 0.56, 0.74) for aortic root. The segmentation network behaves better than manual annotation, and it recognizes part of vessels that were not labeled manually. Image matching between the transformed moving image and the fixed image results in a median value of Recall of 0.90. The proposed approach achieves a good accuracy of vessel segmentation and a good recall value of model-toimage matching.

Deformable Model-to-Image Registration Toward Augmented Reality-Guided Endovascular Interventions

“…The first strategy involves Electrocardiogram (ECG)-synchronized CTA of the aortic root and heart, followed by non-ECG-synchronized helical CTA of the thorax, abdomen, and pelvis. The second strategy comprises ECG-synchronized CTA of the thorax, followed by non-ECG-synchronized helical CTA of the abdomen and pelvis; 2) 2D fluoroscopy image segmentation: acquisition of intra-operative fluoroscopy images depicting various Field-of-View (FoV) along the insertion route, followed by automatic segmentation using a DRU-Net to generate binary images [10]. These fluoroscopic images predominantly focus on two main FoVs during interventions: the entry site, typically the femoral arteries, and the target site, generally the aortic root.…”

“…These fluoroscopy images are typically captured at key stages of the intervention: firstly, following the insertion of the needle into the femoral arteries; secondly, prior to the inflation of the balloon catheter; and finally, subsequent to the placement of the stent at the aortic root; 3) Affine model-to-image registration: conversion of the model-to-image registration problem into an image-toimage registration problem through the projection of a 2D view of the Region of Interest (ROI) from the 3D model, based on the fluoroscopy image. An CNN model is employed to estimate an affine registration matrix, which aligns the ROI projection with the binary image segmented from the fluoroscopy image [10]. The ROI of a patient-specific model is determined interactively by delineating a bounding box on a projection image of the 3D model.…”

“…The ROI of a patient-specific model is determined interactively by delineating a bounding box on a projection image of the 3D model. This process involves the selection of two distinct ROIs for each patient: one encompassing a lower FoV, which includes the femoral arteries, and the other covering an upper FoV, providing visibility of the ascending aorta and aortic root; Extensions in comparison to our previous work [10]:…”

“…4) Deformable model-to-image registration: estimation of the deformation field describing the vessel deformation between the ROI projection and the segmented image using a DRU-Net model; 5) 3D model registration: application of the 2D affine registration matrix and deformation field obtained from the previous modules to the pre-operative 3D model using a Free Form Deformation (FFD) algorithm [24]; 6) Augmented reality visualization: visualization of the deformed 3D model, along with the fluoroscopy image and deformation field, using the AR HMD device. Our contribution to this extended framework introduced in comparison to the previous work [10] is depicted by the red dashed box in Fig. 1, including the model deformation prediction using a DRU-Net and the reconstruction of de-formations onto the pre-operative 3D model.…”

“…To overcome the aforementioned challenges, this paper presents a novel deformable model-to-image registration framework using deep learning, specifically tailored for augmented reality-guided endovascular catheterization. Building upon our previous work [10], which proposed an affine modelto-image registration approach to align segmented fluoroscopy images with a pre-operative 3D model reconstructed from Computed Tomography Angiography (CTA) scans, this study extends the registration pipeline. We introduce a phase for deformation prediction and reconstruction and incorporate immersive AR visualization using a Head-Mounted Display (HMD) device.…”