Real-Time 2D/3D Deformable Registration Using Metric Learning

Abstract

Abstract. We present a novel 2D/3D deformable registration method, called Registration Efficiency and Accuracy through Learning Metric on Shape (REALMS ), that can support real-time Image-Guided Radiation Therapy (IGRT ). The method consists of two stages: planning-time learning and registration. In the planning-time learning, it firstly models the patient's 3D deformation space from the patient's time-varying 3D planning images using a low-dimensional parametrization. Secondly, it samples deformation parameters within the deformation space and generates corresponding simulated projection images from the deformed 3D image. Finally, it learns a Riemannian metric in the projection space for each deformation parameter. The learned distance metric forms a Gaussian kernel of a kernel regression that minimizes the leave-one-out regression residual of the corresponding deformation parameter. In the registration, REALMS interpolates the patient's 3D deformation parameters using the kernel regression with the learned distance metrics. Our test results showed that REALMS can localize the tumor in 10.89 ms (91.82 fps) with 2.56 ± 1.11 mm errors using a single projection image. These promising results show REALMS's high potential to support realtime, accurate, and low-dose IGRT.

Local Metric Learning in 2D/3D Deformable Registration With Application in the Abdomen

“…The findings by Chou et al [2] indicate that REALMS performs reasonably well in lung. However, there are additional challenges posed by registration in abdomen.…”

“…As described in [2], REALMS first models the deformation with a shape space, in which each deformation is formed by a linear combination of basis deformations calculated through PCA analysis. For deformations due to respiration, we use a set of RCCT images { J τ |τ = 1, 2, …, 10} of the patient at planning time that records a cyclically varying target area.…”

“…For deformations due to respiration, we use a set of RCCT images { J τ |τ = 1, 2, …, 10} of the patient at planning time that records a cyclically varying target area. Chou et al [2] computed a Fréchet mean image J̅ as well as the diffeomorphic deformations ϕ from J̅ to each image J τ and performed a statistical analysis based on the 10 deformations. Here, we use an alternative approach, in which we pick a reference image I ref from the 10 RCCTs.…”

“…In this context, the patient’s treatment-time 3D deformations can be computed by performing image registration between the treatment-time on-board planar images (X-ray) and the treatment-planning 3D image (CT). Chou et al [2] recently introduced a real-time 2D/3D deformable registration method, called registration efficiency and accuracy through learning metric on shape (REALMS). At treatment-planning time REALMS learns a Riemannian metric that measures the distance between two projection images.…”

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

“…In our review we found that unsupervised representation learning techniques are a widely adopted technique to reduce the dimensionality of the parameter space while introducing implicit regularization by confining possible solutions to the principal modes of variation across population- or patient-level observations. We identified 12 studies that propose such techniques or use them as part of the registration pipeline ( Brost et al, 2012 ; Lin and Winey, 2012 ; Chou and Pizer, 2013 , 2014 ; Chou et al, 2013 ; Zhao et al, 2014 ; Baka et al, 2015 ; Pei et al, 2017 ; Chen et al, 2018 ; Zhang et al, 2018 ; Foote et al, 2019 ; Li et al, 2020 ; Zhang et al, 2020 ).…”