A Spatiotemporal Agent for Robust Multimodal Registration

Abstract

Multimodal image registration is a crucial step for a variety of medical applications to provide complementary information from the combination of various data sources. Conventional image registration methods aim at finding a suited similarity metric as well as a descriptive image feature, which is quite challenging due to the high diversity of tissue appearance across modalities. In this paper, we present a novel approach to register images via an asynchronously trained reinforcement learning agent automatically. Within this approach, convolutional gated recurrent units (ConvGRU) is incorporated after stacked convolutional layers to extract both spatial and temporal features of the neighboring frames and implicitly learn the similarity metric. Moreover, we propose a customized reward function driven by fixed points error (FPE) to guide the agent to the correct registration direction. A Monte Carlo rollout strategy is also leveraged to perform a look-ahead inference to the elimination of jitter in the test stage. Evaluation is performed on paired CT and MR images from patients diagnosed as nasopharyngeal carcinoma. The results demonstrate that our method achieves state-of-the-art performance in medical image registration.

Stochastic Planner-Actor-Critic for Unsupervised Deformable Image Registration

“…But both of them are complex and computationally costly, and cannot deal with long step-wise registration. Inspired by the way that a human expert aligns two images by applying a sequence of local or global deformations, some RL-based image registration methods have been introduced in the past (Liao et al 2017;Ma et al 2017;Miao and Liao 2019;Sun et al 2018;Hu et al 2021;Luo et al 2020). However, most of them merely focus on global rigid transformation since it only includes rotation and translation and can be easily represented by a low-dimensional discrete parametric model.…”