Robust Non-rigid Registration Through Agent-Based Action Learning

Abstract

Robust image registration in medical imaging is essential for comparison or fusion of images, acquired from various perspectives, modalities or at different times. Typically, an objective function needs to be minimized assuming specific a priori deformation models and predefined or learned similarity measures. However, these approaches have difficulties to cope with large deformations or a large variability in appearance. Using modern deep learning (DL) methods with automated feature design, these limitations could be resolved by learning the intrinsic mapping solely from experience. We investigate in this paper how DL could help organ-specific (ROI-specific) deformable registration, to solve motion compensation or atlas-based segmentation problems for instance in prostate diagnosis. An artificial agent is trained to solve the task of non-rigid registration by exploring the parametric space of a statistical deformation model built from training data. Since it is difficult to extract trustworthy ground-truth deformation fields, we present a training scheme with a large number of synthetically deformed image pairs requiring only a small number of real inter-subject pairs. Our approach was tested on inter-subject registration of prostate MR data and reached a median DICE score of .88 in 2-D and .76 in 3-D, therefore showing improved results compared to state-of-the-art registration algorithms.

Multimodality image registration in the head‐and‐neck using a deep learning‐derived synthetic CT as a bridge

“…In synergy with using a CT synth bridge, improvements in mono‐modality registration would also lead to better multimodal registration in the head‐and‐neck. Currently, research using neural networks offers some exciting new avenues in this regard, including completely learning‐based unsupervised DVF generation . However, the performance of these methods depends on the availability and quality of training sets, which are particularly challenging for multimodel registration.…”

Deep learning in medical image registration: a survey

“…As opposed to the above rigid registration based works, Krebs et al [55] used a reinforcement learning based approach to perform the deformable registration of 2D and 3D prostate MR volumes. They used a low resolution deformation model for the registration and fuzzy action control to influence the stochastic action selection.…”

“…It should be noted that there are some unique challenges associated with deep learning based medical image registration: including the dimensionality of the action space in the deformable registration case. However, we believe that such limitations are surmountable because there is already one proposed method that uses reinforcement learning based registration with a deformable transformation model [55].…”

Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning

“…An artificial agent explores the space of spatial transformation and determines the next best transformation parameter at each time step. Unlike other similar image registration methods that either use pretrained neural network or learn the greedy exploration process in a supervised manner [11,12], our method explores the searching space freely by using a multithread actor-critic scheme (A3C). Our major contributions are:…”