Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning

Sun, Shanhui; Hu, Jing; Yao, Mingqing; Hu, Jinrong; Yang, Xiaodong; Song, Qi

doi:10.1007/978-3-030-20890-5_33

Cited by 26 publications

(35 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They claimed that the learnt similarity metric outperformed MI and its variant MIND [63]. [92] Spine, Cardiac 3D-3D CT-CBCT Rigid [101] Chest, Abdomen 2D-2D CT-Depth Image Rigid [108] Spine 3D-2D 3DCT-Xray Rigid [183] Spine 3D-2D 3DCT-Xray Rigid [144] Nasopharyngeal 2D-2D MR-CT Rigid with scaling…”

Section: Overview Of Workmentioning

confidence: 99%

Deep learning in medical image registration: a review

et al. 2020

View full text Add to dashboard Cite

This paper presents a review of deep learning (DL) based medical image registration methods. We summarized the latest developments and applications of DL-based registration methods in the medical field. These methods were classified into seven categories according to their methods, functions and popularity. A detailed review of each category was presented, highlighting important contributions and identifying specific challenges. A short assessment was presented following the detailed review of each category to summarize its achievements and future potentials. We provided a comprehensive comparison among DL-based methods for lung and brain registration using benchmark datasets. Lastly, we analyzed the statistics of all the cited works from various aspects, revealing the popularity and future trend of DL-based medical image registration. 1 arXiv:1912.12318v1 [eess.IV] 27 Dec 2019 1. Summarize the latest developments in DL-based medical image registration. 2. Highlight contributions, identify challenges and outline future trends. 3. Provide detailed statistics on recent publications from different perspectives. 2 Deep Learning 2 Convolutional Neural NetworkConvolutional neural network (CNN) is a class of deep neural networks with regularized multilayer perceptron. CNN uses convolution operation in place of general matrix multiplication in simple neural networks. The convolutional filters and operations in CNN make it suitable for visual imagery signal processing. Because of its excellent feature extraction ability, CNN is one of the most successful models for image analysis. Since the breakthrough of AlexNet [79], many variants of CNN have been proposed and have achieved the-state-of-art performances in various image processing tasks. A typical CNN usually consists of multiple convolutional layers, max pooling layers, batch normalization layers, dropout layers, a sigmoid or softmax layer. In each convolutional layer, multiple channels of feature maps were extracted by sliding trainable convolutional kernels across the input feature maps. Hierarchical features with high-level abstraction are extracted using multiple convolutional layers. These feature maps usually go through multiple fully connected layer before reaching the final decision layer. Max pooling layers are often used to reduce the image sizes and to promote spatial invariance of the network. Batch normalization is used to reduce internal covariate shift among the training samples. Weight regularization and dropout layers are used to alleviate data overfitting. The loss function is defined as the difference between the predicted and the target output. CNN is usually trained by minimizing the loss via gradient back propagation using optimization methods. Many different types of network architectures have been proposed to improve the performance of CNN [93]. U-Net proposed by Ronneberger et al. is among one of the most used network architectures [120]. U-Net was originally used to perform neuronal structures segmentation. U-Net adopts symmetrical contractiv...

show abstract

Section: Overview Of Workmentioning

confidence: 99%

Deep learning in medical image registration: a review

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Worse still, both methods ignored the temporal dependencies that may accelerate training. In light of these limitations, our previous work [16] presented an end-to-end RL model with long-term recurrent network architecture (LSTM) [13] and a reward function driven by landmark error (LME), which needed no extra storage and explored transformation parameter spaces freely. However, it still suffers from certain limitations.…”

Section: Intensity-based Methods Operate Directly On Image Intensitiesmentioning

confidence: 99%

“…The ConvGRU is used for obtaining the historical characteristics, so as to preserve spatial connectivities in the frames. -We propose a novel reward function driven by fixed-points error (FPE), which is similar to the landmark error proposed in [16] but more computationally efficient. Using the FPE-based reward function facilitates the extension of 3D image registration.…”

Section: Intensity-based Methods Operate Directly On Image Intensitiesmentioning

confidence: 99%

“…This approach is free to search the parameter space during training but suffers timeconsuming. Our previous work [16] introduced a parallel CNN+LSTM architecture and a landmark error (LME) based reward function, which overcame the problem of Ma's and improved the robustness of multimodal image registration. However, the conventional LSTM does not have the flexibility in capturing spatial contextual information between the consecutive frames, and the computation of LME is rather complicated and hence aggravates the burden of RL training.…”

Section: Related Work a Image Registrationmentioning

confidence: 99%

“…However, in this way, the unit discrepancy between scaling and the other two parameters in the transformation matrix might cause erratic training. Although the landmark error (LME) reward function helps convergence [16], searching the key-points using the scale-invariant feature transform (SIFT) [22] is burdensome and make [16] hard to extend to 3D registration. In rigid registration, the transformation T is applied to the entire moving image, including the points in the background.…”

Section: A Environmentmentioning

confidence: 99%

See 2 more Smart Citations

A Spatiotemporal Agent for Robust Multimodal Registration

Luo

Wang

et al. 2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

Multimodal image registration is a crucial step for a variety of medical applications to provide complementary information from the combination of various data sources. Conventional image registration methods aim at finding a suited similarity metric as well as a descriptive image feature, which is quite challenging due to the high diversity of tissue appearance across modalities. In this paper, we present a novel approach to register images via an asynchronously trained reinforcement learning agent automatically. Within this approach, convolutional gated recurrent units (ConvGRU) is incorporated after stacked convolutional layers to extract both spatial and temporal features of the neighboring frames and implicitly learn the similarity metric. Moreover, we propose a customized reward function driven by fixed points error (FPE) to guide the agent to the correct registration direction. A Monte Carlo rollout strategy is also leveraged to perform a look-ahead inference to the elimination of jitter in the test stage. Evaluation is performed on paired CT and MR images from patients diagnosed as nasopharyngeal carcinoma. The results demonstrate that our method achieves state-of-the-art performance in medical image registration.

show abstract

Reinforcement learning in medical image analysis: Concepts, applications, challenges, and future directions

Zhang

Luke

et al. 2023

J Applied Clin Med Phys

View full text Add to dashboard Cite

Motivation Medical image analysis involves a series of tasks used to assist physicians in qualitative and quantitative analyses of lesions or anatomical structures which can significantly improve the accuracy and reliability of medical diagnoses and prognoses. Traditionally, these tedious tasks were finished by experienced physicians or medical physicists and were marred with two major problems, low efficiency and bias. In the past decade, many machine learning methods have been applied to accelerate and automate the image analysis process. Compared to the enormous deployments of supervised and unsupervised learning models, attempts to use reinforcement learning in medical image analysis are still scarce. We hope that this review article could serve as the stepping stone for related research in the future. Significance We found that although reinforcement learning has gradually gained momentum in recent years, many researchers in the medical analysis field still find it hard to understand and deploy in clinical settings. One possible cause is a lack of well‐organized review articles intended for readers without professional computer science backgrounds. Rather than to provide a comprehensive list of all reinforcement learning models applied in medical image analysis, the aim of this review is to help the readers formulate and solve their medical image analysis research through the lens of reinforcement learning. Approach & Results We selected published articles from Google Scholar and PubMed. Considering the scarcity of related articles, we also included some outstanding newest preprints. The papers were carefully reviewed and categorized according to the type of image analysis task. In this article, we first reviewed the basic concepts and popular models of reinforcement learning. Then, we explored the applications of reinforcement learning models in medical image analysis. Finally, we concluded the article by discussing the reviewed reinforcement learning approaches’ limitations and possible future improvements.

show abstract

Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning

Cited by 26 publications

References 27 publications

Deep learning in medical image registration: a review

Deep learning in medical image registration: a review

A Spatiotemporal Agent for Robust Multimodal Registration

Reinforcement learning in medical image analysis: Concepts, applications, challenges, and future directions

Contact Info

Product

Resources

About