Dilated FCN for Multi-Agent 2D/3D Medical Image Registration

Abstract

2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.

Reinforcement learning in medical image analysis: Concepts, applications, challenges, and future directions

“…This chapter of the book can serve as an effective tutorial because they discussed the state, action space, and reward design in detail. In, 73 they attempted to use a multi‐agent attention mechanism to solve the 2D‐3D registration for images with severe artifacts. Instead of choosing the commonly used CNNs, this work adopted dilated fully‐CNN (FCN) as the backbone of the agent.…”

Artificial Intelligence for Emerging Technology in Surgery: Systematic Review and Validation

“…In image registration, the task is the spatial alignment between two medical images, volumes or modalities. 3D-CNNs are used for registering pulmonary CT images [40], anatomical labels [41], 3D volume to 2D X-ray images [42], and predicting deformation from image appearance [43]. In intra-operative guidance, AI has been utilized to provide enhanced visualization and localization in surgery.…”

Modeling the Probabilistic Distribution of Unlabeled Data for One-shot Medical Image Segmentation

“…Image registration (Miao et al 2018;Zitova and Flusser 2003) aims to align an image to a template one, called atlas, by learning shape deformations between them. Most existing registration-based segmentation methods (Wang et al 2020;Xu and Niethammer 2019;Zhu et al 2020) only consider the structure differences between two images.…”