Deep action learning enables robust 3D segmentation of body organs in various CT and MRI images

Abstract

In this study, we propose a novel point cloud based 3D registration and segmentation framework using reinforcement learning. An artificial agent, implemented as a distinct actor based on value networks, is trained to predict the optimal piece-wise linear transformation of a point cloud for the joint tasks of registration and segmentation. The actor network estimates a set of plausible actions and the value network aims to select the optimal action for the current observation. Point-wise features that comprise spatial positions (and surface normal vectors in the case of structured meshes), and their corresponding image features, are used to encode the observation and represent the underlying 3D volume. The actor and value networks are applied iteratively to estimate a sequence of transformations that enable accurate delineation of object boundaries. The proposed approach was extensively evaluated in both segmentation and registration tasks using a variety of challenging clinical datasets. Our method has fewer trainable parameters and lower computational complexity compared to the 3D U-Net, and it is independent of the volume resolution. We show that the proposed method is applicable to mono- and multi-modal segmentation tasks, achieving significant improvements over the state-of-the-art for the latter. The flexibility of the proposed framework is further demonstrated for a multi-modal registration application. As we learn to predict actions rather than a target, the proposed method is more robust compared to the 3D U-Net when dealing with previously unseen datasets, acquired using different protocols or modalities. As a result, the proposed method provides a promising multi-purpose segmentation and registration framework, particular in the context of image-guided interventions.

Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future

“…Further recent hybrids in image segmentation build on deep active contours [88], differentiable meshing [9], and deep action learning [89] for active shape models.…”

BOSS: Bones, Organs and Skin Shape Model

“…It has been shown, that coupling a statistical knee model with a segmentation neural network a more precise segmentation of the knee based on magnetic resonance images can be achieved [8]. Shape models can be beneficial for the segmentation of 3D volumetric images such as Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) [9][10][11], as well as for 2D projection images such as X-rays [12]. This is because the prior knowledge of the body shape can be used as a regularizer.…”

Deep Learning-Based 3D Instance and Semantic Segmentation: A Review

“…In computer vision and graphics, segmenting 3D scenes is a dangerous and bleak issue. The purpose of 3D segmentation is to create computational algorithms that predict the fine-grained labels of objects in a 3D environment for a range of applications, such as medical image analysis, autonomous driving, industrial control, mobile, augmented and virtual reality and robotics [1,2]. The 3D segmentation can be divided into multi-sorts, instance and semantic segmentation.…”