Recurrent Saliency Transformation Network: Incorporating Multi-stage Visual Cues for Small Organ Segmentation

Abstract

We aim at segmenting small organs (e.g., the pancreas) from abdominal CT scans. As the target often occupies a relatively small region in the input image, deep neural networks can be easily confused by the complex and variable background. To alleviate this, researchers proposed a coarse-to-fine approach [46], which used prediction from the first (coarse) stage to indicate a smaller input region for the second (fine) stage. Despite its effectiveness, this algorithm dealt with two stages individually, which lacked optimizing a global energy function, and limited its ability to incorporate multi-stage visual cues. Missing contextual information led to unsatisfying convergence in iterations, and that the fine stage sometimes produced even lower segmentation accuracy than the coarse stage.This paper presents a Recurrent Saliency Transformation Network. The key innovation is a saliency transformation module, which repeatedly converts the segmentation probability map from the previous iteration as spatial weights and applies these weights to the current iteration. This brings us two-fold benefits. In training, it allows joint optimization over the deep networks dealing with different input scales. In testing, it propagates multi-stage visual information throughout iterations to improve segmentation accuracy. Experiments in the NIH pancreas segmentation dataset demonstrate the state-of-the-art accuracy, which outperforms the previous best by an average of over 2%. Much higher accuracies are also reported on several small organs in a larger dataset collected by ourselves. In addition, our approach enjoys better convergence properties, making it more efficient and reliable in practice.

Globally Guided Progressive Fusion Network for 3D Pancreas Segmentation

“…Comparisons with State-of-the-Art Segmentation Algorithms Comparisons against state-of-the-art volumetric segmentation algorithms are reported in Table 1. According to output type, we classify them into three categories: 3D models which predict 3D probability maps directly (such as UNet-Patch [8] and UNet-Full [9]), 2D models which produce 2D segmentation results over slices in the axial view (such as FCN8s [5]), Pseudo-3D (P3D) models which fuse 2D segmentation results for axial, sagittal and coronal views (such as RSTN [11]). Our globally guided progressive fusion network (GGPFN) can be easily integrated into the 2D and P3D segmentation frameworks.…”

“…Precision-recall curves and F-scores of 3D UNet-Patch [8], 3D UNet-Full [9], P3D FCN8s [12], P3D RSTN [11] and our final model P3D GGPFN are presented in Fig. 4.…”

“…Due to the huge memory overhead of 3D convolutions, the input is either decomposed into overlapping 3D patches [2], which ignores the global knowledge, or resized to a volume with a poor resolution [9], which likely gives rise to missed detections. Coarse-to-fine segmentation is a popular and effective choice for improving the accuracy [8,11,10]. However, it is severely dependent on the performance of its coarse segmentation model.…”

“…Omission of regions of interest (ROIs) or inaccurate size of ROIs in the coarse segmentation often lead to irreparable loss. Most of these volumetric segmentation methods have been applied in pancreas segmentation such as [13,11,10].…”

FusionNet: Incorporating Shape and Texture for Abnormality Detection in 3D Abdominal CT Scans

“…Since the focus in this paper is how to combine g(X) and X in f (·), and also the two stages are executed separately, so the form of g(·) is out of range of this study and will be investigated in the future. In this paper we choose a recent stat-of-the-art segmentation framework [11] for g. Since g(·) is a 2D-based method so we need to concatenate the output of different slices to reconstruct the 3D volume like in [10]. We train the segmentation algorithm on X 1 and test it on X 2 .…”