Evaluation of multislice inputs to convolutional neural networks for medical image segmentation

Abstract

Purpose When using convolutional neural networks (CNNs) for segmentation of organs and lesions in medical images, the conventional approach is to work with inputs and outputs either as single slice [two‐dimensional (2D)] or whole volumes [three‐dimensional (3D)]. One common alternative, in this study denoted as pseudo‐3D, is to use a stack of adjacent slices as input and produce a prediction for at least the central slice. This approach gives the network the possibility to capture 3D spatial information, with only a minor additional computational cost. Methods In this study, we systematically evaluate the segmentation performance and computational costs of this pseudo‐3D approach as a function of the number of input slices, and compare the results to conventional end‐to‐end 2D and 3D CNNs, and to triplanar orthogonal 2D CNNs. The standard pseudo‐3D method regards the neighboring slices as multiple input image channels. We additionally design and evaluate a novel, simple approach where the input stack is a volumetric input that is repeatably convolved in 3D to obtain a 2D feature map. This 2D map is in turn fed into a standard 2D network. We conducted experiments using two different CNN backbone architectures and on eight diverse data sets covering different anatomical regions, imaging modalities, and segmentation tasks. Results We found that while both pseudo‐3D methods can process a large number of slices at once and still be computationally much more efficient than fully 3D CNNs, a significant improvement over a regular 2D CNN was only observed with two of the eight data sets. triplanar networks had the poorest performance of all the evaluated models. An analysis of the structural properties of the segmentation masks revealed no relations to the segmentation performance with respect to the number of input slices. A post hoc rank sum test which combined all metrics and data sets yielded that only our newly proposed pseudo‐3D method with an input size of 13 slices outperformed almost all methods. Conclusion In the general case, multislice inputs appear not to improve segmentation results over using 2D or 3D CNNs. For the particular case of 13 input slices, the proposed novel pseudo‐3D method does appear to have a slight advantage across all data sets compared to all other methods evaluated in this work.

Segmentation of Seismic Images

“…In our work we also applied a 2.5D (or pseudo-3D) approach, taking not a single vertical slice from the image cube, but stacking several slices (3 or 5), but it did not give any meaningful improvement compared to the single-slice input. This conclusion agrees with the one by [23].…”

Deep Diffusion MRI Registration (DDMReg): A Deep Learning Method for Diffusion MRI Registration

“…We demonstrated advantages by using pretrained backbone registration networks that enabled the usage of multiple registration subnetworks together. Pretrained backbones have been widely used in deep learning for medical image computing tasks and have been shown to improve both effectiveness and efficiency [52], [53], [54]. In our work, we leveraged pretrained backbones to reduce the computational burden, so that we were able to use a large number of U-nets (41 in our study) in the same network.…”

“…In the literature, backbone networks are often trained for a specific task and later used by fine tuning to another networks that may solve a different task. Pretrained backbones have been widely used in deep learning for medical image computing tasks and have been shown to improve both effectiveness and efficiency [59]- [61]. In our work, we pretrained registration networks to perform tract-specific registration, which are later used as backbones for registration of the entire dMRI data.…”

“…Leveraging pretrained network models as backbones has shown to be successful for improving both effectiveness and efficiency in medical image computing tasks [52], [53], [54]. In our work, given the large number of registration subnetworks and the limit of GPU memory, it is difficult to train all U-net models at the same time.…”

DefED-Net: Deformable Encoder-Decoder Network for Liver and Liver Tumor Segmentation

“…The H-DenseUNet finally achieves excellent liver and liver tumor segmentation. In addition, Vu et al [53] applied the overlay of adjacent slices as input to the central slice prediction, and then fed the obtained 2D feature maps into a standard 2D network for model training. Although these pseudo-3D approaches can segment objects from 3D volume data, they only obtain limited accuracy improvement due to the utilization of local temporal information.…”