Marginal Space Deep Learning: Efficient Architecture for Volumetric Image Parsing

Abstract

Robust and fast solutions for anatomical object detection and segmentation support the entire clinical workflow from diagnosis, patient stratification, therapy planning, intervention and follow-up. Current state-of-the-art techniques for parsing volumetric medical image data are typically based on machine learning methods that exploit large annotated image databases. Two main challenges need to be addressed, these are the efficiency in scanning high-dimensional parametric spaces and the need for representative image features which require significant efforts of manual engineering. We propose a pipeline for object detection and segmentation in the context of volumetric image parsing, solving a two-step learning problem: anatomical pose estimation and boundary delineation. For this task we introduce Marginal Space Deep Learning (MSDL), a novel framework exploiting both the strengths of efficient object parametrization in hierarchical marginal spaces and the automated feature design of Deep Learning (DL) network architectures. In the 3D context, the application of deep learning systems is limited by the very high complexity of the parametrization. More specifically 9 parameters are necessary to describe a restricted affine transformation in 3D, resulting in a prohibitive amount of billions of scanning hypotheses. The mechanism of marginal space learning provides excellent run-time performance by learning classifiers in clustered, high-probability regions in spaces of gradually increasing dimensionality. To further increase computational efficiency and robustness, in our system we learn sparse adaptive data sampling patterns that automatically capture the structure of the input. Given the object localization, we propose a DL-based active shape model to estimate the non-rigid object boundary. Experimental results are presented on the aortic valve in ultrasound using an extensive dataset of 2891 volumes from 869 patients, showing significant improvements of up to 45.2% over the state-of-the-art. To our knowledge, this is the first successful demonstration of the DL potential to detection and segmentation in full 3D data with parametrized representations.

A survey on deep learning in medical image analysis

“…Zheng et al (2015) reduced this complexity by decomposing 3D convolution as three one-dimensional convolutions for carotid artery bifurcation detection in CT data. Ghesu et al (2016b) proposed a sparse adaptive deep neural network powered by marginal space learning in order to deal with data complexity in the detection of the aortic valve in 3D transesophageal echocardiogram.…”

Segmentation of tongue shapes during vowel production in magnetic resonance images based on statistical modelling

“…Statistical models, such as ASMs, have been shown to achieve good segmentation results when combined with robust initialization approaches, which can be achieved, for example, by using machine learning algorithms. Specifically, Marginal Space Deep Learning (MSDL) is an emerging technique used to align the mean shape of a model based on deep learning neural networks for object localization, and sequential estimation of the pose and scale parameters to be used in the ASM fitting 23 . Nevertheless, the proposed segmentation approach has proven to have potential as a tool for speech shape analysis; namely, for the evaluation of the shape of the tongue and movement patterns during speech production, as well as to improve the knowledge about the physiology of this organ that still needs to be further explored.…”

Machine learning–based 3‐D geometry reconstruction and modeling of aortic valve deformation using 3‐D computed tomography images

“…In this study, the type of features were pre-determined, i.e., handcrafted. Recently, deep learning methods have been successfully used for image segmentation (39–42), in which features are learned automatically. We will combine deep learning and the D-SSM in future work.…”