Deep learning for automatic brain tumour segmentation on MRI: evaluation of recommended reporting criteria via a reproduction and replication study

Abstract

ObjectivesTo determine the reproducibility and replicability of studies that develop and validate segmentation methods for brain tumours on MRI and that follow established reproducibility criteria; and to evaluate whether the reporting guidelines are sufficient.MethodsTwo eligible validation studies of distinct deep learning (DL) methods were identified. We implemented the methods using published information and retraced the reported validation steps. We evaluated to what extent the description of the methods enabled reproduction of the results. We further attempted to replicate reported findings on a clinical set of images acquired at our institute consisting of high-grade and low-grade glioma (HGG, LGG), and meningioma (MNG) cases.ResultsWe successfully reproduced one of the two tumour segmentation methods. Insufficient description of the preprocessing pipeline and our inability to replicate the pipeline resulted in failure to reproduce the second method. The replication of the first method showed promising results in terms of Dice similarity coefficient (DSC) and sensitivity (Sen) on HGG cases (DSC=0.77, Sen=0.88) and LGG cases (DSC=0.73, Sen=0.83), however, poorer performance was observed for MNG cases (DSC=0.61, Sen=0.71). Preprocessing errors were identified that contributed to low quantitative scores in some cases.ConclusionsEstablished reproducibility criteria do not sufficiently emphasise description of the preprocessing pipeline. Discrepancies in preprocessing as a result of insufficient reporting are likely to influence segmentation outcomes and hinder clinical utilisation. A detailed description of the whole processing chain, including preprocessing, is thus necessary to obtain stronger evidence of the generalisability of DL-based brain tumour segmentation methods and to facilitate translation of the methods into clinical practice.

CrossTransUnet: A New Computationally Inexpensive Tumor Segmentation Model for Brain MRI

“…This makes it difficult to evaluate different networks accurately and fairly, particularly when the necessary information to replicate the networks and results is not reported. Gryska et al [45] even caution about the difficulties in replicating segmentation models and argue that established reproducibility criteria do not adequately emphasize the importance of describing the preprocessing chain. They conclude that a detailed description of the entire preprocessing chain is necessary to establish solid evidence of the generalizability of segmentation methods.…”

“…In different approaches, Gryska et al [45] replicate two segmentation models (3D dual-path CNN and 2D single-path CNN), seeking to determine the reproducibility and replicability of the studies. The study found that one of the two methods was successfully reproduced, but the second method could not be reproduced due to insufficient description of the preprocessing pipeline.…”

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

“…Specifying preprocessing steps is also important for ensuring the reproducibility of the results. Choices in preprocessing technique can substantially affect the properties of the resulting model, including accuracy and interpretability ( 135 , 136 ).…”