Why rankings of biomedical image analysis competitions should be interpreted with care

Abstract

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Artificial intelligence in digital pathology: a roadmap to routine use in clinical practice

“…The fact that AI researchers are now beginning to focus on (a) providing confidence estimates with their predictions/results and (b) localising pathology‐related features should help with allaying concerns about interpretability and building trust. Besides, there is also need for regulatory processes to learn from the experience of medical imaging communities in evaluating the performance of algorithms for various challenge contests . The future educational needs of the pathology community will change, bringing a need for at least a basic working knowledge of how such algorithms function, with some pathologists taking on a more advanced ‘computational pathologist’ role.…”

The 2016 QSM Challenge: Lessons learned and considerations for a future challenge design

“…The majority of biomedical imaging challenges (such as from MICCAI) conducted in the past years targeted segmentation and classification. 47 However, owing to the underlying physical problem without a ground truth, comparing the performance of QSM reconstruction algorithms is more complex than what can be reflected by a single numeric error measure, and the question on how to measure the quality of the reconstructions remains open. Therefore, for future QSM Reconstruction challenges it could be advisable to consider the overall performance instead of relying on a single metric.…”

“…To avoid controversy, categories should be announced beforehand, and the scoring source code should be made available with the release of the ground truth dataset. 47 In the case of working with in vivo data, thorough background removal steps should be carried out to provide clean local phases. This may be performed by including an additional background field removal step, such as PDF or V-SHARP.…”

Breaking Medical Data Sharing Boundaries by Employing Artificial Radiographs

“…The application of Artificial Intelligence (AI) in medicine promises to personalize diagnosis, decision management and therapy based on the combination of patient information with knowledge of thousands of experts and the outcome of billions of patient [1][2][3][4]. In recent years, a lot of scientific effort has focused on applications of AI in medicine with a particularly strong focus on radiology [5][6][7][8][9][10]. Whenever there has been progress towards this vision of an omniscient radiological AI, it has mostly been anticipated by corresponding technical advances in the field of Computer Vision on natural images.…”