Constructing Benchmark Databases and Protocols for Medical Image Analysis: Diabetic Retinopathy

Abstract

We address the performance evaluation practices for developing medical image analysis methods, in particular, how to establish and share databases of medical images with verified ground truth and solid evaluation protocols. Such databases support the development of better algorithms, execution of profound method comparisons, and, consequently, technology transfer from research laboratories to clinical practice. For this purpose, we propose a framework consisting of reusable methods and tools for the laborious task of constructing a benchmark database. We provide a software tool for medical image annotation helping to collect class label, spatial span, and expert's confidence on lesions and a method to appropriately combine the manual segmentations from multiple experts. The tool and all necessary functionality for method evaluation are provided as public software packages. As a case study, we utilized the framework and tools to establish the DiaRetDB1 V2.1 database for benchmarking diabetic retinopathy detection algorithms. The database contains a set of retinal images, ground truth based on information from multiple experts, and a baseline algorithm for the detection of retinopathy lesions.

Refining Coarse Manual Segmentations with Stable Probability Regions

“…In preliminary tests on annotations in DiaRetDB1 diabetic retinopathy database (DiaRetDB1) [7], the performance of the proposed method was promising as can be seen in Figure 1. The initial results suggest that, for lesions for which color is a good discriminating feature such as haemorrhages and exudates, the proposed method is able to considerably improve the coarse manual data.…”

“…Top row from left to right: RGB image, coarse segmentation of exudates (white pixels mark the representative regions) and spatially accurate ground truth. Bottom row (refinement example by using DiaRetDB1 [7]): RGB image, coarse segmentation of exudates and the refined segmentation result.…”

Simulation and Synthesis in Medical Imaging

“…It is reassuring to see growing awareness on the importance of model verification and validation across engineering, [item 23) in the Appendix], [item 24) in the Appendix] medicine, [item 25) in the Appendix] [item 26) in the Appendix] and biology [item 27) in the Appendix]. While recent years have seen very positive initiatives in this arena, [item 28) in the Appendix]–[item 30) in the Appendix] our community of medical imaging and medical image computing will have to give even more consideration to these topics and develop and promote best practices in the assessment and benchmarking of simulation and synthesis methods.…”

Simulation and Synthesis in Medical Imaging

“…It is reassuring to see growing awareness on the importance of model verification and validation across engineering 28,29 , medicine 30,31 and biology 32 . While recent years have seen very positive initiatives in this arena, 33,34,35 our community of medical imaging and medical image computing will have to give even more consideration to these topics and develop and promote best practices in the assessment and benchmarking of simulation and synthesis methods.…”