Challenge on Ultrasound Beamforming with Deep Learning (CUBDL)

Deep Learning for Ultrasound Image Formation: CUBDL Evaluation Framework and Open Datasets

“…The challenge opened to participants on January 30, 2020 and closed to participants on the submission deadline of June 23, 2020. Although three optional tasks were conceived by the CUBDL organizers [20] and advertised on the CUBDL website [14], all participants submitted networks to be evaluated for Task 1, which was beamforming with deep learning after a single plane wave transmission, with two optional subtasks. Task 1a was explicitly focused on creating a high-quality image from a single plane wave to match a higher quality image created from multiple plane waves.…”

“…Participants were instructed to train networks using their preferred machine learning frameworks (e.g., TensorFlow, PyTorch) and submit final model files to the IEEE DataPort website [16]. The CUBDL organizers then downloaded the submitted models and launched a Python script to perform evaluation [15] on the internationally crowd-sourced database of test data [16].…”

“…Participants were instructed to train networks using their preferred machine learning frameworks (e.g., TensorFlow, PyTorch) and submit final model files to the IEEE DataPort website [16]. The CUBDL organizers then downloaded the submitted models and launched a Python script to perform evaluation [15] on the internationally crowd-sourced database of test data [16]. Evaluation metrics advertised since the launch of the CUBDL website [14] were pre-selected by the CUBDL organizers based on literature from multiple groups reporting beamforming with deep learning (e.g., [8], [9], [11], [23]) and based on common computer vision literature containing assessments of network complexity (e.g., [24]- [26]).…”

“…One outcome of the Challenge on Ultrasound Beamforming with Deep Learning (CUBDL), offered as a component of the 2020 IEEE International Ultrasonics Symposium, was the development of an open evaluation framework that may be used as a standard for deep learning image formation methods in ultrasound imaging. The overarching goal of this challenge was to explore the potential of deep learning to improve ultrasound image quality and to balance these improvements with implementation practicality (i.e., reasonable display frame rates) [14]. Comparisons were completed on the same datasets and with the same evaluation methods and framework.…”

“…Components of this evaluation framework were publicly available while CUBDL was in progress, and the framework is now merged with the CUBDL test data and the additional datasets described herein. This framework merger and associated datasets are publicly available at [15] and [16], respectively (visit https://cubdl.jhu.edu for more details). The summary within this paper and the accessibility of our data and code are provided in efforts to enable future benchmarking and evaluation of new methods.…”