The Challenge Dataset – simple evaluation for safe, transparent healthcare AI deployment

Abstract

In this paper, we demonstrate the use of a "Challenge Dataset": a small, site-specific, manually curated dataset - enriched with uncommon, risk-exposing, and clinically important edge cases - that can facilitate pre-deployment evaluation and identification of clinically relevant AI performance deficits. The five major steps of the Challenge Dataset process are described in detail, including defining use cases, edge case selection, dataset size determination, dataset compilation, and model evaluation. Evaluating performance of four chest X-ray classifiers (one third-party developer model and three models trained on open-source datasets) on a small, manually curated dataset (410 images), we observe a generalization gap of 20.7% (13.5% - 29.1%) for sensitivity and 10.5% (4.3% - 18.3%) for specificity compared to developer-reported values. Performance decreases further when evaluated against edge cases (critical findings: 43.4% [27.4% - 59.8%]; unusual findings: 45.9% [23.1% - 68.7%]; solitary findings 45.9% [23.1% - 68.7%]). Expert manual audit revealed examples of critical model failure (e.g., missed pneumomediastinum) with potential for patient harm. As a measure of effort, we find that the minimum required number of Challenge Dataset cases is about 1% of the annual total for our site (approximately 400 of 40,000). Overall, we find that the Challenge Dataset process provides a method for local pre-deployment evaluation of medical imaging AI models, allowing imaging providers to identify both deficits in model generalizability and specific points of failure prior to clinical deployment.

Why do errors arise in artificial intelligence diagnostic tools in histopathology and how can we minimize them?

“…Typically in the development of AI diagnostic tools, the model is first trained on one dataset, referred to as the training dataset. The model is then subsequently tested on a second testing dataset, comprising data unseen during the training stage for external validation 14 . It is well understood that these two developmental datasets should be different in order to assess the algorithm's performance on unseen data, and ideally some of the test dataset should be from external sources.…”

Haemosuccus pancreaticus and seven episodes of recurrent unlocalised upper gastrointestinal bleeding

“…An edge case is a specific scenario that occurs only at an extreme (usually the maximum or minimum) of operating parameter—the computer science equivalent of a ‘rare presentation’. By assessing the performance or understanding the output of the method or algorithm to ‘rare presentations’, developers begin to understand the limitations of the aforementioned methods/algorithms and can begin to improve them 1. Thus, while upper gastrointestinal (GI) bleeding is common (annual incidence of 1 in 1000),2 exploring the diagnostic odyssey of a case study of haemosuccus pancreaticus (a rare cause of bleeding, <1% of upper GI bleeds)3 reveals both the strengths and weakness of current diagnostic approaches to GI bleeding management 4–6.…”