CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting

Rajpurkar, Pranav; Joshi, Anirudh; Pareek, Anuj; Chen, Phil; Kiani, Amirhossein; Irvin, Jeremy; Ng, Andrew Y.; Lungren, Matthew P.

doi:10.48550/arxiv.2002.11379

Cited by 11 publications

(12 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This variation, also known as dataset shift, can occur due to differences in hospital procedures, equipment manufacturers, image acquisition parameters, disease manifestations, patient populations, etc. Due to dataset shift, models trained using data from one hospital may perform poorly on data from another hospital [55]. We note here that this inability to generalize to datasets from an unseen origin is different from the problem of overfitting, where the model shows poor performance even on test sets from the same origin.…”

Section: Generalization Of Models To Unseen Datasetsmentioning

confidence: 83%

Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practice

Kulkarni¹,

Gawali²,

Kharat³

2021

Preprint

View full text Add to dashboard Cite

The use of machine learning to develop intelligent software tools for interpretation of radiology images has gained widespread attention in recent years. The development, deployment, and eventual adoption of these models in clinical practice, however, remains fraught with challenges. In this paper, we propose a list of key considerations that machine learning researchers must recognize and address to make their models accurate, robust, and usable in practice. Namely, we discuss: insufficient training data, decentralized datasets, high cost of annotations, ambiguous ground truth, imbalance in class representation, asymmetric misclassification costs, relevant performance metrics, generalization of models to unseen datasets, model decay, adversarial attacks, explainability, fairness and bias, and clinical validation. We describe each consideration and identify techniques to address it. Although these techniques have been discussed in prior research literature, by freshly examining them in the context of medical imaging and compiling them in the form of a laundry list, we hope to make them more accessible to researchers, software developers, radiologists, and other stakeholders.

show abstract

Section: Generalization Of Models To Unseen Datasetsmentioning

confidence: 83%

Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practice

Kulkarni¹,

Gawali²,

Kharat³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Computer-aided diagnosis (CAD) systems for chest radiographs (also referred to as Chest X-ray or CXR) have recently achieved great success thanks to the availability of large labeled datasets and the recent advances of high-performance supervised learning algorithms [1][2][3][4][5][6][7] . Leveraging deep convolutional neural networks (CNN) 8 , these systems can reach the expert-level performance in classifying common lung diseases and related findings.…”

Section: Background and Summarymentioning

confidence: 99%

VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations

Nguyen¹,

Lam²,

Le³

et al. 2020

Preprint

View full text Add to dashboard Cite

Most of the existing chest X-ray datasets include labels from a list of findings without specifying their locations on the radiographs. This limits the development of machine learning algorithms for the detection and localization of chest abnormalities. In this work, we describe a dataset of more than 100,000 chest X-ray scans that were retrospectively collected from two major hospitals in Vietnam. Out of this raw data, we release 18,000 images that were manually annotated by a total of 17 experienced radiologists with 22 local labels of rectangles surrounding abnormalities and 6 global labels of suspected diseases. The released dataset is divided into a training set of 15,000 and a test set of 3,000. Each scan in the training set was independently labeled by 3 radiologists, while each scan in the test set was labeled by the consensus of 5 radiologists. We designed and built a labeling platform for DICOM images to facilitate these annotation procedures. All images are made publicly available in DICOM format in company with the labels of the training set. The labels of the test set are hidden at the time of writing this paper as they will be used for benchmarking machine learning algorithms on an open platform.

show abstract

“…It is hoped that a Computer-Aided Diagnosis (CAD) system using artificial intelligence (AI) can effectively assist radiologists and help mitigate the misdiagnosis rate on CXRs. Leveraging recent advances in deep learning [1], such systems have achieved a great success in detecting a wide range of abnormalities on CXRs [2][3][4][5][6][7][8][9][10][11][12]. Most of the existing systems are supervised-learning models that were trained and validated on different parts of a dataset that was collected and labeled in a retrospective fashion.…”

Section: Introductionmentioning

confidence: 99%

A clinical validation of VinDr-CXR, an AI system for detecting abnormal chest radiographs

Nguyen,

Nguyen

et al. 2021

Preprint

View full text Add to dashboard Cite

Computer-Aided Diagnosis (CAD) systems for chest radiographs using artificial intelligence (AI) have recently shown a great potential as a second opinion for radiologists. The performances of such systems, however, were mostly evaluated on a fixed dataset in a retrospective manner and, thus, far from the real performances in clinical practice. In this work, we demonstrate a mechanism for validating an AI-based system for detecting abnormalities on X-ray scans, VinDr-CXR, at the Phu Tho General Hospital-a provincial hospital in the North of Vietnam. The AI system was directly integrated into the Picture Archiving and Communication System (PACS) of the hospital after being trained on a fixed annotated dataset from other sources. The performance of the system was prospectively measured by matching and comparing the AI results with the radiology reports of 6,285 chest X-ray examinations extracted from the Hospital Information System (HIS) over the last two months of 2020. The normal/abnormal status of a radiology report was determined by a set of rules and served as the ground truth. Our system achieves an F1 score-the harmonic average of the recall and the precision-of 0.653 (95% CI 0.635, 0.671) for detecting any abnormalities on chest X-rays. Despite a significant drop from the in-lab performance, this result establishes a high level of confidence in applying such a system in real-life situations.

show abstract

CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting

Cited by 11 publications

References 25 publications

Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practice

Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practice

VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations

A clinical validation of VinDr-CXR, an AI system for detecting abnormal chest radiographs

Contact Info

Product

Resources

About