Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest X-ray dataset

Filice, Ross W.; Stein, Anouk; Wu, Carol C.; Arteaga, Veronica; Borstelmann, Stephen M.; Gaddikeri, Ramya S.; Galperin-Aizenberg, Maya; Gill, Ritu R.; Godoy, Myrna C.; Hobbs, Stephen B.; Jeudy, Jean; Lakhani, Paras; Laroia, Archana; Nayak, Sundeep M.; Parekh, Maansi; Prasanna, Prasanth; Shah, Palmi; Vummidi, Dharshan; Yaddanapudi, Kavitha; Shih, George

doi:10.1007/s10278-019-00299-9

Cited by 38 publications

(24 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These X-ray images have text-mined labels with 14 common thorax disease extracted from relevant radiological reports 21 . These labels were obtained through natural language processing and are inherently inaccurate 22 . For example, there are 5302 X-ray images labeled as pneumothorax, among which are a mixture of images with and without pneumothorax.…”

Section: Methodsmentioning

confidence: 99%

Detection of the location of pneumothorax in chest X-rays using small artificial neural networks and a simple training process

Cho

Kim

Lim

et al. 2021

Sci Rep

View full text Add to dashboard Cite

The purpose of this study was to evaluate the diagnostic performance achieved by using fully-connected small artificial neural networks (ANNs) and a simple training process, the Kim-Monte Carlo algorithm, to detect the location of pneumothorax in chest X-rays. A total of 1,000 chest X-ray images with pneumothorax were taken randomly from NIH (the National Institutes of Health) public image database and used as the training and test sets. Each X-ray image with pneumothorax was divided into 49 boxes for pneumothorax localization. For each of the boxes in the chest X-ray images contained in the test set, the area under the receiver operating characteristic (ROC) curve (AUC) was 0.882, and the sensitivity and specificity were 80.6% and 83.0%, respectively. In addition, a common currently used deep-learning method for image recognition, the convolution neural network (CNN), was also applied to the same dataset for comparison purposes. The performance of the fully-connected small ANN was better than that of the CNN. Regarding the diagnostic performances of the CNN with different activation functions, the CNN with a sigmoid activation function for fully-connected hidden nodes was better than the CNN with the rectified linear unit (RELU) activation function. This study showed that our approach can accurately detect the location of pneumothorax in chest X-rays, significantly reduce the time delay incurred when diagnosing urgent diseases such as pneumothorax, and increase the effectiveness of clinical practice and patient care.

show abstract

Section: Methodsmentioning

confidence: 99%

Detection of the location of pneumothorax in chest X-rays using small artificial neural networks and a simple training process

Cho

Kim

Lim

et al. 2021

Sci Rep

View full text Add to dashboard Cite

show abstract

“…However, large volumes of chest radiographs (CXR) in routine clinical environment may yield longer turnaround times for radiology reporting which can delay urgent treatment; this issue as well as latent critical findings can be potentially addressed by the use of artificial intelligence (AI)-assisted reporting or an AI-based image triage. Several AI algorithms, trained on publicly available datasets, have demonstrated potential to detect PTX in CXRs with diagnostic accuracies that have been quantified by area under receiver operating characteristics (AUROCs) of up to 0.937 [8][9][10][11][12][13]. In studies evaluating these algorithms, the performance was evaluated on data derived from public datasets [8,14,15].…”

Section: Introductionmentioning

confidence: 99%

Pneumothorax detection in chest radiographs: optimizing artificial intelligence system for accuracy and confounding bias reduction using in-image annotations in algorithm training

et al. 2021

View full text Add to dashboard Cite

Objectives Diagnostic accuracy of artificial intelligence (AI) pneumothorax (PTX) detection in chest radiographs (CXR) is limited by the noisy annotation quality of public training data and confounding thoracic tubes (TT). We hypothesize that in-image annotations of the dehiscent visceral pleura for algorithm training boosts algorithm’s performance and suppresses confounders. Methods Our single-center evaluation cohort of 3062 supine CXRs includes 760 PTX-positive cases with radiological annotations of PTX size and inserted TTs. Three step-by-step improved algorithms (differing in algorithm architecture, training data from public datasets/clinical sites, and in-image annotations included in algorithm training) were characterized by area under the receiver operating characteristics (AUROC) in detailed subgroup analyses and referenced to the well-established “CheXNet” algorithm. Results Performances of established algorithms exclusively trained on publicly available data without in-image annotations are limited to AUROCs of 0.778 and strongly biased towards TTs that can completely eliminate algorithm’s discriminative power in individual subgroups. Contrarily, our final “algorithm 2” which was trained on a lower number of images but additionally with in-image annotations of the dehiscent pleura achieved an overall AUROC of 0.877 for unilateral PTX detection with a significantly reduced TT-related confounding bias. Conclusions We demonstrated strong limitations of an established PTX-detecting AI algorithm that can be significantly reduced by designing an AI system capable of learning to both classify and localize PTX. Our results are aimed at drawing attention to the necessity of high-quality in-image localization in training data to reduce the risks of unintentionally biasing the training process of pathology-detecting AI algorithms. Key Points • Established pneumothorax-detecting artificial intelligence algorithms trained on public training data are strongly limited and biased by confounding thoracic tubes. • We used high-quality in-image annotated training data to effectively boost algorithm performance and suppress the impact of confounding thoracic tubes. • Based on our results, we hypothesize that even hidden confounders might be effectively addressed by in-image annotations of pathology-related image features.

show abstract

“…Various studies have shown the utility of crowdsourcing and citizen science in biological and medical image annotation [38][39][40][41]. Crowdsourcing for annotation and evaluation is advantageous because it is scalable, high throughput, cost-efficient, and accurate [42][43][44].…”

Section: Discussionmentioning

confidence: 99%

Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density

Chang

Beers

Brink

et al. 2020

Journal of the American College of Radiology

View full text Add to dashboard Cite

Objective: We developed deep learning algorithms to automatically assess BI-RADS breast density.Methods: Using a large multi-institution patient cohort of 108,230 digital screening mammograms from the Digital Mammographic Imaging Screening Trial, we investigated the effect of data, model, and training parameters on overall model performance and provided crowdsourcing evaluation from the attendees of the ACR 2019 Annual Meeting.

show abstract

Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest X-ray dataset

Cited by 38 publications

References 14 publications

Detection of the location of pneumothorax in chest X-rays using small artificial neural networks and a simple training process

Detection of the location of pneumothorax in chest X-rays using small artificial neural networks and a simple training process

Pneumothorax detection in chest radiographs: optimizing artificial intelligence system for accuracy and confounding bias reduction using in-image annotations in algorithm training

Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density

Contact Info

Product

Resources

About