Comprehensive AI Model Development for Gleason Grading: From Scanning, Cloud-Based Annotation to Pathologist-AI Interaction

Huo, Xinmi; Ong, Kok Haur; Lau, Kean‐Wah; Gole, Leena; Tan, Char Loo; Zhang, Chongchong; Zhang, Yonghui; Zhu, Xiaohui; Li, Longjie; Han, Ho Jae; Young, D. R.; Lu, Haoda; Xu, Jun; Chen, Wanyuan; Sanders, Stephan; Kuan, Lee Hwee; Hue, Susan Swee-Shan; Yu, Weimiao; Tan, Soo Yong

doi:10.2139/ssrn.4172090

Cited by 3 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this way, iQC achieved AUROC of 0.9966 for the biopsy/nonbiopsy prediction task (Fig 3A). Testing this on all other VAMC data, we found AUROC substantially dropped to AUROC of 0.8346 (Fig 3B To test how well iQC generalized to unseen data, and more specifically to non-Veteran data, we evaluated iQC's biopsy/nonbiopsy predictor on AGGC2022 data [15]. We found an AUROC of 0.9824 (Fig 3E).…”

Section: Resultsmentioning

confidence: 87%

“…To test how well iQC generalized to unseen data, and more specifically to non-Veteran data, we evaluated iQC's biopsy/nonbiopsy predictor on AGGC2022 data [15]. We found an AUROC of 0.9824 (Fig 3E).…”

Section: Resultsmentioning

confidence: 99%

“…A2 : Coloring A1 according to the VAMC/Institution that made the slide, it appears Institution Beta ( β ) accounts for most “fail all tissue” slides, which is a batch effect. A3 : Zero AGGC2022 slides fail iQC’s quality control, suggesting this dataset is high quality[15]. Most slides are “review…”, which is a warning.…”

Section: Resultsmentioning

confidence: 99%

“…γ, δ, ϵ ) is high but this is an underpowered test because there are only 5 nonbiopsies. E : AUROC on the public AGGC2022 dataset[15] is high, indicating iQC’s biopsy/nonbiopsy predictor may generalize well to unseen data. F1 , F2 : iQC mistakenly classified this prostatectomy sample as a needle biopsy, perhaps because the distribution of tissue is long and thin, very loosely like a needle biopsy.…”

Section: Resultsmentioning

confidence: 99%

“…For comparison, iQC only marks 227 slides (6.2%) as "fail all tissue" (panel A2). A5: Zero AGGC slides fail iQC's quality control, suggesting this dataset is high quality [18]. Most slides are "review...", which is a warning.…”

Section: Iqc Surgical Prediction Performance Strong On External High ...mentioning

confidence: 99%

See 4 more Smart Citations

iQC: machine-learning-driven prediction of surgical procedure uncovers systematic confounds of cancer whole slide images in specific medical centers

Schaumberg,

Lewis,

Nazarian

et al. 2023

Preprint

View full text Add to dashboard Cite

Problem: The past decades have yielded an explosion of research using artificial intelligence for cancer detection and diagnosis in the field of computational pathology. Yet, an often unspoken assumption of this research is that a glass microscopy slide faithfully represents the underlying disease. Here we show systematic failure modes may dominate the slides digitized from a given medical center, such that neither the whole slide images nor the glass slides are suitable for rendering a diagnosis. Methods: We quantitatively define high quality data as a set of whole slide images where the type of surgery the patient received may be accurately predicted by an automated system such as ours, called "iQC". We find iQC accurately distinguished biopsies from nonbiopsies, e.g. prostatectomies or transurethral resections (TURPs, a.k.a. prostate chips), only when the data qualitatively appeared to be high quality, e.g. vibrant histopathology stains and minimal artifacts. Crucially, prostate needle biopsies appear as thin strands of tissue, whereas prostatectomies and TURPs appear as larger rectangular blocks of tissue. Therefore, when the data are of high quality, iQC (i) accurately classifies pixels as tissue, (ii) accurately generates statistics that describe the distribution of tissue in a slide, and (iii) accurately predicts surgical procedure. Results: While we do not control any medical center's protocols for making or storing slides, we developed the iQC tool to hold all medical centers and datasets to the same objective standard of quality. We validate this standard across five Veterans Affairs Medical Centers (VAMCs) and the Automated Gleason Grading Challenge (AGGC) 2022 public dataset. For our surgical procedure prediction task, we report an Area Under Receiver Operating Characteristic (AUROC) of 0.9966-1.000 at the VAMCs that consistently produce high quality data and AUROC of 0.9824 for the AGGC dataset. In contrast, we report an AUROC of 0.7115 at the VAMC that consistently produced poor quality data. An attending pathologist determined poor data quality was likely driven by faded histopathology stains and protocol differences among VAMCs. Specifically, this VAMC produced slides entirely by hand, whereas all other VAMCs leveraged automated methods to produce slides. Conclusion: Our surgical procedure prediction AUROC may be a quantitative indicator positively associated with high data quality at a medical center or for a specific dataset. To produce high quality data, we recommend producing slides using robotics or other forms of automation whenever possible. We recommend scanning slides digitally before the glass slide has time to develop signs of age, e.g faded stains and acrylamide bubbles. To our knowledge, iQC is the first automated system in computational pathology that validates data quality against objective evidence, e.g. surgical procedure data available in the EHR or LIMS, which requires zero efforts or annotations from anatomic pathologists.

show abstract

Section: Resultsmentioning

confidence: 87%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Iqc Surgical Prediction Performance Strong On External High ...mentioning

confidence: 99%

See 3 more Smart Citations