FDA-approved deep learning software application versus radiologists with different levels of expertise: detection of intracranial hemorrhage in a retrospective single-center study

Kau, Thomas; Ziurlys, Mindaugas; Taschwer, Manuel; Kloss‐Brandstätter, Anita; Gräβner, G.; Deutschmann, Hannes

doi:10.1007/s00234-021-02874-w

Cited by 9 publications

(8 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…S et al 2022/ USA [ 29 ] Prospective ICH DNN NA Single/Real-time data 2D 98 99 NA 0.99 Seyam, M., et al 2022/ Switzerland [ 40 ] Prospective ICH DL 256-section scanners (Somatom Force and Somatom Definition Flash, Siemens) Single/Real-time data 2D 87.2 93.9 93 NA Altuve, M., & Pérez, A. 2022/Venezuela [ 22 ] Retrospective ICH ResNet-18 NA Single/Real-time data 2D 95.65 96.2 95.93 NA Tang, Z., et al 2022/China[ 41 ] Retrospective ICH CNN NA Single/Real-time data 2D 91.97 88.37 90.58 NA Cortes-Ferre L, et al 2022/ Spain [ 26 ] Retrospective ICH DL NA Single/Benchmark 2D 91.4 94 92.7 0.978 Kau, T., et al 2022/ Austria [ 30 ] ...…”

Section: Resultsmentioning

confidence: 99%

Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study

Maghami,

Sattari,

Tahmasbi

et al. 2023

BioMed Eng OnLine

View full text Add to dashboard Cite

Background This systematic review and meta-analysis were conducted to objectively evaluate the evidence of machine learning (ML) in the patient diagnosis of Intracranial Hemorrhage (ICH) on computed tomography (CT) scans. Methods Until May 2023, systematic searches were conducted in ISI Web of Science, PubMed, Scopus, Cochrane Library, IEEE Xplore Digital Library, CINAHL, Science Direct, PROSPERO, and EMBASE for studies that evaluated the diagnostic precision of ML model-assisted ICH detection. Patients with and without ICH as the target condition who were receiving CT-Scan were eligible for the research, which used ML algorithms based on radiologists' reports as the gold reference standard. For meta-analysis, pooled sensitivities, specificities, and a summary receiver operating characteristics curve (SROC) were used. Results At last, after screening the title, abstract, and full paper, twenty-six retrospective and three prospective, and two retrospective/prospective studies were included. The overall (Diagnostic Test Accuracy) DTA of retrospective studies with a pooled sensitivity was 0.917 (95% CI 0.88–0.943, I2 = 99%). The pooled specificity was 0.945 (95% CI 0.918–0.964, I2 = 100%). The pooled diagnostic odds ratio (DOR) was 219.47 (95% CI 104.78–459.66, I2 = 100%). These results were significant for the specificity of the different network architecture models (p-value = 0.0289). However, the results for sensitivity (p-value = 0.6417) and DOR (p-value = 0.2187) were not significant. The ResNet algorithm has higher pooled specificity than other algorithms with 0.935 (95% CI 0.854–0.973, I2 = 93%). Conclusion This meta-analysis on DTA of ML algorithms for detecting ICH by assessing non-contrast CT-Scans shows the ML has an acceptable performance in diagnosing ICH. Using ResNet in ICH detection remains promising prediction was improved via training in an Architecture Learning Network (ALN).

show abstract

Section: Resultsmentioning

confidence: 99%

Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study

Maghami,

Sattari,

Tahmasbi

et al. 2023

BioMed Eng OnLine

View full text Add to dashboard Cite

show abstract

“…The error rate of radiologists is incredibly low (Strub et al, 2007) and previous assessment of other ICH detection software (AIDoc) has shown that residents, even when under time pressure, outperform the algorithm (Kau et al, 2022). We found that scans presenting SDH resulted in the highest frequency of FN outputs (Figure II).…”

Section: Discussionmentioning

confidence: 99%

“…Although AI has the potential to optimize the reading of medical imaging, there is evidence to suggest that these models are not completely generalizable and do not identify ICH with the same proficiency across different sites. One group evaluated AIDoc, a commercially approved deep learning software, in the detection of ICH (Kau et al, 2022). They found an accuracy of 94%, a marked decrease from a different study reporting an accuracy of 98% (Ojeda et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

A retrospective analysis of the diagnostic performance of an FDA approved software for the detection of intracranial hemorrhage

Pourmussa,

Gorovoy

2023

Preprint

View full text Add to dashboard Cite

ObjectiveTo determine the sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) of Rapid ICH, a commercially available AI model, in detecting intracranial hemorrhage (ICH) on non-contrast computed tomography (NCCT) examinations of the head at a single regional medical center.MethodsRapidAI’s Rapid ICH is incorporated into real time hospital workflow to assist radiologists in the identification of ICH on NCCT examinations of the head. 412 examinations from August 2022 to January 2023 were pulled for analysis. Scans in which it was unclear if ICH was present or not, as well as scans significantly affected by motion artifact were excluded from the study. The sensitivity, specificity, accuracy, PPV, and NPV of the software were then assessed retrospectively for the remaining 406 NCCT examinations using prior radiologist report as the ground-truth. A two tailedztest with α = 0.05 was preformed to determine if the sensitivity and specificity of the software in this study were significantly different from Rapid ICH’s reported sensitivity and specificity. Additionally, the software’s performance was analyzed separately for the male and female populations and a chi-square test of independence was used to determine if model correctness significantly depended on sex.ResultsOf the 406 scans assessed, Rapid ICH flagged 82 ICH positive cases and 324 ICH negative cases. There were 80 examinations (19.7%) truly positive for ICH and 326 examinations (80.3%) negative for ICH. This resulted in a sensitivity of 71.3%, 95% CI [61.3%-81.2%], a specificity of 92.3%, 95% CI [89.4%-95.2%], an accuracy of 88.2%, 95% CI [85.0%-91.3%], a PPV of 69.5%, 95% CI [59.5%-79.5%], and an NPV of 92.9%, 95% CI [90.1%-95.7%]. Two examinations were excluded due to no existing information on patient sex in the electronic medical record. The resulting sensitivity was significantly different from the sensitivity reported by Rapid ICH (95%),z= 2.60,p= .009 although the resulting specificity was not significantly different from the specificity reported by Rapid ICH (94%),z= 0.65,p= .517. The model performance did not depend on sex per the chi-square test of independence:X2(1 degree of freedom,N= 404) = 1.95,p= .162 (p> 0.05).ConclusionRapid ICH demonstrates exceptional capability in the identification of ICH, but its performance when used at this site differs from the values advertised by the company, and from assessments of the model’s performance by other research groups. Specifically, the sensitivity of the software at this site is significantly different from the sensitivity reported by the company. These results underscore the necessity for independent evaluation of the software at institutions where it is implemented.

show abstract

“…In these days, deep learning models can achieve good performance in many applications [1][2][3][4][5]. Generally, the performance of a deep learning model depends on the captured features [6][7][8][9][10].…”

Section: Introductionmentioning

confidence: 99%

“…Our contribution can be summarized as the following. (1) We built a novel framework that achieves the scalability of the deep learning system. As the number of domains increased, the difficulty of transfer learning is increased as it has to consider the performance of all domains.…”

Section: Introductionmentioning

confidence: 99%

The Classification of Multi‐Domain Samples Based on the Cooperation of Multiple Models

Song

et al. 2022

Complexity

View full text Add to dashboard Cite

This article proposed a novel classification framework that can classify the samples of multiple domains based on the outputs of multiple models. Different from the existing methods that train single model on all domains, our framework trains multiple models on each domain. On a testing sample, the outputs of all trained models are used to predict the domain of this sample. Then, this sample is classified by the output of models that belong to the predicted domain. Experiments show that our framework achieved higher accuracy than the existing methods. Furthermore, our framework achieves good scalability on multiple domains.

show abstract

FDA-approved deep learning software application versus radiologists with different levels of expertise: detection of intracranial hemorrhage in a retrospective single-center study

Cited by 9 publications

References 25 publications

Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study

Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study

A retrospective analysis of the diagnostic performance of an FDA approved software for the detection of intracranial hemorrhage

The Classification of Multi‐Domain Samples Based on the Cooperation of Multiple Models

Contact Info

Product

Resources

About