Artificial intelligence in radiology: 100 commercially available products and their scientific evidence

Abstract

Objectives Map the current landscape of commercially available artificial intelligence (AI) software for radiology and review the availability of their scientific evidence. Methods We created an online overview of CE-marked AI software products for clinical radiology based on vendor-supplied product specifications (www.aiforradiology.com). Characteristics such as modality, subspeciality, main task, regulatory information, deployment, and pricing model were retrieved. We conducted an extensive literature search on the available scientific evidence of these products. Articles were classified according to a hierarchical model of efficacy. Results The overview included 100 CE-marked AI products from 54 different vendors. For 64/100 products, there was no peer-reviewed evidence of its efficacy. We observed a large heterogeneity in deployment methods, pricing models, and regulatory classes. The evidence of the remaining 36/100 products comprised 237 papers that predominantly (65%) focused on diagnostic accuracy (efficacy level 2). From the 100 products, 18 had evidence that regarded level 3 or higher, validating the (potential) impact on diagnostic thinking, patient outcome, or costs. Half of the available evidence (116/237) were independent and not (co-)funded or (co-)authored by the vendor. Conclusions Even though the commercial supply of AI software in radiology already holds 100 CE-marked products, we conclude that the sector is still in its infancy. For 64/100 products, peer-reviewed evidence on its efficacy is lacking. Only 18/100 AI products have demonstrated (potential) clinical impact. Key Points • Artificial intelligence in radiology is still in its infancy even though already 100 CE-marked AI products are commercially available. • Only 36 out of 100 products have peer-reviewed evidence of which most studies demonstrate lower levels of efficacy. • There is a wide variety in deployment strategies, pricing models, and CE marking class of AI products for radiology.

Artificial Intelligence Based Algorithms for Prostate Cancer Classification and Detection on Magnetic Resonance Imaging: A Narrative Review

“…For all studies, AI algorithm characteristics, MRI sequences used, study design and cohort size, ground truth for PCa, and performance were extracted. Studies were graded using an adaptation of the hierarchical model for diagnostic imaging efficacy from Fryback and Thornbury, applicable for assessment of AI software in clinical practice (Table 1) [20,21]. For categorization between various AI-CAD algorithms, studies were categorized within two common tasks [13,19]:…”

“…Hierarchical model of efficacy to assess the contribution of AI software to the diagnostic imaging process. An adapted model from van Leeuwen et al [21], based on Fryback and Thornbury's hierarchical model of efficacy [20].…”

Accuracy of Machine Learning Algorithms for the Classification of Molecular Features of Gliomas on MRI: A Systematic Literature Review and Meta-Analysis

“…Automated diagnosis from medical imaging through artificial intelligence could help to overcome the mismatch between the increasing amount of diagnostic images and the capacity of available specialists [ 91 ]. More than 100 MLAs have now CE-marked, 57 of which can be used within neuroimaging features.…”

Machine learning solutions in radiology: does the emperor have no clothes?

“…As reported in the current issue of European Radiology, this trend is also reflected in the number of commercial products using ML that are being offered to radiology units. The authors have presented several issues which may negatively impact the real-world applicability of such software solutions, as well as provide a useful database to guide radiologists in their assessment (www.aiforradiology.com) [2]. Recently, guidelines to assess commercial ML solutions (the ECLAIR guidelines) have also been proposed to help physicians in this task [3].…”

“…First, there is the question of scientific validation of CE-marked ML packages which is lacking at best. As clearly shown by the authors in the results section, even though already 100 CE-marked AI products are commercially available, only 36 of these products have peerreviewed evidence and most demonstrate only lower levels of efficacy [2]. Studies were almost exclusively retrospective and did not sufficiently address clinical impact of the ML software, limiting themselves in most cases to feasibility investigations and isolated performance assessment.…”