Piloting a Survey-Based Assessment of Transparency and Trustworthiness with Three Medical AI Tools

Fehr, Jana; Jaramillo-Gutierrez, Giovanna; Oala, Luis; Gröschel, Matthias I.; Bierwirth, Manuel; Balachandran, Pradeep; Werneck-Leite, Alixandro; Lippert, Christoph

doi:10.3390/healthcare10101923

Cited by 5 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As mentioned above, the application of artificial intelligence to healthcare involves several ethical concerns, such as unfair algorithmic bias [ 13 , 14 ]. In the vast majority of works, predictor models are treated as “black boxes”, without understanding the internal performance and being unable to explain how it reached a certain prediction.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Improving Intensive Care Unit Early Readmission Prediction Using Optimized and Explainable Machine Learning

González-Nóvoa

Campanioni

Busto

et al. 2023

IJERPH

View full text Add to dashboard Cite

It is of great interest to develop and introduce new techniques to automatically and efficiently analyze the enormous amount of data generated in today’s hospitals, using state-of-the-art artificial intelligence methods. Patients readmitted to the ICU in the same hospital stay have a higher risk of mortality, morbidity, longer length of stay, and increased cost. The methodology proposed to predict ICU readmission could improve the patients’ care. The objective of this work is to explore and evaluate the potential improvement of existing models for predicting early ICU patient readmission by using optimized artificial intelligence algorithms and explainability techniques. In this work, XGBoost is used as a predictor model, combined with Bayesian techniques to optimize it. The results obtained predicted early ICU readmission (AUROC of 0.92 ± 0.03) improves state-of-the-art consulted works (whose AUROC oscillate between 0.66 and 0.78). Moreover, we explain the internal functioning of the model by using Shapley Additive Explanation-based techniques, allowing us to understand the model internal performance and to obtain useful information, as patient-specific information, the thresholds from which a feature begins to be critical for a certain group of patients, and the feature importance ranking.

show abstract

Section: Discussionmentioning

confidence: 99%

“…The application of artificial intelligence to healthcare involves several ethical concerns, such as unfair algorithmic bias [ 13 , 14 , 15 , 16 ]. This is strongly related with the explainability of AI models.…”

Section: Introductionmentioning

confidence: 99%

Improving Intensive Care Unit Early Readmission Prediction Using Optimized and Explainable Machine Learning

González-Nóvoa

Campanioni

Busto

et al. 2023

IJERPH

View full text Add to dashboard Cite

show abstract

“…Along with those reported in the previous section, we could observe other major trends being further continued. They mainly relate to: (i) the assessment of transparency and trustworthiness of medical artificial intelligence tools to deal with issues originated within a broad and comprehensive digital landscape [28,29], and (ii) the importance of data collection, management, and analysis in the One (Digital) Health domain, and its integration with the FAIR (Findable, Accessible, Interoperable, and Reusable) principles to create an innovative framework [6,16,24,30]. These aspects leave a broad room for further investigations.…”

Section: Discussionmentioning

confidence: 99%

Leveraging Data and Technology to Enhance Interdisciplinary Collaboration and Health Outcomes

Tamburis,

Benis

2023

Yearb Med Inform

View full text Add to dashboard Cite

Objective: To give an overview of recent research and propose a selection of best papers published in 2022 in Informatics for One Health. Methods: An extensive search using PubMed and Web of Science was conducted to identify peer-reviewed articles published between December 2021 and December 2022, in order to find relevant publications in the ‘Informatics for One Health’ field. The selection process comprised three steps: (i) eight candidate best papers were first selected by the two section editors; (ii) external reviewers from internationally renowned research teams reviewed each candidate best paper; and (iii) the editorial committee of the Yearbook conducted the final best paper selection. Results: The candidate best papers represent studies that characterized significant challenges facing Informatics for One Health. Other trends of interest related to the deployment of medical artificial intelligence tools and the implementation of the FAIR principles within the One Health broad scenario. In general, papers identified in the search fell into one of the following categories: 1) Health improvement via digital technology; 2) Climate change/Environment/Biodiversity; and 3) Maturity of healthcare services. Conclusion: The topic turns extremely important in the next future for what concerns the need to understand complex interactions in order to safeguard the health of populations and ecosystems.

show abstract

“…We used a previously developed survey-based assessment ( 30 ) to assess whether the publicly available product documentation suffices transparency considerations for trustworthy medical AI. The survey was designed to elicit transparent reporting about the model design, development and validation of learning-based AI algorithms that predict health outcomes.…”

Section: Methodsmentioning

confidence: 99%

“…The survey was designed to elicit transparent reporting about the model design, development and validation of learning-based AI algorithms that predict health outcomes. The survey includes 78 questions about the (1) intended use of the product, (2) the machine learning methodology (3) training data information (4) implemented ethical considerations, (5) technical and clinical validation conduct and results following medical AI audit proposals ( 40 – 42 ), and (6) caveats for clinical deployment ( 30 ). These questions were drawn from existing reporting guidelines for machine learning algorithms ( 22 , 23 ) in healthcare ( 9 , 27 , 43 , 44 ), diagnostic accuracy studies ( 45 ), medical AI validation studies ( 25 , 26 , 28 ), and trustworthy AI guidelines ( 20 , 21 , 29 , 46 – 48 ).…”

Section: Methodsmentioning

confidence: 99%

A trustworthy AI reality-check: the lack of transparency of artificial intelligence products in healthcare

Fehr,

Citro,

Malpani

et al. 2024

Front. Digit. Health

Self Cite

View full text Add to dashboard Cite

Trustworthy medical AI requires transparency about the development and testing of underlying algorithms to identify biases and communicate potential risks of harm. Abundant guidance exists on how to achieve transparency for medical AI products, but it is unclear whether publicly available information adequately informs about their risks. To assess this, we retrieved public documentation on the 14 available CE-certified AI-based radiology products of the II b risk category in the EU from vendor websites, scientific publications, and the European EUDAMED database. Using a self-designed survey, we reported on their development, validation, ethical considerations, and deployment caveats, according to trustworthy AI guidelines. We scored each question with either 0, 0.5, or 1, to rate if the required information was “unavailable”, “partially available,” or “fully available.” The transparency of each product was calculated relative to all 55 questions. Transparency scores ranged from 6.4% to 60.9%, with a median of 29.1%. Major transparency gaps included missing documentation on training data, ethical considerations, and limitations for deployment. Ethical aspects like consent, safety monitoring, and GDPR-compliance were rarely documented. Furthermore, deployment caveats for different demographics and medical settings were scarce. In conclusion, public documentation of authorized medical AI products in Europe lacks sufficient public transparency to inform about safety and risks. We call on lawmakers and regulators to establish legally mandated requirements for public and substantive transparency to fulfill the promise of trustworthy AI for health.

show abstract

Piloting a Survey-Based Assessment of Transparency and Trustworthiness with Three Medical AI Tools

Cited by 5 publications

References 32 publications

Improving Intensive Care Unit Early Readmission Prediction Using Optimized and Explainable Machine Learning

Improving Intensive Care Unit Early Readmission Prediction Using Optimized and Explainable Machine Learning

Leveraging Data and Technology to Enhance Interdisciplinary Collaboration and Health Outcomes

A trustworthy AI reality-check: the lack of transparency of artificial intelligence products in healthcare

Contact Info

Product

Resources

About