Abstract:Early identification of patients with life-threatening risks such as delirium is crucial in order to initiate preventive actions as quickly as possible. Despite intense research on machine learning for the prediction of clinical outcomes, the acceptance of the integration of such complex models in clinical routine remains unclear. The aim of this study was to evaluate user acceptance of an already implemented machine learning-based application predicting the risk of delirium for in-patients. We applied a mixed… Show more
“…An important aspect is the acceptance by healthcare professionals and the actual use of such system in clinical routine. A previous evaluation of a similar tool, predicting delirium in hospitalized patients, showed a high user acceptance regarding ease of use and usefulness [ 23 ]. Naturally, we aim to include user satisfaction and acceptance of this technology when it comes to predicting dysphagia.…”
Section: Discussionmentioning
confidence: 99%
“… Fig. 1 Visualization of the machine learning-based dysphagia prediction in the hospital information system ( a ) [ 23 ] and in a web application ( b ). This screenshot is fictional and not referring to a real patient …”
Based on a large number of pre-existing documented electronic health records (EHR), we developed a machine learning (ML) algorithm for detection of dysphagia and aspiration pneumonia. The aim of our study was to prospectively apply this algorithm in two large patient cohorts. The tool was integrated in the hospital information system of a secondary care hospital in Austria. Based on existing data such as diagnoses, laboratory, and medication, dysphagia risk was predicted automatically, and patients were stratified into three risk groups. Patients’ risk groups and risk factors were visualized in a web application. Prospective predictions of 1270 admissions to geriatric or internal medicine departments were compared with the occurrence of dysphagia or aspiration pneumonia of routinely documented events. The discriminative performance for internal medicine patients (n = 885) was excellent with an AUROC of 0.841, a sensitivity of 74.2%, and a specificity of 84.1%. For the smaller geriatric cohort (n = 221), the AUROC was 0.758, sensitivity 44.4%, and specificity 93.0%. For both cohorts, calibration plots showed a slight overestimation of the risk. This is the first study to evaluate the performance of a ML-based prediction tool for dysphagia in a prospective clinical setting. Future studies should validate the predictions on data of systematic dysphagia screening by specialists and evaluate user satisfaction and acceptance. The ML-based dysphagia prediction tool achieved an excellent performance in the internal medicine cohort. More data are needed to determine the performance in geriatric patients.
“…An important aspect is the acceptance by healthcare professionals and the actual use of such system in clinical routine. A previous evaluation of a similar tool, predicting delirium in hospitalized patients, showed a high user acceptance regarding ease of use and usefulness [ 23 ]. Naturally, we aim to include user satisfaction and acceptance of this technology when it comes to predicting dysphagia.…”
Section: Discussionmentioning
confidence: 99%
“… Fig. 1 Visualization of the machine learning-based dysphagia prediction in the hospital information system ( a ) [ 23 ] and in a web application ( b ). This screenshot is fictional and not referring to a real patient …”
Based on a large number of pre-existing documented electronic health records (EHR), we developed a machine learning (ML) algorithm for detection of dysphagia and aspiration pneumonia. The aim of our study was to prospectively apply this algorithm in two large patient cohorts. The tool was integrated in the hospital information system of a secondary care hospital in Austria. Based on existing data such as diagnoses, laboratory, and medication, dysphagia risk was predicted automatically, and patients were stratified into three risk groups. Patients’ risk groups and risk factors were visualized in a web application. Prospective predictions of 1270 admissions to geriatric or internal medicine departments were compared with the occurrence of dysphagia or aspiration pneumonia of routinely documented events. The discriminative performance for internal medicine patients (n = 885) was excellent with an AUROC of 0.841, a sensitivity of 74.2%, and a specificity of 84.1%. For the smaller geriatric cohort (n = 221), the AUROC was 0.758, sensitivity 44.4%, and specificity 93.0%. For both cohorts, calibration plots showed a slight overestimation of the risk. This is the first study to evaluate the performance of a ML-based prediction tool for dysphagia in a prospective clinical setting. Future studies should validate the predictions on data of systematic dysphagia screening by specialists and evaluate user satisfaction and acceptance. The ML-based dysphagia prediction tool achieved an excellent performance in the internal medicine cohort. More data are needed to determine the performance in geriatric patients.
“…If an AI model is not accepted by the users, it will not influence clinical decision-making 69. Factors such as usefulness and ease of use, which are described in the technology acceptance model, are demonstrated to improve the likelihood of successful implementation and should therefore be taken into account 70 71. Furthermore, implementation efforts should be accompanied by clear and standardised communication of AI model information towards end users to promote transparency and trust, for example, by providing an ‘AI model facts label’ 72.…”
Section: Phase Iv: Implementing and Governing Of Aimentioning
ObjectiveAlthough the role of artificial intelligence (AI) in medicine is increasingly studied, most patients do not benefit because the majority of AI models remain in the testing and prototyping environment. The development and implementation trajectory of clinical AI models are complex and a structured overview is missing. We therefore propose a step-by-step overview to enhance clinicians’ understanding and to promote quality of medical AI research.MethodsWe summarised key elements (such as current guidelines, challenges, regulatory documents and good practices) that are needed to develop and safely implement AI in medicine.ConclusionThis overview complements other frameworks in a way that it is accessible to stakeholders without prior AI knowledge and as such provides a step-by-step approach incorporating all the key elements and current guidelines that are essential for implementation, and can thereby help to move AI from bytes to bedside.
“…The type of clinician involved in implementing machine learning tools was most commonly a physician (n=3, 30.0%) or nurse (n=3, 30.0%). Regarding the PROGRESS-PLUS criteria (online supplemental appendix 9), seven studies reported patient or clinician sex [27][28][29][30][31][32][33] and four studies reported additional PROGRESS-PLUS criteria (race, type of health insurance, rural/urban). 27 28 34 35 The demographic variables were only calculated for studies that reported them.…”
ObjectivesTo identify ML tools in hospital settings and how they were implemented to inform decision-making for patient care through a scoping review. We investigated the following research questions: What ML interventions have been used to inform decision-making for patient care in hospital settings? What strategies have been used to implement these ML interventions?DesignA scoping review was undertaken. MEDLINE, Embase, Cochrane Central Register of Controlled Trials (CENTRAL) and the Cochrane Database of Systematic Reviews (CDSR) were searched from 2009 until June 2021. Two reviewers screened titles and abstracts, full-text articles, and charted data independently. Conflicts were resolved by another reviewer. Data were summarised descriptively using simple content analysis.SettingHospital setting.ParticipantAny type of clinician caring for any type of patient.InterventionMachine learning tools used by clinicians to inform decision-making for patient care, such as AI-based computerised decision support systems or “‘model-based’” decision support systems.Primary and secondary outcome measuresPatient and study characteristics, as well as intervention characteristics including the type of machine learning tool, implementation strategies, target population. Equity issues were examined with PROGRESS-PLUS criteria.ResultsAfter screening 17 386 citations and 3474 full-text articles, 20 unique studies and 1 companion report were included. The included articles totalled 82 656 patients and 915 clinicians. Seven studies reported gender and four studies reported PROGRESS-PLUS criteria (race, health insurance, rural/urban). Common implementation strategies for the tools were clinician reminders that integrated ML predictions (44.4%), facilitated relay of clinical information (17.8%) and staff education (15.6%). Common barriers to successful implementation of ML tools were time (11.1%) and reliability (11.1%), and common facilitators were time/efficiency (13.6%) and perceived usefulness (13.6%).ConclusionsWe found limited evidence related to the implementation of ML tools to assist clinicians with patient healthcare decisions in hospital settings. Future research should examine other approaches to integrating ML into hospital clinician decisions related to patient care, and report on PROGRESS-PLUS items.FundingCanadian Institutes of Health Research (CIHR) Foundation grant awarded to SES and the CIHR Strategy for Patient Oriented-Research Initiative (GSR-154442).Scoping review registrationhttps://osf.io/e2mna.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
