Background Machine learning (ML)–based clinical decision support systems (CDSS) are popular in clinical practice settings but are often criticized for being limited in usability, interpretability, and effectiveness. Evaluating the implementation of ML-based CDSS is critical to ensure CDSS is acceptable and useful to clinicians and helps them deliver high-quality health care. Malnutrition is a common and underdiagnosed condition among hospital patients, which can have serious adverse impacts. Early identification and treatment of malnutrition are important. Objective This study aims to evaluate the implementation of an ML tool, Malnutrition Universal Screening Tool (MUST)–Plus, that predicts hospital patients at high risk for malnutrition and identify best implementation practices applicable to this and other ML-based CDSS. Methods We conducted a qualitative postimplementation evaluation using in-depth interviews with registered dietitians (RDs) who use MUST-Plus output in their everyday work. After coding the data, we mapped emergent themes onto select domains of the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework. Results We interviewed 17 of the 24 RDs approached (71%), representing 37% of those who use MUST-Plus output. Several themes emerged: (1) enhancements to the tool were made to improve accuracy and usability; (2) MUST-Plus helped identify patients that would not otherwise be seen; perceived usefulness was highest in the original site; (3) perceived accuracy varied by respondent and site; (4) RDs valued autonomy in prioritizing patients; (5) depth of tool understanding varied by hospital and level; (6) MUST-Plus was integrated into workflows and electronic health records; and (7) RDs expressed a desire to eventually have 1 automated screener. Conclusions Our findings suggest that continuous involvement of stakeholders at new sites given staff turnover is vital to ensure buy-in. Qualitative research can help identify the potential bias of ML tools and should be widely used to ensure health equity. Ongoing collaboration among CDSS developers, data scientists, and clinical providers may help refine CDSS for optimal use and improve the acceptability of CDSS in the clinical context.
BACKGROUND Machine learning (ML)-based CDSS are popular in clinical practice settings but are often criticized for being limited in usability, interpretability and effectiveness. Evaluating implementation of ML-based CDSS is critical to ensure CDSS is acceptable and useful to clinicians and helps them deliver high quality health care. Malnutrition is a common and underdiagnosed condition among hospital patients that can have serious adverse impacts. Early identification and treatment of malnutrition is important. OBJECTIVE To evaluate the implementation of a ML tool, Malnutrition Universal Screening Tool (MUST)-Plus, that predicts hospital patients at high risk for malnutrition and identify implementation best practices applicable to this and other ML-based clinical decision support systems (CDSS). METHODS We conducted a qualitative post-implementation evaluation using in-depth interviews with registered dietitians (RDs) who use MUST-Plus output in their everyday work. After coding the data, we mapped emergent themes onto select domains of the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework. RESULTS We interviewed 17 of the 24 RDs approached (70.8%), representing 36.9% of those who use MUST-Plus output. Several themes emerged: (1) Enhancements to the tool were made to improve accuracy and usability; (2) MUST-Plus helped identify patients that would not otherwise be seen. Perceived usefulness was highest in the original site; (3) Perceived accuracy varied by respondent and site; (4) RDs valued autonomy in prioritizing patients; (5) Depth of tool understanding varied by hospital and level; (6) MUST-Plus was integrated into workflows and electronic health records; (7) RDs expressed a desire to eventually have one automated screener. Our findings suggest that continuous involvement of stakeholders at new sites given staff turnover is vital to ensure buy-in. Qualitative research can help identify potential bias of ML tools and should be widely used to ensure health equity. CONCLUSIONS Ongoing collaboration between CDSS developers, data scientists, and clinicians may help refine CDSS for optimal use and improve acceptability of CDSS in the clinical context. CLINICALTRIAL N/A
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.