A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care

Abstract

Background Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and retraining. Objectives This study aimed to present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end users. Methods Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semistructured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies. Results Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of “good.” We reviewed persona-based case studies that illustrate functionalities of this novel prototype. Conclusion The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.

Review of Visualization Approaches in Deep Learning Models of Glaucoma

“…In time-sensitive clinical settings, an easy-to-use interface was the most important CDS characteristic to clinicians 67 . The “System Usability Scale (SUS)” and the “Post-Study System Usability Questionnaire” are useful tools for assessing the usability of software interfaces 68 . The Post-Study System Usability Questionnaire is a usability evaluation survey form based on scenarios that were developed by IBM 69 .…”

“…67 The "System Usability Scale (SUS)" and the "Post-Study System Usability Questionnaire" are useful tools for assessing the usability of software interfaces. 68 The Post-Study System Usability Questionnaire is a usability evaluation survey form based on scenarios that were developed by IBM. 69 It consists of 19 items focused on 5 usability characteristics of a system: rapid completion of the task, ease of learning, high-quality documentation and online information, functional adequacy, and rapid acquisition of productivity.…”

Curricular Interventions in Medical Schools: Maximizing Community Engagement Through Communities of Practice

“…iven the rapid changes in the demographic, social, and personal health landscapes, the health care sector needs to continually update and reinvent itself to meet the evolving needs of people it serves. Changes in health insurance products, 1 use of information technology, 2 big data analytics, 3 and predictive modeling, 4 as well as changing diagnostic tools, 5 prescription drugs, 6 and expectations of consumers 7 will continue to revolutionize how health care is delivered in the coming decade. These changes will increase pressures on health professions training programs to keep up with the changes that are occurring in the real world.…”