Clinical Utility and Functionality of an Artificial Intelligence–Based App to Predict Mortality in COVID-19: Mixed Methods Analysis

Abstract

Background The artificial neural network (ANN) is an increasingly important tool in the context of solving complex medical classification problems. However, one of the principal challenges in leveraging artificial intelligence technology in the health care setting has been the relative inability to translate models into clinician workflow. Objective Here we demonstrate the development of a COVID-19 outcome prediction app that utilizes an ANN and assesses its usability in the clinical setting. Methods Usability assessment was conducted using the app, followed by a semistructured end-user interview. Usability was specified by effectiveness, efficiency, and satisfaction measures. These data were reported with descriptive statistics. The end-user interview data were analyzed using the thematic framework method, which allowed for the development of themes from the interview narratives. In total, 31 National Health Service physicians at a West London teaching hospital, including foundation physicians, senior house officers, registrars, and consultants, were included in this study. Results All participants were able to complete the assessment, with a mean time to complete separate patient vignettes of 59.35 (SD 10.35) seconds. The mean system usability scale score was 91.94 (SD 8.54), which corresponds to a qualitative rating of “excellent.” The clinicians found the app intuitive and easy to use, with the majority describing its predictions as a useful adjunct to their clinical practice. The main concern was related to the use of the app in isolation rather than in conjunction with other clinical parameters. However, most clinicians speculated that the app could positively reinforce or validate their clinical decision-making. Conclusions Translating artificial intelligence technologies into the clinical setting remains an important but challenging task. We demonstrate the effectiveness, efficiency, and system usability of a web-based app designed to predict the outcomes of patients with COVID-19 from an ANN.

An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study

“…Our findings align with, and build on, the work of others. For example, similar to our results, other clinicians have responded favorably to the usability of tools that use AI [ 36 , 37 ]. Although usability and accuracy were deemed important, our respondents asked for steps that could be taken in response to predictions and wanted to know that those actions would lead to better health, echoing the sentiment found in other studies [ 35 ].…”

“…As the absence of engagement has the potential to reduce trust and increase errors, researchers are starting to pay attention to end users [ 35 ] and are finding that usability of and satisfaction with AI tools are generally high [ 35 - 37 ]. Although most of these tools have targeted specialists, 1 study examined how primary care physicians use an AI tool to diagnose skin lesions [ 38 ].…”

“…Although most of these tools have targeted specialists, 1 study examined how primary care physicians use an AI tool to diagnose skin lesions [ 38 ]. Most of these studies used quantitative methods and examined tools that have already been developed [ 35 - 37 ]. This study is novel because it qualitatively assesses the use of a poorly controlled diabetes risk tool that has yet to be created and is based on the theory that early engagement with clinicians and staff will lead to methodological and design decisions that will support the tool’s implementation.…”

Incubation period, clinical and lung CT features for early prediction of COVID-19 deterioration: development and internal verification of a risk model

“…Since the emergence of the COVID-19 pandemic, many predictive models concerning the diagnosis and prognosis have emerged [ 5 , 9 , 16 , 29 – 31 ]. In the development of COVID-19 prediction models, artificial intelligence, including machine learning and deep learning, has been widely used to improve the accuracy and expansibility of the prediction models [ 32 ]. Lasso regression and Logistic regression used in our research belong to the machine learning algorithms.…”

Application of machine learning approach in emergency department to support clinical decision making for SARS-CoV-2 infected patients

“…Artificial intelligence and related tools could overcome these issues by supporting physicians in clinical routine. Accordingly, several clinical indexes, machine learning (ML) models and apps have been demonstrated to be a powerful approach for predicting hospitalization, disease severity and mortality in ED [ 14 – 18 ].…”