Can a Single Variable Predict Early Dropout From Digital Health Interventions? Comparison of Predictive Models From Two Large Randomized Trials

Abstract

Background A single generalizable metric that accurately predicts early dropout from digital health interventions has the potential to readily inform intervention targets and treatment augmentations that could boost retention and intervention outcomes. We recently identified a type of early dropout from digital health interventions for smoking cessation, specifically, users who logged in during the first week of the intervention and had little to no activity thereafter. These users also had a substantially lower smoking cessation rate with our iCanQuit smoking cessation app compared with users who used the app for longer periods. Objective This study aimed to explore whether log-in count data, using standard statistical methods, can precisely predict whether an individual will become an iCanQuit early dropout while validating the approach using other statistical methods and randomized trial data from 3 other digital interventions for smoking cessation (combined randomized N=4529). Methods Standard logistic regression models were used to predict early dropouts for individuals receiving the iCanQuit smoking cessation intervention app, the National Cancer Institute QuitGuide smoking cessation intervention app, the WebQuit.org smoking cessation intervention website, and the Smokefree.gov smoking cessation intervention website. The main predictors were the number of times a participant logged in per day during the first 7 days following randomization. The area under the curve (AUC) assessed the performance of the logistic regression models, which were compared with decision trees, support vector machine, and neural network models. We also examined whether 13 baseline variables that included a variety of demographics (eg, race and ethnicity, gender, and age) and smoking characteristics (eg, use of e-cigarettes and confidence in being smoke free) might improve this prediction. Results The AUC for each logistic regression model using only the first 7 days of log-in count variables was 0.94 (95% CI 0.90-0.97) for iCanQuit, 0.88 (95% CI 0.83-0.93) for QuitGuide, 0.85 (95% CI 0.80-0.88) for WebQuit.org, and 0.60 (95% CI 0.54-0.66) for Smokefree.gov. Replacing logistic regression models with more complex decision trees, support vector machines, or neural network models did not significantly increase the AUC, nor did including additional baseline variables as predictors. The sensitivity and specificity were generally good, and they were excellent for iCanQuit (ie, 0.91 and 0.85, respectively, at the 0.5 classification threshold). Conclusions Logistic regression models using only the first 7 days of log-in count data were generally good at predicting early dropouts. These models performed well when using simple, automated, and readily available log-in count data, whereas including self-reported baseline variables did not improve the prediction. The results will inform the early identification of people at risk of early dropout from digital health interventions with the goal of intervening further by providing them with augmented treatments to increase their retention and, ultimately, their intervention outcomes.

Evaluating the impact of motivational interviewing on engagement and outcomes in a web-based self-help intervention for gambling disorder: A randomised controlled trial

“…Nearly all participants had terminated program use by that time as well, whether by completing all modules, deciding not to return, or forgetting to log in. Similar findings have been observed with other mental health and addiction interventions ( Bricker et al, 2023 ), whereby differential trajectories of treatment engagement and behavioral change are most prominent in the early days and weeks of treatment. New technologies that permit much higher sampling frequencies with convenience and ease (e.g., smartphone apps with daily diaries) will be extremely useful to zoom in on recovery dynamics at previously inaccessible time scales.…”

Evaluating Barriers and Facilitators to the Uptake of mHealth Apps in Cancer Care Using the Consolidated Framework for Implementation Research: Scoping Literature Review

“…A final number of 91 studies were included for analysis. Figure 1 describes the PRISMA flowchart [ 36 , 37 ].…”

“…However, there is limited evidence on the impact of the uptake and use of mHealth apps within the clinical setting. DHTs, including mHealth, are challenged by the phenomenon of early dropouts and abandonment [ 36 ]. To date, the implementation of mHealth apps has been analyzed less extensively.…”

Leveraging Emerging Technologies to Expand Accessibility and Improve Precision in Rehabilitation and Exercise for People with Disabilities

“…Furthermore, the increasing affordability and quality of real-time video streaming and video conferencing has also expanded the capabilities of tele-rehabilitation by enabling the live streaming of video content and allowing patients to interact with their healthcare providers through real-time two-way video conferencing in a secure, interactive environment [90,91]. While tele-rehabilitation delivered through various methods has demonstrated immense potential to make a substantial impact on health and function in people with disabilities, the available evidence also suggests that the effectiveness of tele-rehabilitation interventions may be limited by poor adherence and a lack of clinical supervision [57,59,61,64,65,69,75,[79][80][81][82]92]. Undoubtedly, corrective and encouraging feedback is critical to the safety and effectiveness of rehabilitation, but this is difficult to provide during tele-rehabilitation sessions where direct clinical supervision is limited or absent altogether.…”