Evaluating Digital Health Interventions

Abstract

Digital health interventions (DHI) have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety and personalisation. Achieving these improvements requires a cumulative knowledge base to inform development and deployment of DHI. However, evaluations of DHI present special challenges. This paper aims to examine these challenges and outline an evaluation strategy in terms of the Research Questions (RQs) needed to appraise DHIs. As DHI are at the intersection of biomedical, behavioural, computing and engineering research, methods drawn from all these disciplines are required. Relevant RQs include defining the problem and the likely benefit of the DHI, which in turn requires establishing the likely reach and uptake of the intervention, the causal model describing how the intervention will achieve its intended benefit, key components and how they interact with one another, and estimating overall benefit in terms of effectiveness, cost-effectiveness and harms. While Randomised Controlled Trials (RCTs) are important for evaluation of effectiveness and cost-effectiveness, they are best undertaken only when: a) the intervention and its delivery package are stable; b) these can be implemented with high fidelity and c) there is a reasonable likelihood that the overall benefits will be clinically meaningful (improved outcomes or equivalent outcomes at less cost). Broadening the portfolio of RQs and evaluation methods will help with developing the necessary knowledge base to inform decisions on policy, practice and research.

Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions

“…33 Recent advances in sequence analysis, data mining, and novel visualization tools are facilitating analyses of usage patterns, and there is scope for substantial progress in this field. 23 DBCIs have the potential to generate data sets sufficiently large to be able to reliably model and experimentally test 34 mediation of outcomes by engagement with particular intervention components and to statistically control for confounding moderator effects such as baseline motiva-tion levels. 22,26,35,36 Importantly, usage metrics can be collated with data on users' behavior collected by smartphone sensors, such as movement or location.…”

“…The novel research designs that can support these analyses are discussed in companion papers in this issue. 15,34,38 Psychophysiological measurements, ranging from skin conductance and heart rate to facial expression or fMRI, have been used to measure users' task engagement. 39 Such measures can help identify aspects of the inter-vention that attract attention or evoke emotional arousal, suggesting mechanisms through which DBCI content or design impact shortterm engagement.…”

Designing and Undertaking a Health Economics Study of Digital Health Interventions

“…These recommendations appear salient for evaluation of DHIs. For example, there is already recognition that RCTs are not always appropriate as a means to establish effectiveness, 5 and a similar argument holds for evaluation of cost effectiveness.…”

Methods to Evaluate the Effects of Internet-Based Digital Health Interventions for Citizens: Systematic Review of Reviews (Preprint)

“…Digital health interventions have enormous potential as scalable tools to support better health and healthcare delivery by improving a number of different outcomes such as effectiveness, efficiency, accessibility, safety and personalisation [11]. Although evidence of the potential benefit of digital health for improving care delivery and patient outcomes has been described [8], numerous factors can affect patient and public engagement in using digital health interventions such as lack of motivation, busy lifestyle, poor digital literacy, complexity and usability [2].…”

“…Although evidence of the potential benefit of digital health for improving care delivery and patient outcomes has been described [8], numerous factors can affect patient and public engagement in using digital health interventions such as lack of motivation, busy lifestyle, poor digital literacy, complexity and usability [2]. Other difficulties include the rapid change of technology, which requires digital health interventions to constantly evolve and be updated [11].…”