A Systematic Review of User Engagement with Digital Mental Health Interventions and Proposed Reporting Guidelines for Randomized Controlled Trials (Preprint)

Abstract

BACKGROUND Digital mental health interventions (DMHIs) have been indicated as efficacious in clinical trial settings. However, in the wake of several large implementation studies, patient engagement with these interventions has emerged as a concern. In order to address engagement, we must first understand what standard engagement levels are in the context of randomized controlled trials (RCTs) and how these compare to other treatments. OBJECTIVE This systematic review aimed to synthesize data on intervention engagement in RCTs of mobile-application-based interventions intended to treat symptoms of depression. METHODS We conducted a systematic review of RCTs that evaluated use of a mobile-application-based intervention in adults with depression, for which depressive symptoms were a primary outcome of interest. We then extracted two kinds of information from each article: intervention details and indices of participant DMHI engagement. Key DMHI engagement data extracted included: (a) recommended intervention use as communicated to participants at enrollment; (b) method of app usage monitoring; (c) rate of intervention uptake among those assigned to the intervention; (d) level of app use metrics reported (i.e., number of uses and time spent using the app); (e) duration of app use metrics (i.e., weekly use patterns); (f) and number of intervention completers. RESULTS The systematic search retrieved 3137 results. 22 independent studies were eligible for inclusion. Within included studies, 13 evaluated an app intended to be used as a daily self-management/skill building tool; 5 evaluated an app intended to provide support in the context of clinician-administered care or to facilitate communication with clinicians; and 4 evaluated an app involving a discrete number of lessons/modules typically to be completed on a weekly basis. Only 64% of studies included in this review specified rate of uptake, defined as the number of participants randomized to the intervention condition who used the app at least once. Level-of-use metrics were also only reported in 64% of the studies reviewed (though not directly overlapping those reporting uptake). Approximately a quarter of studies (23%) reported duration-of-use metrics. Only half (50%) of studies reported the number of participants considered to have completed the app-based components of the intervention as intended or other metrics related to completion. CONCLUSIONS A number of basic metrics of intervention engagement were not routinely reported in RCTs included in this review. Such variability makes it impossible to draw conclusions about standard patient engagement levels with DMHIs. Additionally, this variability in reporting suggests a failure to establish sufficient reporting standards. Based on these findings, we suggest a five-element set of reporting guidelines of minimum necessary information when publishing RCTs of DMHIs. These include: (a) intervention instructions and retention criteria; (b) rate of uptake; (c) level of use; (d) duration of use; and (e) number of completers.

Effective recruitment or bot attack? The challenge of online research surveys and recommendations to reduce risk and improve robustness: a case study commentary (Preprint)

“…Removing bot data is important for research integrity, but this should be done cautiously so as not to remove data belonging to valid participants. This is particularly valid when considering removing data with incomplete answers (particularly qualitative answers), as this may result in removing data from people who may struggle with literacy but also misrepresent attrition and engagement -both critical issues in digital interventions [39][40][41]. Data removal should be done by two independent, blinded researchers and disclosed in publications to ensure transparency and rigour and decrease biases.…”