A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint

Abstract

Trials of digital interventions can yield extensive, in-depth usage data, yet usage analyses tend to focus on broad descriptive summaries of how an intervention has been used by the whole sample. This paper proposes a novel framework to guide systematic, fine-grained usage analyses that better enables understanding of how an intervention works, when, and for whom. The framework comprises three stages to assist in the following: (1) familiarization with the intervention and its relationship to the captured data, (2) identification of meaningful measures of usage and specifying research questions to guide systematic analyses of usage data, and (3) preparation of datasheets and consideration of available analytical methods with which to examine the data. The framework can be applied to inform data capture during the development of a digital intervention and/or in the analysis of data after the completion of an evaluation trial. We will demonstrate how the framework shaped preparation and aided efficient data capture for a digital intervention to lower transmission of cold and flu viruses in the home, as well as how it informed a systematic, in-depth analysis of usage data collected from a separate digital intervention designed to promote self-management of colds and flu. The Analyzing and Measuring Usage and Engagement Data (AMUsED) framework guides systematic and efficient in-depth usage analyses that will support standardized reporting with transparent and replicable findings. These detailed findings may also enable examination of what constitutes effective engagement with particular interventions.

User Engagement Among Diverse Adults in a 12-Month Text Message–Delivered Diabetes Support Intervention: Results from a Randomized Controlled Trial

“…There is growing evidence for the efficacy of text message–delivered interventions to improve adherence and clinical outcomes [ 11 ], and this study addresses important gaps by examining long-term engagement and associations between engagement and participant and intervention characteristics. As the literature on engagement is growing, there have been calls for more standardized engagement reporting to compare results across studies and draw more concrete conclusions about the role of engagement [ 44 , 45 ]. Kelders and Kip [ 46 ] recently developed a self-report scale to capture multiple components of engagement with health technologies (ie, behavior, cognition, and affect).…”

Understanding Self-Guided Web-Based Educational Interventions for Patients With Chronic Health Conditions: Systematic Review of Intervention Features and Adherence

“…Intended usage is estimated by the developers and refers to the usage level needed to have the maximum benefit from the intervention (eg, clinical outcomes), and defining the intended usage would allow for standardization in the calculation of adherence [ 12 ]. Although Kelders et al [ 12 ] used the term intended usage, others adopted the term effective engagement [ 68 , 69 ], defined as "sufficient engagement with the intervention to achieve intended outcomes" [ 69 ]. As both terminologies focused on the identification of the parameters and the related minimum threshold that can have an impact on the intended behavior [ 12 , 68 , 69 ], these terms were used interchangeably.…”

“…Effective engagement should reflect the multidimension of the intervention in relation to the primary outcome, and both objective and subjective measurements should be evaluated [ 70 ]. The back-ended intervention usage data are considered an objective measurement [ 70 ] and can be assessed by using the Analyzing and Measuring Usage and Engagement Data framework [ 68 ]. This framework is designed for web-based interventions and can be used during the intervention development phase or after data collection.…”

“…In stage 1, the usage of data is classified into 3 categories: intervention characteristics (eg, architecture and content), accrued data (eg, data collected during the use of the intervention), and contextual data (eg, factors influencing the use of the intervention). Stage 2 consists of the selection of meaningful measures of usage and generation of research questions related to the primary outcome, usage data collected, and characteristics of the target population (eg, a web-based intervention focusing on the reduction of hospital visits can have “Will the number of content views be associated with hospital visits?” as a research question [ 68 ]). The final stage focuses on the selection of analytical tools and data preparation.…”

Engagement with a Web-Based Health Promotion Intervention among Vocational School Students: A Secondary User and Usage Analysis

“…The scientific discussion on the operationalization of engagement and the selection of use metrics is ongoing [35,94,95]. Since we have taken logins as a meaningful measure for our secondary post-hoc usage analysis [96] to get a general overview of the participation, we cannot draw any conclusions about individual experience. Against the background of the known high drop-out rates, in-depth analyses of the perception and usability of web-based measures for potential users are necessary.…”