Proportionate methods for evaluating a simple digital mental health tool

Abstract

backgroundTraditional evaluation methods are not keeping pace with rapid developments in mobile health. More flexible methodologies are needed to evaluate mHealth technologies, particularly simple, self-help tools. One approach is to combine a variety of methods and data to build a comprehensive picture of how a technology is used and its impact on users. Objective This paper aims to demonstrate how analytical data and user feedback can be triangulated to provide a proportionate and practical approach to the evaluation of a mental well-being smartphone app (In Hand). Methods A three-part process was used to collect data: (1) app analytics; (2) an online user survey and (3) interviews with users. Findings Analytics showed that >50% of user sessions counted as 'meaningful engagement'. User survey findings (n=108) revealed that In Hand was perceived to be helpful on several dimensions of mental well-being. Interviews (n=8) provided insight into how these self-reported positive effects were understood by users. conclusions This evaluation demonstrates how different methods can be combined to complete a real world, naturalistic evaluation of a self-help digital tool and provide insights into how and why an app is used and its impact on users' well-being. clinical implications This triangulation approach to evaluation provides insight into how well-being apps are used and their perceived impact on users' mental well-being. This approach is useful for mental healthcare professionals and commissioners who wish to recommend simple digital tools to their patients and evaluate their uptake, use and benefits.

A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review

“…Of the analytic indicators identified in this review, the number of measures recorded by users on an app was the most commonly used indicator of engagement with mHealth apps for chronic conditions. Researchers evaluated a range of measures that aligned with their target chronic condition, such as blood glucose [56,60,61,64,73], weight [56,73,74], symptoms [66,68,69], patient-reported outcomes [38,46,52,65,71], diary entries [47,66], and steps [51]. There was some overlap in the types of measures being collected across apps targeting the same chronic conditions, such as the number of blood glucose readings recorded as an indicator of engagement with diabetes apps.…”

“…Researchers often chose to complement assessing the number of measures recorded on an app with the frequency by which the measures were recorded. Stratifying frequency of interactions by specific date ranges was also common; Davies et al measured the number of users who used a mental health app at least once after 1 week, 4 weeks, and 20 weeks [38]. They also applied within-date range indicators such as the number of users who used the app once, 2 to 3 times, 4 to 6 times, or 6 or more times per week.…”

“…The amount of time that users engaged with an app was considered by a subset of researchers to be an analytic indicator of engagement. Researchers measured the time spent on unique sections of an app [66], the time spent on unique pages [56], the length of a unique session [38,42,43,71], the length between unique sessions [72], and the total time spent on an app [62,68,73]. Davies et al also segmented sessions by those that were in the 30- to 60-second range [38].…”

“…Researchers measured the time spent on unique sections of an app [66], the time spent on unique pages [56], the length of a unique session [38,42,43,71], the length between unique sessions [72], and the total time spent on an app [62,68,73]. Davies et al also segmented sessions by those that were in the 30- to 60-second range [38]. Measuring time spent engaging with an app helped researchers to distinguish between exploratory and purposeful engagement; a rapid succession of short page views was indicative of scanning through content, whereas prolonged viewing suggested greater intention and interest in content.…”

Potential of using visual imagery to revolutionise measurement of emotional health

“…The children and young people consulted added, ‘more able to take control’, ‘ready to talk to someone else’ and ‘less stressed’ as meaningful dimensions. The modified scale was then included in a survey of app users with the goal of enabling individuals to better express whether and how the app was helpful in improving mental health 8. Therefore, we can see from recent perspectives of children and young people that traditional methods are already being adapted by them as they are not deemed necessarily best or most comprehensive ways of capturing emotional health in children and young people of today.…”