How Notifications Affect Engagement With a Behavior Change App: Results From a Micro-Randomized Trial (Preprint)

Abstract

UNSTRUCTURED Push notifications policies can be developed to prompt engagement with a behaviour change app. An optimal policy would send the right support, to the right person at the right moment, while avoiding states of disengagement due to the burden of less helpful notifications over time. The Micro-Randomized Trial (MRT) is an experimental design in which users are repeatedly randomized to notifications many times, for the purpose of developing such optimal policies. Drink Less is a behaviour change app to help higher risk drinkers in the UK reduce their alcohol consumption. To target users’ motivation and perceived usefulness of the app, we developed a bank of 30 new evidence-informed notifications. Our MRT randomised 350 users to test if receiving a notification at 8PM, compared to receiving no notification, increased the probability of opening the app in the subsequent hour, over the first 30 days since downloading Drink Less. Our secondary objectives included comparing the effect of a new message with the standard message of “Please complete your mood and drinks diary” and effect moderation over time. To understand time-to-disengagement, two additional randomised arms were added to complement the MRT, with 98 additional users randomised to receive no notification and 121 users randomised to receive the standard notification daily at 11 am. Ancillary analyses explored effect moderation by recent states of habituation, defined as ‘did the user receive a notification the day before?’ and already engaged, defined as ‘did the user open the app between 8PM-9PM the day before?’. Receiving a notification, compared with not, increased the probability of opening the app in the next hour 3.5-fold (95% confidence interval (CI) 2.91, 4.25). Both message types were equivalently effective. Receiving a new message, compared with receiving no message, led to a 3.4-fold (95% CI 2.77, 4.13) increase in engagement, and receiving the standard message, compared with receiving no notification, also led to a 3.7-fold (95% CI 2.99, 4.49) increase. The effect of the notification did not change significantly over time. The multiplicative effect of a user being ‘already engaged’ non-significantly lowered the new notification effect by 0.80 (95% CI 0.55, 1.16). Time-to-disengagement, between the three arms, was not significantly different. Notifications are powerful tools to increase ‘in-the-moment’ engagement. However, to improve longer-term engagement, researchers should carefully consider tailoring notifications to an individual’s recent history of engagement. Sending a notification to encourage engagement, only when users are at a greater risk of disengagement, may reduce burden, and could keep users engaged for longer.

Methods for Evaluating the Efficacy and Effectiveness of Direct-to-Consumer Mobile Health Apps: A Scoping Review

“…A wide variety of study designs exist to evaluate digital health products [21,24,25], but it is not always clear which would be optimal in different contexts, with a range of different frameworks available [22]. Additionally, alternative approaches are being considered, like microrandomised trials, but uptake may be slow [20,[26][27][28].…”

Methods for evaluating the efficacy and effectiveness of direct-to-consumer mobile health apps: a scoping review

“…A wide variety of study designs exist to evaluate digital health products [21,24,25], but it is not always clear which would be optimal in different contexts, with a range of different frameworks available [22]. Additionally, alternative approaches are being considered, like micro-randomised trials, but uptake may be slow [20,[26][27][28].…”