Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps

Abstract

BackgroundThere are over 165,000 mHealth apps currently available to patients, but few have undergone an external quality review. Furthermore, no standardized review method exists, and little has been done to examine the consistency of the evaluation systems themselves.ObjectiveWe sought to determine which measures for evaluating the quality of mHealth apps have the greatest interrater reliability.MethodsWe identified 22 measures for evaluating the quality of apps from the literature. A panel of 6 reviewers reviewed the top 10 depression apps and 10 smoking cessation apps from the Apple iTunes App Store on these measures. Krippendorff’s alpha was calculated for each of the measures and reported by app category and in aggregate.ResultsThe measure for interactiveness and feedback was found to have the greatest overall interrater reliability (alpha=.69). Presence of password protection (alpha=.65), whether the app was uploaded by a health care agency (alpha=.63), the number of consumer ratings (alpha=.59), and several other measures had moderate interrater reliability (alphas>.5). There was the least agreement over whether apps had errors or performance issues (alpha=.15), stated advertising policies (alpha=.16), and were easy to use (alpha=.18). There were substantial differences in the interrater reliabilities of a number of measures when they were applied to depression versus smoking apps.ConclusionsWe found wide variation in the interrater reliability of measures used to evaluate apps, and some measures are more robust across categories of apps than others. The measures with the highest degree of interrater reliability tended to be those that involved the least rater discretion. Clinical quality measures such as effectiveness, ease of use, and performance had relatively poor interrater reliability. Subsequent research is needed to determine consistent means for evaluating the performance of apps. Patients and clinicians should consider conducting their own assessments of apps, in conjunction with evaluating information from reviews.

Understanding the quality, effectiveness and attributes of top-rated smartphone health apps

“…A Canadian-based effort that gathered diverse mental health stakeholders, including patients, noted that they were not able to find a single scale or framework that adequately addressed their needs 21. Prior efforts to create repositories of higher quality apps have failed at the national level22 when closely scrutinised and other efforts to score apps by features have also been found to be unreliable23 when closely examined. Furthermore, several of our findings have been supported by the software engineering literature as well, including the correlation between days since last update and app ratings and the number of unsupported claims made by apps 14.…”

Novel Health Mobile Technology as an Emerging Strategy in Diabetes Management

“…Presently about 165,000 mHealth apps are available [6]. Various dedicated mHealth applications have been developed for the chronic diseases management recently.…”

Free mobile apps on depression for Indian users: A brief overview and critique

“…The use of mobile platforms for delivery of self-help interventions specifically in the field of mental health has also seen phenomenal growth across the globe in the last few years; though several of them are yet to be formally evaluated (Huguet et al, 2016;Savic et al, 2013). Although the utility of mobile apps in resource-scarce settings is being recognized; the need for determining consistent ways of evaluating performance of mental health apps has also been emphasized (Powell et al, 2016). Several risks and ethical issues have been highlighted in the use of mobile apps for mental health (Giota and Kleftaras, 2014;Rozental et al, 2014).…”