Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps

Abstract

ObjectivesDespite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks.DesignWe conducted a scoping review of mobile health app evaluation frameworks.Data sourcesReferences were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020.Eligibility criteriaPapers were selected for inclusion if they meet the predetermined eligibility criteria—presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions.Data extraction and synthesisTwo reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility—as well as any potential gaps—of MIND.ResultsNew app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today.ConclusionAs the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.

Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management

“…Secondly, there is no consensus approach to evaluating mHeath apps (Torous et al 2018 ). A recent scoping review found the MIND categories and questions largely overlap with content encompassed in other existing app evaluation frameworks, suggesting good coherence (Lagan et al 2021 ); however, some aspects of user experience (e.g., usability and aesthetic appeal) are not addressed. It has been suggested that the subjectivity inherent in rating such aspects is a likely source of inconsistencies in app evaluations (Carlo et al 2019 ).…”

Designing Disease-Specific mHealth Apps for Clinical Value

“…Many organizations and academics have developed criteria for evaluating apps, including the author of this paper [13,17,[19][20][21]. However, no matter how comprehensive the criteria, they are both aspirational, incomplete and highly generic.…”

Psychiatry in the Digital Age: A Blessing or a Curse?

“…More than 200 healthcare applications (apps) for smartphones and tablets are uploaded daily to Apple's App Store and Google Play [60] (p. 93). Approximately 300,000 health apps are currently available, of which 10,000 to 20,000 are mental health apps [61,62]. These publicly available apps are opening further opportunities to provide mobile healthcare (mHealth) anytime, anywhere.…”

“…This need has led to the development of app-rating guidelines, such as the Mobile App Rating Scale (MARS) and the American Psychiatric Association's app evaluation framework [83,84]. Unfortunately, the increasing number of app evaluation frameworks makes it difficult for clinicians and patients to select an appropriate evaluation framework and find an appropriate mental health app [62]. Therefore, app-rating platforms, such as PsyberGuide, have gone one step further by identifying, describing, and rating publicly available mental health apps [85].…”