Usability Evaluations of Mobile Mental Health Technologies: Systematic Review

Abstract

Background Many mobile health (mHealth) apps for mental health have been made available in recent years. Although there is reason to be optimistic about their effect on improving health and increasing access to care, there is a call for more knowledge concerning how mHealth apps are used in practice. Objective This study aimed to review the literature on how usability is being addressed and measured in mHealth interventions for mental health problems. Methods We conducted a systematic literature review through a search for peer-reviewed studies published between 2001 and 2018 in the following electronic databases: EMBASE, CINAHL, PsycINFO, PubMed, and Web of Science. Two reviewers independently assessed all abstracts against the inclusion and exclusion criteria, following the Preferred Reporting Items for Systematic Review and Meta-Analysis guidelines. Results A total of 299 studies were initially identified based on the inclusion keywords. Following a review of the title, abstract, and full text, 42 studies were found that fulfilled the criteria, most of which evaluated usability with patients (n=29) and health care providers (n=11) as opposed to healthy users (n=8) and were directed at a wide variety of mental health problems (n=24). Half of the studies set out to evaluate usability (n=21), and the remainder focused on feasibility (n=10) or acceptability (n=10). Regarding the maturity of the evaluated systems, most were either prototypes or previously tested versions of the technology, and the studies included few accounts of sketching and participatory design processes. The most common reason referred to for developing mobile mental health apps was the availability of mobile devices to users, their popularity, and how people in general became accustomed to using them for various purposes. Conclusions This study provides a detailed account of how evidence of usability of mHealth apps is gathered in the form of usability evaluations from the perspective of computer science and human-computer interaction, including how users feature in the evaluation, how the study objectives and outcomes are stated, which research methods and techniques are used, and what the notion of mobility features is for mHealth apps. Most studies described their methods as trials, gathered data from a small sample size, and carried out a summative evaluation using a single questionnaire, which indicates that usability evaluation was not the main focus. As many studies described using an adapted version of a standard usability questionnaire, there may be a need for developing a standardized mHealth usability questionnaire.

Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults

“…Half of the reviews included in this scoping review did not refer to the characteristics of the participants included in the primary studies reviewed. Of the reviews, 50% (10/20) reviews that reported on any of the participants' characteristics, 4 reported mean age or age range [ 36 , 41 , 46 , 49 ], 4 reported the gender of participants [ 36 , 41 , 44 , 46 ], 8 reported the sample size [ 35 , 36 , 39 , 41 , 42 , 46 , 47 , 49 ], and 7 reported on other characteristics of participants by describing them as healthy participants or as having a specific clinical condition [ 36 , 37 , 39 , 41 , 44 , 46 , 49 ]. Nevertheless, 20% (4/20) reviews that reported the age of the participants also reported that not all primary studies detailed such information.…”

“…Of the 20 systematic reviews included, only 3 (15%) [ 33 , 40 , 41 ] did not refer to the methods and techniques of usability used. Among the inquiry methods, the questionnaires/scales (15/20, 75%) and interviews (12/20, 60%) were most commonly reported.…”

“…Among scales/questionnaires, which constitute the technique most often reported, the most common usability assessment scales were the System Usability Scale [ 29 , 32 , 41 - 43 , 46 , 47 ] and the Post-Study System Usability Questionnaire [ 41 , 42 , 46 , 47 ]. The other scales/questionnaires include the Questionnaire for User Interaction Satisfaction [ 29 , 42 , 47 ], the Software Usability Measurement Inventory [ 32 , 42 ], the Usefulness, Satisfaction, and Ease of use Questionnaire [ 32 , 41 ], the Computer System Usability Questionnaire [ 32 , 47 ], the After-Scenario Questionnaire [ 46 , 47 ], the Perceived Useful and Ease of Use [ 32 ], the IsoMetrics usability inventory [ 32 ], the Health Information Technology Usability Evaluation Scale [ 32 ], the user Mobile Application Rating Scale [ 32 ]; the IBM ease of use [ 42 ], and the ISO 9241–11 Questionnaire [ 43 ]. In addition, several reviews have reported the use of nonvalidated questionnaires [ 32 , 41 , 43 , 46 ].…”

“…The other scales/questionnaires include the Questionnaire for User Interaction Satisfaction [ 29 , 42 , 47 ], the Software Usability Measurement Inventory [ 32 , 42 ], the Usefulness, Satisfaction, and Ease of use Questionnaire [ 32 , 41 ], the Computer System Usability Questionnaire [ 32 , 47 ], the After-Scenario Questionnaire [ 46 , 47 ], the Perceived Useful and Ease of Use [ 32 ], the IsoMetrics usability inventory [ 32 ], the Health Information Technology Usability Evaluation Scale [ 32 ], the user Mobile Application Rating Scale [ 32 ]; the IBM ease of use [ 42 ], and the ISO 9241–11 Questionnaire [ 43 ]. In addition, several reviews have reported the use of nonvalidated questionnaires [ 32 , 41 , 43 , 46 ]. One review reported that 26% of the included studies used a remote assessment of usability, where participants are in an uncontrolled environment [ 31 ].…”

Musings on methods and modalities in the study and care of older adults with serious mental illness

“…Many have been incorporated into selfhealth programs and are most often used alone, but sometimes with other people. Apart from studies of usability, there have been few controlled studies of their clinical benefits, and a majority have been of very short duration and involved younger and mixed age samples (Aref-Adib et al, 2019;Inal et al, 2020;Rathbone and Prescott, 2017). Rathbone and Prescott's (2017) review found only four studies of persons with schizophrenia; a more recent review by Aref-Adib et al (2019) identified 26 studies using digital interventions that included persons with bipolar illness or schizophrenia.…”