Evaluation Criteria for Weight Management Apps: Validation Using a Modified Delphi Process

Abstract

Background The use of apps for weight management has increased over recent years; however, there is a lack of evidence regarding the efficacy and safety of these apps. The EVALAPPS project will develop and validate an assessment instrument to specifically assess the safety and efficacy of weight management apps. Objective The aim of this study was to reach a consensus among stakeholders on a comprehensive set of criteria to guide development of the EVALAPPS assessment instrument. A modified Delphi process was used in order to verify the robustness of the criteria that had been identified through a literature review and to prioritize a set of the identified criteria. Methods Stakeholders (n=31) were invited to participate in a 2-round Delphi process with 114 initial criteria that had been identified from the literature. In round 1, participants rated criteria according to relevance on a scale from 0 (“I suggest this criterion is excluded”) to 5 (“This criterion is extremely relevant”). A criterion was accepted if the median rating was 4 or higher and if the relative intraquartile range was equal to 0.67 or lower. In round 2, participants were asked about criteria that had been discarded in round 1. A prioritization strategy was used to identify crucial criteria according to (1) the importance attributed by participants (criteria with a mean rating of 4.00 or higher), (2) the level of consensus (criteria with a score of 4 or 5 by at least 80% of the participants). Results The response rate was 83.9% (26/31) in round 1 and 90.3% (28/31) in round 2. A total of 107 out of 114 criteria (93.9%) were accepted by consensus—105 criteria in round 1 and 2 criteria in round 2. After prioritization, 53 criteria were deemed crucial. These related mainly to the dimensions of security and privacy (13/53, 24.5%) and usability (9/53, 17.0%), followed by activity data (5/53, 9.4%), clinical effectiveness (5/53, 9.4%), and reliability (5/53, 9.4%). Conclusions Results confirmed the robustness of the criteria that were identified, with those relating to security and privacy being deemed most relevant by stakeholders. Additionally, a specific set of criteria based on health indicators (activity data, physical state data, and personal data) was also prioritized.

A Proposal for a Robust Validated Weighted General Data Protection Regulation-based Scale to Assess the Fairness of Privacy Policies of Mobile Health Applications: a Delphi Study

“…Tentative items may be eventually added to the scale if, at least, 80% of the experts rated the item as either 4 or 5 [43]. Compulsory items are considered robust with the same criteria, although they should not be eliminated from the scale if it is desired to comply with the GDPR.…”

“…Quantifying the degree of consensus among the experts is an important component for performing a good Delphi data analysis and interpretation. In this study, we have used the interquartile interval (IQR) as a measure of the deviation of the opinion of an expert from the opinion of the whole panel [43,44]. A suitable IQR-based criterion to determine that there is a consensus among the experts is that IQR value is equal or less than 1 for a 5-point Likert scale [45].…”

Codesign and Feasibility Testing of a Tool to Evaluate Overweight and Obesity Apps

“…The current study consists of two stages aligned with both study aims: developing the format and sections of the EVALAPPS tool-the content was previously defined [8,9]-and pilot testing. In the tool development stage, a prototype of the EVALAPPS tool as an app was developed, and in the second stage, it was tested.…”

“…The main objective of the EVALAPPS project is to design and validate a tool to assess the safety and effectiveness of health-related apps whose main goal is to manage and prevent overweight conditions and obesity. Previous phases of the project developed the content of the tool through (a) a systematic review to identify potential effectiveness criteria already used in former validation studies [8] and (b) a Delphi study to achieve consensus among a broad group of stakeholders on a comprehensive set of criteria to guide the development of the EVALAPPS assessment tool and to prioritize them [9]. While the systematic review aimed to consider the content of already available tools used for mobile health solutions assessment, the Delphi process, in which 31 stakeholders participated, generated a list of 53 criteria (from 114) that were considered essential for the EVALAPPS evaluation tool.…”

Investigating the Affordances of Wellness mHealth Apps

“…Some of these apps are designed to manage chronic diseases or to treat a patient [3], while others are developed to promote healthy behaviors to improve users' wellness [27]. Wellness mHealth apps help to enhance users' well-being by encouraging and tracking physical activity [27], weight management [28], dietary interventions [30], stress management [30], and more. Wellness mHealth apps may vary in their features and designs, but they share one ultimate goal, which is to improve users' well-being.…”