Evaluation Criteria for Weight Management Apps: Validation Using a Modified Delphi Process (Preprint)

Abstract

BACKGROUND The use of Mobile Applications (apps) for managing overweightness/obesity has increased over recent years. However, there is a lack of evidence on the efficacy, effectiveness and safety of these apps. The EVALAPPS project will develop and validate an assessment instrument to specifically assess these dimensions of overweightness/obesity management apps. OBJECTIVE To reach a consensus among stakeholders on a comprehensive set of criteria to guide development of the EVALAPPS assessment instrument. A modified Delphi process was used in order to: 1) verify the robustness of the criteria identified through a literature review; 2) prioritize a set of the identified criteria. METHODS 31 stakeholders were invited to participate in a 2 round Delphi process with an initial number of 114 criteria identified from the literature. In Round 1 participants rated criteria according to relevance (0 = “I suggest this criterion is excluded”, 5 = “This criterion is extremely relevant”). A criterion was accepted if a median value equal or greater than 4 and RIQR equal or less than 0.67 was reached. In Round 2 participants were asked about criteria discarded in round 1. The prioritization strategy was based on identifying those indicators categorized as Crucial in Round 1 according to: a) the importance attributed by participants (criteria with a mean value ≥ 4.00); and, b) the level of agreement (criteria scoring almost 4-5 by 80% of the participants). RESULTS The response rate was 84% in Round 1, 90% in Round 2. A total number of 107 criteria were accepted by consensus (94%), 105 in Round 1 and 2 in Round 2. After the prioritization strategy, 53 criteria were deemed Crucial. These related mainly to the Security and Privacy (24.5%) and Usability (17.0%) dimensions, followed by Health indicators: Activity data, Clinical Effectiveness and Reliability (9.4% each). CONCLUSIONS Results confirmed the robustness of the identified criteria, with those relating to Security and Privacy being deemed most relevant by stakeholders. Additionally, a specific set of criteria based on Health indicators was also prioritized (taken together, 20.7%). CLINICALTRIAL

A Proposal for a Robust Validated Weighted General Data Protection Regulation-Based Scale to Assess the Quality of Privacy Policies of Mobile Health Applications: An eDelphi Study

“…Tentative items would be eventually added to the scale if, at least, 80% of the experts rated the item as either 4 or 5. The use of this criteria for inclusion has been widely used in literature 37,52,53 . Compulsory items are considered robust with the same criteria, although they should not be eliminated from the scale as, otherwise, privacy policy would not comply with the GDPR.…”

“…Quantifying the degree of consensus among the experts is an important component for performing a good Delphi data analysis and interpretation. In this study, we have used the interquartile interval (IQR) as a measure of the deviation of the opinion of an expert from the opinion of the whole panel 49,52 . A suitable IQR-based criterion to determine that there is a consensus among the experts is that IQR value is equal or less than 1 for a 5-point Likert scale 55 .…”