Prioritization of Quality Principles for Health Apps Using the Kano Model: Survey Study

Abstract

Background Health apps are often used without adequately taking aspects related to their quality under consideration. This may partially be due to inadequate awareness about necessary criteria and how to prioritize them when evaluating an app. Objective The aim of this study was to introduce a method for prioritizing quality attributes in the mobile health context. To this end, physicians were asked about their assessment of nine app quality principles relevant in health contexts and their responses were used as a basis for designing a method for app prioritization. Ultimately, the goal was to aid in making better use of limited resources (eg, time) by assisting with the decision as to the specific quality principles that deserve priority in everyday medical practice and those that can be given lower priority, even in cases where the overall principles are rated similarly. Methods A total of 9503 members of two German professional societies in the field of orthopedics were invited by email to participate in an anonymous online survey over a 1-month period. Participants were asked to rate a set of nine app quality principles using a Kano survey with functional and dysfunctional (ie, positively and negatively worded) questions. The evaluation was based on the work of Kano (baseline), supplemented by a self-designed approach. Results Among the 9503 invited members, 382 completed relevant parts of the survey (return rate of 4.02%). These participants were equally and randomly assigned to two groups (test group and validation group, n=191 each). Demographic characteristics did not significantly differ between groups (all P>.05). Participants were predominantly male (328/382, 85.9%) and older than 40 years (290/382, 75.9%). Given similar ratings, common evaluation strategies for Kano surveys did not allow for conclusive prioritization of the principles, and the same was true when using the more elaborate approach of satisfaction and dissatisfaction indices following the work of Timko. Therefore, an extended, so-called “in-line-of-sight” method was developed and applied for this evaluation. Modified from the Timko method, this approach is based on a “point of view” (POV) metric, which generates a ranking coefficient. Although the principles were previously almost exclusively rated as must-be (with the exception of resource efficiency), which was not conducive to their prioritization, the new method applied from the must-be POV resulted in identical rankings for the test and validation groups: (1) legal conformity, (2) content validity, (3) risk adequacy, (4) practicality, (5) ethical soundness, (6) usability, (7) transparency, (8) technical adequacy, and (9) resource efficiency. Conclusions Established survey methodologies based on the work of Kano predominantly seek to categorize the attributes to be evaluated. The methodology presented here is an interesting option for prioritization, and enables focusing on the most important criteria, thus saving valuable time when reviewing apps for use in the medical field, even with otherwise largely similar categorization results. The extent to which this approach is applicable beyond the scenario presented herein requires further investigation.

Orthopaedic and trauma surgeons’ prioritisation of app quality principles based on their demographic background

“…Answer options for both types of questions were “I would be very pleased”, “I’d expect this”, “I don’t care”, “I could accept that”, and “That would really bother me”. The complete list of the relevance related as well as the functional and dysfunctional questions we used for assessing the nine quality principles can be found in [ 13 ] and Supplementary Tables S 1 and S 2 (translated from the original German language versions).…”

“…According to Kano, a product feature can be rated as either attractive (A), must-be (M), one-dimensional (O), indifferent (I), reverse (R), or questionable (Q). To reflect an individual’s appraisal of the respective feature, the answer combinations for the functional and dysfunctional questions are used (see [ 13 ] or [ 28 ] for a more detailed explanation as well as a tabulation of the possible answer combinations and their assigned categories). An “attractive” feature (A) increases satisfaction [ 28 ].…”

“…Therefore, we present an exemplary, more detailed evaluation of the previously introduced prioritisation method for quality criteria based on a Kano survey [ 13 ]. Specifically, we evaluated the feasibility of determining an adequate and stable ranking for nine (app) quality criteria for use in health contexts concerning potential demographic influences.…”

“…Therefore, it was desired to reduce the list of quality criteria or to enable a ranking of these to prioritise the most important ones. To this end, we turned to another group of stakeholders and assessed the data obtained using the prioritisation mentioned above [ 13 ].…”

Proposal of a New Rating Concept for Digital Health Applications in Orthopedics and Traumatology

“…Passing the regulatory process is one possible criterion for quality, but “quality” comprises many more aspects, such as technical soundness, as proof of basic functionality. Clearly, there is a need for criteria with which users (healthy laypeople, patients or medical staff) can objectively identify whether certain app products are suitable for their own needs [ 16 , 17 ].…”