Assessing Apps for Health Care Workers Using the ISYScore-Pro Scale: Development and Validation Study

Abstract

Background The presence of mobile phone and smart devices has allowed for the use of mobile apps to support patient care. However, there is a paucity in our knowledge regarding recommendations for mobile apps specific to health care professionals. Objective The aim of this study is to establish a validated instrument to assess mobile apps for health care providers and health systems. Our objective is to create and validate a tool that evaluates mobile health apps aimed at health care professionals based on a trust, utility, and interest scale. Methods A five-step methodology framework guided our approach. The first step consisted of building a scale to evaluate apps for health care professionals based on a literature review. This was followed with expert panel validation through a Delphi method of (rated) web-based questionnaires to empirically evaluate the inclusion and weight of the indicators identified through the literature review. Repeated iterations were followed until a consensus greater than 75% was reached. The scale was then tested using a pilot to assess reliability. Interrater agreement of the pilot was measured using a weighted Cohen kappa. Results Using a literature review, a first draft of the scale was developed. This was followed with two Delphi rounds between the local research group and an external panel of experts. After consensus was reached, the resulting ISYScore-Pro 17-item scale was tested. A total of 280 apps were originally identified for potential testing (140 iOS apps and 140 Android apps). These were categorized using International Statistical Classification of Diseases, Tenth Revision. Once duplicates were removed and they were downloaded to confirm their specificity to the target audience (ie, health care professionals), 66 remained. Of these, only 18 met the final criteria for inclusion in validating the ISYScore-Pro scale (interrator reliabilty 92.2%; kappa 0.840, 95% CI 0.834-0.847; P<.001). Conclusions We have developed a reproducible methodology to objectively evaluate mobile health apps targeted to health care professionals and providers, the ISYScore-Pro scale. Future research will be needed to adapt the scale to other languages and across other domains (eg, legal compliance or security).

Clinical Validation of Digital Healthcare Solutions: State of the Art, Challenges and Opportunities

“…Out of the 15 examined articles, the focus was primarily on evaluating mHealth apps, with nine frameworks (60%) specifically tailored for this purpose [ 45 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ]. Four frameworks were applicable to all eHealth technologies [ 11 , 51 , 55 , 56 ], one extended its scope to cover both mHealth apps, AI, and robotics [ 22 ], and another one proposed quality criteria for AI-based prediction models [ 57 ].…”

“…A total of 12 of the 14 frameworks (87%) were developed relying on eHealth experts, healthcare professionals or researchers’ opinions [ 11 , 45 , 47 , 48 , 49 , 50 , 51 , 53 , 54 , 55 , 56 , 58 ]; additionally, some frameworks incorporated input from other stakeholders, such as technology companies, family members, policymakers, potential users, or app developers and engineers [ 22 , 49 , 52 , 53 , 58 , 59 ].…”

“…Six focused on all stakeholders, including governments, patients, clinicians, and health researchers [ 45 , 47 , 53 , 54 , 59 , 60 ]. Three targeted both patients and clinicians [ 49 , 50 , 56 ], two targeted decision-makers [ 22 , 52 ], two targeted only healthcare professionals [ 48 , 51 ], and one specifically targeted eHealth implementers in developing countries [ 55 ].…”

A systematic appraisal of the information, engagement, aesthetic and functional quality of nutrition-related smartphone apps for children and adolescents

“…We used Google Advanced Search ( 28 ) to search for apps within the Apple store (itunes.apple.com) by using the following terms: ‘nutrition AND kids,’ OR ‘foods AND kids,’ OR ‘beverages AND kids,’ OR ‘cooking AND kids,’ OR ‘obesity AND kids,’ OR ‘weight AND kids.’ We used Google Advanced Search because it allows users to apply filters (such as ‘all of these words’ or ‘any combination of these words’) that enhanced the quality of our search. This approach has also been used by other researchers conducting ratings of mobile apps ( 29 – 31 ) . We searched only within the Apple app store to reduce bias in this way: Apps within the Apple app store are generally of higher quality than Google Play apps because the approval process for Google Play is much faster and less strict ( 32 ) .…”

“…We used Google Advanced Search because it allows users to apply filters (such as 'all of these words' or 'any combination of these words') that enhanced the quality of our search. This approach has also been used by other researchers conducting ratings of mobile apps (29)(30)(31) . We searched only within the Apple app store to reduce bias in this way: Apps within the Apple app store are generally of higher quality than Google Play apps because the approval process for Google Play is much faster and less strict (32) .…”