Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study

Abstract

Background Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. Objective The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. Methods Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an “eHealth evaluation cycle” and subsequently compose the online “eHealth methodology guide.” Results The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an “eHealth evaluation cycle” was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the “eHealth methodology guide” was composed by assigning the 75 evaluation approaches to the specific study phases of the “eHealth evaluation cycle.” Conclusions Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online “eHealth methodology guide.” By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the “eHealth evaluation cycle,” the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Psychological eHealth interventions for people with intellectual disabilities: A scoping review

“…Thornicroft et al, 2011 ) and in particular eHealth interventions (e.g. Bonten et al, 2020 ) often starts with an initial phase in which the opportunities of eHealth interventions are explored in feasibility/pilot studies. Within those studies, the aim is to provide insight into the possibilities of psychological eHealth interventions rather than conducting these interventions to explore its effectiveness.…”

A Serious Puzzle Game to Enhance Adherence to Antirheumatic Drugs in Patients With Rheumatoid Arthritis: Systematic Development Using Intervention Mapping

“…Stage 1 consisted of 2 rounds of 2 weeks of user testing at home after which data on acceptability were collected. Acceptability was determined using the Technology Acceptance Model as underpinning, which is a well-established model for usability evaluation of eHealth [39][40][41]. This model postulates that ease of using a technology influences the perceived usefulness and the attitude toward using and together form the behavioral intention to use a technology, which leads to actual use.…”

“…Lastly, the Technology Acceptance Model, a well-established model for evaluating usability of eHealth, will be applied to collect patient acceptance of the puzzle game. Data collection will be similar to stage 1 of the user testing: the System Usability Scale will assess ease of use, GameFlow will assess perceived usefulness, part of the uMARS will assess the attitude toward using, and Google Analytics will collect actual use [39][40][41][42][43][44][45][46].…”

Time to act mature—Gearing eHealth evaluations towards technology readiness levels

“…1 In the scientific community that focuses on eHealth evaluation, there is no consensus on how to create the best evaluation design. 2 , 3 According to the standards of evidence-based medicine, large prospective randomized controlled trials (RCTs) are considered the gold standard for evaluating the safety and effectiveness of medical interventions. 4 As the characteristics of an RCT do not match well with the evaluation of eHealth, it is currently acknowledged among experts that there is an urgent need for other evaluation designs.…”