Digital health; what do we mean by clinical validation?

Shah, Sachin Shailendra; Gvozdanovic, Andrew

doi:10.1080/17434440.2021.2012447

Cited by 9 publications

(15 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fourth, objective and homogeneous outcome measures should be evaluated for digital interventions using appropriate study designs with long-term follow-up. The potential for digital interventions to improve health has been scarcely realized, partly due to an insufficient knowledge base of guiding principles in the development and evaluation of such interventions [ 72 ]. While the gold standard for evaluating a health intervention is conducting a randomized controlled trial, these can take a long time and typically require many participants and extensive financial resources.…”

Section: Discussionmentioning

confidence: 99%

Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review

Santis¹,

Jahnel²,

Matthias³

et al. 2022

JMIR Public Health Surveill

View full text Add to dashboard Cite

Background Digital interventions are interventions supported by digital tools or technologies, such as mobile apps, wearables, or web-based software. Digital interventions in the context of public health are specifically designed to promote and improve health. Recent reviews have shown that many digital interventions target physical activity promotion; however, it is unclear how such digital interventions are evaluated. Objective We aimed to investigate evaluation strategies in the context of digital interventions for physical activity promotion using a scoping review of published reviews. We focused on the target (ie, user outcomes or tool performance), methods (ie, tool data or self-reported data), and theoretical frameworks of the evaluation strategies. Methods A protocol for this study was preregistered and published. From among 300 reviews published up to March 19, 2021 in Medline, PsycINFO, and CINAHL databases, 40 reviews (1 rapid, 9 scoping, and 30 systematic) were included in this scoping review. Two authors independently performed study selection and data coding. Consensus was reached by discussion. If applicable, data were coded quantitatively into predefined categories or qualitatively using definitions or author statements from the included reviews. Data were analyzed using either descriptive statistics, for quantitative data (relative frequencies out of all studies), or narrative synthesis focusing on common themes, for qualitative data. Results Most reviews that were included in our scoping review were published in the period from 2019 to 2021 and originated from Europe or Australia. Most primary studies cited in the reviews included adult populations in clinical or nonclinical settings, and focused on mobile apps or wearables for physical activity promotion. The evaluation target was a user outcome (efficacy, acceptability, usability, feasibility, or engagement) in 38 of the 40 reviews or tool performance in 24 of the 40 reviews. Evaluation methods relied upon objective tool data (in 35/40 reviews) or other data from self-reports or assessments (in 28/40 reviews). Evaluation frameworks based on behavior change theory, including goal setting, self-monitoring, feedback on behavior, and educational or motivational content, were mentioned in 22 out of 40 reviews. Behavior change theory was included in the development phases of digital interventions according to the findings of 20 out of 22 reviews. Conclusions The evaluation of digital interventions is a high priority according to the reviews included in this scoping review. Evaluations of digital interventions, including mobile apps or wearables for physical activity promotion, typically target user outcomes and rely upon objective tool data. Behavior change theory may provide useful guidance not only for development of digital interventions but also for the evaluation of user outcomes in the context of physical activity promotion. Future research should investigate factors that could improve the efficacy of digital interventions and the standardization of terminology and reporting in this field. International Registered Report Identifier (IRRID) RR2-10.2196/35332

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review

Santis¹,

Jahnel²,

Matthias³

et al. 2022

JMIR Public Health Surveill

View full text Add to dashboard Cite

show abstract

“…There is a strong need for studies of the real-world impact of OSCs on health care systems [7,14,23]. Robust prospective clinical studies comparing OSCs to existing provision, conducted by independent researchers, will be required in the 'preprimary care' and community setting to obtain a complete assessment of clinical product performance [27]. Some prospective clinical studies have been conducted to date comparing OSC triage to laypersons [28,29] and comparing OSC diagnoses to clinician diagnosis in real-world patients [30][31][32].…”

Section: Protocols For Prospective Real-world Evidencementioning

confidence: 99%

Section: Recommendations For Future Workmentioning

confidence: 99%

Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

Painter¹,

Hayhoe²,

Sasco³

et al. 2022

J Med Internet Res

View full text Add to dashboard Cite

The use of patient-facing online symptom checkers (OSCs) has expanded in recent years, but their accuracy, safety, and impact on patient behaviors and health care systems remain unclear. The lack of a standardized process of clinical evaluation has resulted in significant variation in approaches to OSC validation and evaluation. The aim of this paper is to characterize a set of congruent requirements for a standardized vignette-based clinical evaluation process of OSCs. Discrepancies in the findings of comparative studies to date suggest that different steps in OSC evaluation methodology can significantly influence outcomes. A standardized process with a clear specification for vignette-based clinical evaluation is urgently needed to guide developers and facilitate the objective comparison of OSCs. We propose 15 recommendation requirements for an OSC evaluation standard. A third-party evaluation process and protocols for prospective real-world evidence studies should also be prioritized to quality assure OSC assessment.

show abstract

“…Nevertheless, such studies have typically focused on feasibility and usability in small-scale prototype implementations [ 21 - 23 ]. Digital health products are often dealt with through an “implement now, clinically validate later” ethos [ 24 ], as they encompass a large number of different technologies. Thus, there is no clear consensus on methods for assessing the clinical effectiveness of digital health interventions [ 24 ], and data on the impact of chatbot interventions on clinical outcomes are scarce [ 25 ].…”

Section: Introductionmentioning

confidence: 99%

“…Digital health products are often dealt with through an “implement now, clinically validate later” ethos [ 24 ], as they encompass a large number of different technologies. Thus, there is no clear consensus on methods for assessing the clinical effectiveness of digital health interventions [ 24 ], and data on the impact of chatbot interventions on clinical outcomes are scarce [ 25 ]. Large user samples are necessary to gain more robust insights into chatbot implementation.…”

Section: Introductionmentioning

confidence: 99%

Adapting and Evaluating an AI-Based Chatbot Through Patient and Stakeholder Engagement to Provide Information for Different Health Conditions: Master Protocol for an Adaptive Platform Trial (the MARVIN Chatbots Study)

Ma,

Achiche,

Pomey

et al. 2024

JMIR Res Protoc

View full text Add to dashboard Cite

Background Artificial intelligence (AI)–based chatbots could help address some of the challenges patients face in acquiring information essential to their self-health management, including unreliable sources and overburdened health care professionals. Research to ensure the proper design, implementation, and uptake of chatbots is imperative. Inclusive digital health research and responsible AI integration into health care require active and sustained patient and stakeholder engagement, yet corresponding activities and guidance are limited for this purpose. Objective In response, this manuscript presents a master protocol for the development, testing, and implementation of a chatbot family in partnership with stakeholders. This protocol aims to help efficiently translate an initial chatbot intervention (MARVIN) to multiple health domains and populations. Methods The MARVIN chatbots study has an adaptive platform trial design consisting of multiple parallel individual chatbot substudies with four common objectives: (1) co-construct a tailored AI chatbot for a specific health care setting, (2) assess its usability with a small sample of participants, (3) measure implementation outcomes (usability, acceptability, appropriateness, adoption, and fidelity) within a large sample, and (4) evaluate the impact of patient and stakeholder partnerships on chatbot development. For objective 1, a needs assessment will be conducted within the setting, involving four 2-hour focus groups with 5 participants each. Then, a co-construction design committee will be formed with patient partners, health care professionals, and researchers who will participate in 6 workshops for chatbot development, testing, and improvement. For objective 2, a total of 30 participants will interact with the prototype for 3 weeks and assess its usability through a survey and 3 focus groups. Positive usability outcomes will lead to the initiation of objective 3, whereby the public will be able to access the chatbot for a 12-month real-world implementation study using web-based questionnaires to measure usability, acceptability, and appropriateness for 150 participants and meta-use data to inform adoption and fidelity. After each objective, for objective 4, focus groups will be conducted with the design committee to better understand their perspectives on the engagement process. Results From July 2022 to October 2023, this master protocol led to four substudies conducted at the McGill University Health Centre or the Centre hospitalier de l’Université de Montréal (both in Montreal, Quebec, Canada): (1) MARVIN for HIV (large-scale implementation expected in mid-2024), (2) MARVIN-Pharma for community pharmacists providing HIV care (usability study planned for mid-2024), (3) MARVINA for breast cancer, and (4) MARVIN-CHAMP for pediatric infectious conditions (both in preparation, with development to begin in early 2024). Conclusions This master protocol offers an approach to chatbot development in partnership with patients and health care professionals that includes a comprehensive assessment of implementation outcomes. It also contributes to best practice recommendations for patient and stakeholder engagement in digital health research. Trial Registration ClinicalTrials.gov NCT05789901; https://classic.clinicaltrials.gov/ct2/show/NCT05789901 International Registered Report Identifier (IRRID) PRR1-10.2196/54668

show abstract

Digital health; what do we mean by clinical validation?

Cited by 9 publications

References 6 publications

Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review

Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review

Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

Adapting and Evaluating an AI-Based Chatbot Through Patient and Stakeholder Engagement to Provide Information for Different Health Conditions: Master Protocol for an Adaptive Platform Trial (the MARVIN Chatbots Study)

Contact Info

Product

Resources

About