Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials

Austrian, Jonathan; Mendoza, Felicia; Szerencsy, Adam; Fenelon, Lucille; Horwitz, Leora I.; Jones, Simon; Кузнецова, М. М.; Mann, David

doi:10.2196/16651

Cited by 35 publications

(26 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Phase 3, we piloted the winning idea in a PIC practice with one physician over 5 five-hour shifts. When piloting, we used a mixed methods rapid implementation approach called rapid prototyping, which systematically tested and refined the strategy to ensure the implementation method was acceptable and feasible [ 38 , 39 , 48 , 49 ]. All study procedures were approved by the University of Pennsylvania’s Institutional Review Board.…”

Section: General Methodsmentioning

confidence: 99%

“…Rapid prototyping is the systematic testing of ideas in order to create and refine strategies quickly [ 55 ]. First employed in industrial design, this method has been extended to healthcare contexts where effective implementation strategies don’t yet exist and immediate feedback is necessary to optimize healthcare quality and safety [ 38 , 39 , 48 , 49 , 56 , 57 ]. Rapid prototyping facilitates learning as quickly as possible whether a strategy works and allows researchers to make adjustments as needed.…”

Section: General Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A pilot study of participatory and rapid implementation approaches to increase depression screening in primary care

et al. 2021

View full text Add to dashboard Cite

Background Most individuals with depression go unidentified and untreated. In 2016 the US Preventive Services Task Force released guidelines recommending universal screening in primary care to identify patients with depression and to link them to treatment. Feasible, acceptable, and effective strategies to implement these guidelines are needed. Methods This three-phased study employed rapid participatory methods to design and test strategies to increase depression screening at Penn Medicine, a large health system with 90 primary care practices. First, researchers solicited ideas and barriers from stakeholders to increase screening using an innovation tournament—a crowdsourcing method that invites stakeholders to submit ideas to address a workplace challenge. Second, a panel of stakeholders and scientists deliberated over and ranked the tournament ideas. An instant runoff election was held to select the winning idea. Third, the research team piloted the winning idea in a primary care practice using rapid prototyping, an approach that quickly refines and iterates strategy designs. Results The innovation tournament yielded 31 ideas and 32 barriers from diverse stakeholders (12 primary care physicians, 10 medical assistants, 4 nurse practitioners, 2 practice managers, and 4 patient support assistants). A panel of 6 stakeholders and scientists deliberated on the ideas and voted for patient self-report (i.e., through tablet computers, text message, or an online patient portal) as the winning idea. The research team rapid prototyped tablets in one primary care practice with one physician over 5 five-hour shifts to examine the feasibility, acceptability, and effectiveness of the strategy. Most patients, the physician, and medical assistants found the tablets acceptable and feasible. However, patient support assistants struggled to incorporate them in their workflow and expressed concerns about scaling up the process. Depression screening rates were higher using tablets compared to usual care; follow-up was comparable between tablets and usual care. Conclusions Rapid participatory methods engaged and amplified the voices of diverse stakeholders in primary care. These methods helped design an acceptable and feasible implementation strategy that showed promise for increasing depression screening in a primary care setting. The next step is to evaluate the strategy in a randomized controlled trial across primary care practices.

show abstract

Section: General Methodsmentioning

confidence: 99%

Section: General Methodsmentioning

confidence: 99%

A pilot study of participatory and rapid implementation approaches to increase depression screening in primary care

et al. 2021

View full text Add to dashboard Cite

show abstract

“…If the optimization criteria are met, then this can often justify movement to Phase III. Plausible optimization trials for this phase could include but are not limited to: A/B testing (as used in the technology industry for improving usability) [67][68][69] , factorial trials as used in MOST 53,54,56,57 , sequential multiple assigned randomized trials [62][63][64][65] , micro-randomized trials [58][59][60][61]70 , and control optimization trials 71,72 . Nahum-Shani, I. et al 39 provide guidance on when to use which optimization design.…”

Section: Question 2c: Have the Optimization Criteria Relevant To The ...mentioning

confidence: 99%

The Digital Therapeutics Real World Evidence Framework: An approach for guiding evidence-based DTx design, development, testing, and monitoring

Kim¹,

Patrick²,

Nebeker³

et al. 2022

Preprint

View full text Add to dashboard Cite

Digital Therapeutics (DTx) are increasingly seen as a promising way to provide safe, effective, accessible, sustainable, scalable, and equitable approaches to advance individual and population health. Developing DTx is inherently complex in that DTx may include multiple interacting components, such as tools to support activities like medication adherence, health behavior goal-setting or self-monitoring, and algorithms that adapt the provision of these according to individual needs. As recommended in a recent worldwide review of regulatory approaches for DTx, a way to address this complexity is to base the design, development, testing, and iterative improvements of DTx on real-world evidence (RWE). Notably, that review highlights the need for improved guidance on when and how to use RWE for DTx. In this paper, we propose the DTx RWE Framework to provide a pragmatic, iterative, milestone-driven approach for producing RWE for DTx regulation. The DTx RWE Framework is based on best practices from human-centered design, optimization trials for behavioral interventions, behavioral health research, and implementation science. It maps these onto the traditional four phase development model for pharmaceuticals but includes key adaptations relevant to RWE production for DTx, such as including early and ongoing partnerships between developers of DTx and their intended users in clinical and/or community settings. Adoption of our DTx RWE Framework may help address known problems with current DTx, including questionable marketing claims, lack of evidence-based guidance for patients, providers, and health care leaders and the need to improve economic incentives that advance the adoption and use of DTx.

show abstract

“…In addition to online environments, A/B testing could also be used to improve communication tools in the clinical setting. For example, a research team at NYU Langone Health used A/B testing as a strategy to improve clinical decision support features in their medical center’s electronic health record system [ 45 ]. The team tested three different versions of a tobacco cessation alert that varied in the framing of the message: financial (emphasizing the additional revenue that a physician could generate by performing tobacco cessation counseling) vs. evidence-based (highlighting that tobacco cessation was part of providing high-quality care) vs. regulatory (stressing that tobacco cessation counseling was integral to the institution’s expectations and policies), and in an additional round of testing, assessed the impact of adding images to reinforce the message framing [ 45 ].…”

Section: Approaches To Make Health Communication Research More Timely...mentioning

confidence: 99%

“…For example, a research team at NYU Langone Health used A/B testing as a strategy to improve clinical decision support features in their medical center’s electronic health record system [ 45 ]. The team tested three different versions of a tobacco cessation alert that varied in the framing of the message: financial (emphasizing the additional revenue that a physician could generate by performing tobacco cessation counseling) vs. evidence-based (highlighting that tobacco cessation was part of providing high-quality care) vs. regulatory (stressing that tobacco cessation counseling was integral to the institution’s expectations and policies), and in an additional round of testing, assessed the impact of adding images to reinforce the message framing [ 45 ]. The experiments indicated that neither the framing nor the addition of images resulted in significant differences in acceptance rates for the tobacco alert, possibly because changes to alert presentation might not be sufficient to modify clinical practice behaviors in a significant way [ 45 ].…”

Section: Approaches To Make Health Communication Research More Timely...mentioning

confidence: 99%

An Overview of Innovative Approaches to Support Timely and Agile Health Communication Research and Practice

Gaysynsky

Heley

Chou

2022

IJERPH

View full text Add to dashboard Cite

Innovative approaches are needed to make health communication research and practice more timely, responsive, and effective in a rapidly changing information ecosystem. In this paper we provide an overview of strategies that can enhance the delivery and effectiveness of health communication campaigns and interventions, as well as research approaches that can generate useful data and insights for decisionmakers and campaign designers, thereby reducing the research-to-practice gap. The discussion focuses on the following approaches: digital segmentation and microtargeting, social media influencer campaigns, recommender systems, adaptive interventions, A/B testing, efficient message testing protocols, rapid cycle iterative message testing, megastudies, and agent-based modeling. For each method highlighted, we also outline important practical and ethical considerations for utilizing the approach in the context of health communication research and practice, including issues related to transparency, privacy, equity, and potential for harm.

show abstract

Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials

Cited by 35 publications

References 31 publications

A pilot study of participatory and rapid implementation approaches to increase depression screening in primary care

A pilot study of participatory and rapid implementation approaches to increase depression screening in primary care

The Digital Therapeutics Real World Evidence Framework: An approach for guiding evidence-based DTx design, development, testing, and monitoring

An Overview of Innovative Approaches to Support Timely and Agile Health Communication Research and Practice

Contact Info

Product

Resources

About