MCMTC: A Pragmatic Framework for Selecting an Experimental Design to Inform the Development of Digital Interventions

Abstract

Advances in digital technologies have created unprecedented opportunities to deliver effective and scalable behavior change interventions. Many digital interventions include multiple components, namely several aspects of the intervention that can be differentiated for systematic investigation. Various types of experimental approaches have been developed in recent years to enable researchers to obtain the empirical evidence necessary for the development of effective multiple-component interventions. These include factorial designs, Sequential Multiple Assignment Randomized Trials (SMARTs), and Micro-Randomized Trials (MRTs). An important challenge facing researchers concerns selecting the right type of design to match their scientific questions. Here, we propose MCMTC – a pragmatic framework that can be used to guide investigators interested in developing digital interventions in deciding which experimental approach to select. This framework includes five questions that investigators are encouraged to answer in the process of selecting the most suitable design: (1) Multiple-component intervention: Is the goal to develop an intervention that includes multiple components; (2) Component selection: Are there open scientific questions about the selection of specific components for inclusion in the intervention; (3) More than a single component: Are there open scientific questions about the inclusion of more than a single component in the intervention; (4) Timing: Are there open scientific questions about the timing of component delivery, that is when to deliver specific components; and (5) Change: Are the components in question designed to address conditions that change relatively slowly (e.g., over months or weeks) or rapidly (e.g., every day, hours, minutes). Throughout we use examples of tobacco cessation digital interventions to illustrate the process of selecting a design by answering these questions. For simplicity we focus exclusively on four experimental approaches—standard two- or multi-arm randomized trials, classic factorial designs, SMARTs, and MRTs—acknowledging that the array of possible experimental approaches for developing digital interventions is not limited to these designs.

Hybrid Experimental Designs for Intervention Development: What, Why, and How

“…2, C and F). Note that this comparison would involve using outcome data from the entire sample of nonresponders to estimate the effect, which can be viewed as the main effect of the subsequent components among nonresponders averaging over the initial components (Nahum-Shani, Dziak, & Wetter, 2022;Nahum-Shani et al, 2020). The multiple, sequential, random assignments in this SMART yield four "embedded" ADIs (Table 2).…”

“…2, D–F). Note that this comparison would involve using outcome data from the entire sample to estimate the effect, which can be viewed as the main effect of the initial components averaging over the subsequent components for responders and nonresponders (Nahum-Shani, Dziak, & Wetter, 2022; Nahum-Shani et al, 2020). The second question can be answered by comparing the mean outcome across the two experimental conditions in which nonresponders were offered the step-up subsequently (Fig.…”

The Digital Therapeutics Real World Evidence Framework: An approach for guiding evidence-based DTx design, development, testing, and monitoring

“…Phase I activities correspond to the "double diamond" approach 37 used in human-centered design and related methods where the problem and solution specification are delineated. Phase II activities are drawn from two National Institutes of Health (NIH)-recognized approaches for behavioral intervention development, namely the Obesity-Related Behavioral Intervention Trials (ORBIT) model 20 and its extensions 38 and the Multiphase Optimization Strategy (MOST) 39,40 . Phase III activities are based on insights from pragmatic clinical trial best practices, including PRECIS-2 41 and RE-AIM (Reach Effectiveness, Adoption, Implementation, and Maintenance) 42,43 , and recommendations from an NIH-recognized expert panel on comparator selection for behavioral interventions 44 .…”

“…Plausible optimization trials for this phase could include but are not limited to: A/B testing (as used in the technology industry for improving usability) [67][68][69] , factorial trials as used in MOST 53,54,56,57 , sequential multiple assigned randomized trials [62][63][64][65] , micro-randomized trials [58][59][60][61]70 , and control optimization trials 71,72 . Nahum-Shani, I. et al 39 provide guidance on when to use which optimization design.…”

Sample size estimation for comparing dynamic treatment regimens in a SMART: A Monte Carlo-based approach and case study with longitudinal overdispersed count outcomes

“…DTRs are intended to provide replicable protocols for tailoring sequences of treatments in clinical practice. 3,4…”