The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): two novel evaluation methods for developing optimal training programs

Abstract

SummaryCurrent methodologies in training evaluation studies largely employ a single method entitled random confirmatory trials, prompting several concerns. First, practitioners and researchers often analyze the effectiveness of their entire omnibus training, rather than the individual elements or identifiable components of the training program. This slows the testing of theory and development of optimal training programs. Second, a common training is typically administered to all employees within an organization or workgroup; however, certain factors may cause individualized training to be more effective. Given these concerns, the current paper presents two training evaluation methodologies to overcome these problems: the multiphase optimization strategy and sequential multiple assignment randomized trials. The multiphase optimization strategy is a method to evaluate a standard training, which emphasizes the importance of a multi-stage training evaluation process to analyze individual training elements. In contrast, sequential multiple assignment randomized trial is used to evaluate an adaptive training that varies over time and/or trainees. These methodologies jointly overcome the problems noted earlier, and they can be integrated to address several of the key challenges facing training researchers and practitioners. Copyright © 2016 John Wiley & Sons, Ltd.Keywords: training; methodology; statistics Despite the widespread study of organizational training, certain methodological issues systematically appear in training scholarship. Currently, the most used method to evaluate a training program is called random confirmatory trials (RCTs;Campbell, 1988;Tannenbaum & Yukl, 1992). While this method provides many benefits, RCTs also have many drawbacks. Notably, although a training program may consist of several individual elements, 1 RCTs only analyze the effectiveness of the overall training (Bass & Avolio, 1990;Burke & Day, 1986;Smith & Smith, 2007). For example, Barling, Weber, and Kelloway (1996) investigated the effectiveness of a training program to improve managers' transformational leadership. While the training consisted of four clearly identifiable elements to achieve this goal, their training evaluation methodology only analyzed the effectiveness of all the elements together. Although RCTs have provided insightful evidence on general training, little is known about the effectiveness of the individual training elements. It is possible that a single element entirely drives employee improvement, or some elements may even detract from the overall effect. Unfortunately, through only analyzing an entire regimen, the successful or unsuccessful elements cannot be identified (Isler et al., 2009;Lesch, 2008). Organizations may *Correspondence to: Matt C. Howard, 5811 USA Drive S., Rm. 346, Mitchell College of Business, University of South Alabama, Mobile, AL, 36688, U.S.A. E-mail: mhoward@southalabama.edu. 1The term "training" refers to an intervention created to improve employee attributes and/or performance, with t...

Measuring fidelity to behavioural support delivery for smoking cessation and its association with outcomes

“…There is potential for developing shorter versions of the fidelity index and for optimized intervention delivery if the active ingredients can independently predict quit outcomes. Studies using multi‐phase optimization strategy and sequential multiple‐assignment randomized trials could move the field forward in providing such mechanistic evidence .…”

Let's Keep Looking for Other Roads: Improving Approaches to Identifying and Addressing Key Drivers

“…Cucina et al (2017) did not provide any suggestions for designing organizational interventions, which is perhaps the most important aspect of their entire approach. I found this unfortunate, as many important developments in intervention creation have been made fairly recently—many occurring in the training literature (Gully & Chen, 2010; Howard & Jacobs, 2016).…”

“…It also forces practitioners to analyze the entire omnibus intervention rather than the individual components that compose the intervention. Howard and Jacobs (2016) recently detailed two sophisticated approaches to design an intervention—the multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART)—which overcome both of these concerns.…”

Noninferiority and equivalence tests in sequential, multiple assignment, randomized trials (SMARTs).

“…In the area of health psychology, SMARTs have been recommended as an experimental tool that can be used to inform the development of adaptive behavior change interventions that capitalize on advances in mobile and wireless technology (Riley et al, 2015). In organizational psychology, Howard and Jacobs (2016) employed a SMART to optimize the sequencing of training methods, Eden (2017) discussed the utility of the SMART as an experimental approach that can inform the adaptation of organizational policies and practices. In their introduction to AIs and SMARTs in the area of educational and school psychology, August, Piehler and Miller (2018) provide multiple examples of SMART applications in this area.…”