Formative evaluation: Developmental testing and expert review

Geis, George L.

doi:10.1002/pfi.4160260402

Cited by 11 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Others have proposed novel methods for formative evaluations, which occur during the creation of a training program (Brown & Gerhardt, ); however, these evaluation methods cannot provide decisive inferences about the effectiveness of a particular training or its elements beyond an alternative, and they are often based upon qualitative inferences derived from very small sample sizes (i.e., n = 6, Medley‐Mark & Weston, ; n = 6, 3, and 3, Saroyan, ). Other authors only apply RCTs for their formative evaluations (Dick & Carey, ; Geis, ). Lastly, some authors have sporadically implemented alternative designs, but systematic application of these methods has not been seen (Aguinis & Kraiger, ; Shadish et al, 2002; Salas et al, ).…”

Section: Training Backgroundmentioning

confidence: 99%

The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): two novel evaluation methods for developing optimal training programs

Howard

Jacobs

2016

J Organ Behavior

View full text Add to dashboard Cite

SummaryCurrent methodologies in training evaluation studies largely employ a single method entitled random confirmatory trials, prompting several concerns. First, practitioners and researchers often analyze the effectiveness of their entire omnibus training, rather than the individual elements or identifiable components of the training program. This slows the testing of theory and development of optimal training programs. Second, a common training is typically administered to all employees within an organization or workgroup; however, certain factors may cause individualized training to be more effective. Given these concerns, the current paper presents two training evaluation methodologies to overcome these problems: the multiphase optimization strategy and sequential multiple assignment randomized trials. The multiphase optimization strategy is a method to evaluate a standard training, which emphasizes the importance of a multi-stage training evaluation process to analyze individual training elements. In contrast, sequential multiple assignment randomized trial is used to evaluate an adaptive training that varies over time and/or trainees. These methodologies jointly overcome the problems noted earlier, and they can be integrated to address several of the key challenges facing training researchers and practitioners. Copyright © 2016 John Wiley & Sons, Ltd.Keywords: training; methodology; statistics Despite the widespread study of organizational training, certain methodological issues systematically appear in training scholarship. Currently, the most used method to evaluate a training program is called random confirmatory trials (RCTs;Campbell, 1988;Tannenbaum & Yukl, 1992). While this method provides many benefits, RCTs also have many drawbacks. Notably, although a training program may consist of several individual elements, 1 RCTs only analyze the effectiveness of the overall training (Bass & Avolio, 1990;Burke & Day, 1986;Smith & Smith, 2007). For example, Barling, Weber, and Kelloway (1996) investigated the effectiveness of a training program to improve managers' transformational leadership. While the training consisted of four clearly identifiable elements to achieve this goal, their training evaluation methodology only analyzed the effectiveness of all the elements together. Although RCTs have provided insightful evidence on general training, little is known about the effectiveness of the individual training elements. It is possible that a single element entirely drives employee improvement, or some elements may even detract from the overall effect. Unfortunately, through only analyzing an entire regimen, the successful or unsuccessful elements cannot be identified (Isler et al., 2009;Lesch, 2008). Organizations may *Correspondence to: Matt C. Howard, 5811 USA Drive S., Rm. 346, Mitchell College of Business, University of South Alabama, Mobile, AL, 36688, U.S.A. E-mail: mhoward@southalabama.edu. 1The term "training" refers to an intervention created to improve employee attributes and/or performance, with t...

show abstract

Section: Training Backgroundmentioning

confidence: 99%

The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): two novel evaluation methods for developing optimal training programs

Howard

Jacobs

2016

J Organ Behavior

View full text Add to dashboard Cite

show abstract

“…Although there are many papers that describe the results of formative evaluations, few studies compare formative evaluation models and procedures. Geis (1987) summarized the research on formative evaluation as follows: The situation has improved somewhat in recent years. An electronic literature search was conducted to locate studies that report results of different types of formative evaluation practices.…”

Section: Formative Evaluation Researchmentioning

confidence: 99%

Formative Evaluation: An Integrative Practice Model and Case Study

Brown

Gerhardt

2002

Personnel Psychology

View full text Add to dashboard Cite

Training evaluation research and practice has been dominated by a focus on outcomes of completed training programs, or on methods used to assess these outcomes. This focus has largely neglected formative evaluation, which involves evaluating training during design and development. The purpose of this paper is to review existing models of formative evaluation and suggest an integrative practice model that is specifically targeted at improving training delivered in work organizations.

show abstract

“…The kinds of expert most frequently cited are subject matter, presentation and user (eg Kemp, 1985), although Geis (1987), for example, does mention others: gatekeepers, sponsors, former students. The intention of this paper is to focus on a particular kind of expert only recently delineated -the cultural expert (Weston and McAlpine, 1990).…”

Section: Introductionmentioning

confidence: 98%

Cross‐Cultural Instructional Design: Using the Cultural Expert to Formatively Evaluate Process and Product

McAlpine

1992

Educational and Training Technology International

View full text Add to dashboard Cite

Instructional design models prescribe the use of experts, both to develop materials and during formative evaluation, to provide data to inform the revision process. The kinds of expert that are most frequently cited are subject matter, presentation and user, although others are also cited: gatekeepers, sponsors, former students. The intention of this paper is to focus on a particular kind of expert, one who can be essential to the design process in cross-cultural settings: the cultural expert. The cultural expert might be seen as a specific instance of a gatekeeper: someone who makes judgements regarding social, moral, and ethical issues relating to the design and delivery of instruction. Strategies for working with the cultural expert are described.

show abstract

Formative evaluation: Developmental testing and expert review

Cited by 11 publications

References 0 publications

The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): two novel evaluation methods for developing optimal training programs

The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): two novel evaluation methods for developing optimal training programs

Formative Evaluation: An Integrative Practice Model and Case Study

Cross‐Cultural Instructional Design: Using the Cultural Expert to Formatively Evaluate Process and Product

Contact Info

Product

Resources

About