Beyond Formal Experimental Design: Towards an Expanded View of the Training Evaluation Process

Abstract

Textbook treatments of training evaluation typically equate evaluation with the measurement of change and focus on formal experimental design as the mechanism for controlling threats to the inference that the training intervention produced whatever change was observed. This paper notes that two separate questions may be of interest: How much change has occurred? and. Has a target performance level been reached? We show that the evaluation mechanisms needed to answer the two types of questions are markedly different, and discuss circumstances under which an evaluator's interests will focus on one, the other, or both of these questions. We then discuss alternatives to formal design as mechanisms for reducing various threats to validity, and discuss trade-offs between intemal validity and statistical conclusion validity.In this paper we wish to develop a series of ideas that expand on traditional approaches to training program evaluation. First, we note that evaluating training is typically equated with the tneasurement of change, and evaluation designs are critiqued in terms of their adequacy for answering the question. Can the degree of change attributable to training be quantified? We will argue that while quantifying the degree of change is important in some circumstances, in a variety of applied situations the organization's primaiy interest is in determining whether trainees have reached some target performance level. We will show that the training evaluation methods needed to quantify the degree of change due to training are different from those needed to determine whether a target performance level has been reached. We will develop prescriptions for when it is necessary to measure change and when it is sufficient to measure level of achievement.Second, textbook treatments often view formal experimental design as the sole mechanism for avoiding threats to intemal validify in settings

360‐degree feedback with systematic coaching: Empirical analysis suggests a winning combination

“…For instance, had we split the sample size in two to create a control group, we would have ruled out the ambiguity regarding potential alternative explanations, but in terms of power, "the true experimental design [would have been] strikingly inadequate" (Sackett & Mullen, 1993, p. 624). This is because the power of the experiment would have gone down from .98 as present in this study (for N = 80 [87 in our study] d = .5, r xy = .3) to .59 (for n e = 40 and n c = 40 [43.5 in our study], d = .5, r xy = .3), a 40% reduction in probability of correctly rejecting the null hypothesis when it is false (Arvey & Cole, 1989;Arvey et al, 1985;Cohen, 1988;Sackett & Mullen, 1993).…”

“…As in much previous research on training evaluations (e.g., Arvey, Cole, Hazucha, & Hartanto, 1985;Sackett & Mullen, 1993), the constraint of not including a control group in our study was the population of only N = 87 available in this organization. Sackett and Mullen (1993, p. 624) explicate this situation by noting that "… we see no ready mechanism for combating the low statistical power of the true experimental design in the setting where N is constrained.…”

Knowledge for Theory and Practice

“…These include the difficulties of meeting conventional scientific requirements of internal and external validity (Cook & Campbell, 1979;Sackett & Mullen, 1993). Furthermore, practitioner involvement may compromise the independence and objectivity of the academic researcher (Beyer & Trice, 1982;Grey, 2001;Hackman, 1985), and participating organizations may view the research findings as proprietary and, thus, not available for dissemination in the public domain (Lawler et al, 1985).…”

The effects of alternative reports of human resource development results on managerial support

“…However, such precautions are not foolproof. For instance, Sackett and Mullen (1993) assert that "a pre-experimental [generally uninterpretable] design, paired with careful investigation into the plausibility of various threats, is still better than no evaluation at all, given that organizations must make decisions about future training efforts with or without evaluation data" (p. 621). Perhaps the real ethical issue is not whether an experimental evaluation design is preferable to a preexperimental one, but whether or not one chooses to base decisions about HRD programs on systematically gathered information.…”