Dimensions and design criteria for developing training reaction evaluations

Abstract

Although there is research on training program evaluation, there is little systematic research on the design, development and use of training reaction evaluations. In order to obtain meaningful information from a program evaluation, evaluation professionals must be rigorous in the design and development of all aspects of an evaluation study, including instrumentation. The purposes of the study were to classify the dimensions of information sought using reaction evaluations and to establish design criteria for developing training reaction instruments. There were three major phases of the study:(1) classification of the dimensions and questionnaire design criteria used in reaction evaluations;(2) validation of the classi ed dimensions and the questionnaire design criteria by subject matter experts; and (3) assessment of a sample of training reaction instruments currently used in training programs in US corporations. The research ndings were: eleven dimensions for reaction evaluation were identi ed and classi ed by purpose. Five overall design criteria, each consisting of several sub-criteria, were judged important in the design of reactionnaires. These include: introduction and directions; question format; question construction; questionnaire layout; and data analysis. It was concluded that a well-designed training reaction instrument integrates the proper application of design criteria with appropriate reaction dimensions. Most training reaction instruments used by US corporations consisted of questions representing only a few dimensions. The instruments varied in form and length. Few of them properly utilized the established questionnaire design criteria.

More than smile sheets: Rasch analysis of training reactions in a medical center

“…Questionnaire Items Prior research indicates that the reaction questions need to assess multiple ways of looking at the underlying perception of value of a training program (Lee & Pershing, 2002). Meta-analysis (Alliger et al, 1997), several structural analyses of instruments (Brown, 2005;Morgan & Casper, 2000), and a review of applied practices (Lee & Pershing, 2002) have demonstrated that the measurement of sentiments about training quality needs to assess multiple features of a program.…”

“…Meta-analysis (Alliger et al, 1997), several structural analyses of instruments (Brown, 2005;Morgan & Casper, 2000), and a review of applied practices (Lee & Pershing, 2002) have demonstrated that the measurement of sentiments about training quality needs to assess multiple features of a program. Consistent with the need to assess sentiments from a number of viewpoints, multiple items were used in this instrument to address different elements of a training program in this organization.…”

Reflections and future prospects for evaluation in human resource development

“…The authors concluded that if the most prevalent level of evaluation is to become more useful for evaluators, instructors, and learners then design and content need to adhere to uniform and consistent standards. Similar findings were reported in an examination of reaction instruments used in corporate settings (Lee & Pershing, 2002). In a study of corporate elearning program evaluations simplistic evaluation measures were identified as the primary focus (MacPherson et al, 2004).…”

“…The researchers who have examined and proposed evaluation models use Kirkpatrick's model as a basis for their investigations. Some researchers have introduced new comprehensive models (Alvarez et al 2004; Holton, 1996; Kaufman & Watkins, 2000), others have focused on one of the levels such as reaction or return on investments (Lee & Pershing, 2002; Pershing & Pershing, 2001; Wang, et al 2002). One study was identified that provided a detailed analysis on learning or behavior (Dooley et al, 2004).…”

Exploring the Mediating Role of Training on the Relationship between Future Orientation and Context Performance

“…The third factor to training relates to the perceived efficiency of training. It relates to the organization and the mode of operation of the course and it refers to an evaluation about regarding the physical location and administration of the course [27]. We capture these arguments in our study hypothesis.…”