Levels of evaluation: Beyond Kirkpatrick

Abstract

For a process that is so important and rational, evaluation brings both promise and threat. The promise lies in determining what worked and what didn't, which allows us to continuously improve. The threat of evaluation flows from the fear that performance data will be misused for blaming and not for fixing and improving.Whether it is recognized or not, evaluation is vital to success and continuous improvement (Deming, 1986, personal communication, May 10, 1990Kaufman and Zahn, 1993). It provides the feedback that allows us to measure discrepancies and institute improvements. Evaluation is essential, so we had better use it wisely Uhfortunately, it is seldom either used wisely or used to answer the most vital questions that face every organization. It is an underutilized endeavor that is often inappropriately applied and that sometimes has harmful results (Keller and Moller, 1983). We suggest that there are at least three reasons for this: (1) the most-used definitions and models of evaluation often are too restricted, (2) we do not ask the right questions for evaluation to answer, and (3) the relationship between ends and means is not made clear in the evaluation, planning, and implementation processes.A more effective approach to evaluation will encompass an expanded concept of evaluation that includes result-related questions that contribute to continuous improvement by comparing intentions with results. Definitions of EvaluationDefinitions of evaluation range widely (Isaac and Michael, 1971;Popham, 1974; Kaufman, 1992b, 19951, varying from "supplying information to deci-Note: We appreciate reviews and suggestions for revision provided by Atusi Harumi, University of Houston-Clear Lake, Texas, and David Wilkinson, L a Trobe University, Victoria, Australia.FORUM is a nonrefereed section inviting readers' reactions and opinions.

Achieving scale of farmer reach with improved common bean technologies: the role of village-based advisors

“…Lukuyu et al (2012) assessed the effectiveness of volunteer farmers in disseminating technologies in Kenya, by considering their technical (practical agricultural expertise and knowledge) and individual characteristics (honesty, interest, and willingness to work without expecting any reward). Kaufman and Keller (1994) framework for assessing effectiveness focuses on four dimensions; (i) learner satisfaction and relevance of training; (ii) participant knowledge, skills, and attitude; (iii) participant application of learning; and (iv) increase in productivity and efficiency. For this study, we adapted the framework by Kaufman and Keller to assess the effectiveness of VBA approach.…”

Can Health-care Assistant Training improve the relational care of older people? (CHAT) A development and feasibility study of a complex intervention

“…53 The Kirkpatrick training evaluation model has been criticised for a lack of attention to the environment in which trainee skills are practised. 54 This was in part addressed by the phase 2 process evaluation for which we drew on recently published guidance. 55 A range of methods was used to inform our understanding of the different contexts in which the training intervention was delivered, the process of intervention delivery and the mechanisms of impact.…”

Establishing and Embedding Evaluation as a Central Landmark on the Organisational Horizon

“…This sometimes has negative effects on the transparency, openness and accountability of evaluation respondents, due to concerns that honesty may adversely affect career progression. Kaufman and Keller (1996) state that, 'the threat of evaluation flows from the fear that performance data will be misused for blaming and not for fixing or improving ' (p. 371). One of the Performance and Evaluation Team's major 'cultural' challenges has been to promote evaluation as a continuous improvement tool, rather than a method of validating or apportioning blame.…”