Toward an Agenda for Research on Evaluation

Abstract

Evaluation seems to be almost everywhere these days. We read about the findings of large-scale program evaluations in the newspaper, we receive report cards on our neighborhood schools, we allow ourselves to be interviewed for evaluations of conferences we attend. Yet we know remarkably little about how evaluation is being practiced, why it is being practiced, by whom and where it is being practiced, and to what effect. As evaluators, we know a great deal about the evaluations that we conduct, and we may also know a fair amount about those evaluations that we come into contact with professionally, as citizens, or as participants. However, the big picture is less clear. Evaluators are unlikely to be able to say much descriptively about the size, scope, and shape of the evaluation enterprise-things like how many evaluations were completed last year, how much money was spent on evaluation, who did the evaluations, and how they were conducted. More troubling, if evaluators try to tell you anything deeper about the practice of evaluation, what they tell you almost certainly comes from their personal experience, interactions, reading, training, or intuition. The views you hear on the key issues in evaluation-such as what kind of evaluation to do in various circumstances, or when and why evaluation affects important actions-almost certainly are not based on rigorous, systematic evidence. Why not? Because there is a serious shortage of rigorous, systematic evidence that can guide evaluation or that evaluators can use for self-reflection or for improving their next evaluation.This volume focuses on one new and noteworthy source of systematic evidence about evaluation: in Chapter One, Christina Christie presents her analysis of, first, the similarities or dissimilarities among the views of eight

Methodological pluralism: The gold standard of STEM evaluation

“…Evaluators need to identify when and where to use methodologies, appropriate ways to merge data collected using diverse methodologies, and how to fairly report the results to a variety of audiences. As Henry and Mark (2003) point out, publications on how to do evaluation are "generally based on personal experience, observation, and the individual's sometimes idiosyncratic beliefs and values-and not on carefully gathered evidence that can be described, shared, and critiqued" (p. 70). The field of evaluation requires research on which methods are best in which situations.…”

Using Citation Analysis Methods to Assess the Influence of Science, Technology, Engineering, and Mathematics Education Evaluations

“…In response to the call for more research about how evaluations can be influential (Henry & Mark, 2003b;Scriven, 2007), this study was designed to identify the potential strengths and weaknesses of citation analysis for examining influence of STEM education evaluations using the products of four large-scale, multisite STEM program evaluation efforts. Influence is represented by the use of the knowledge generated by the evaluation as determined by the number of times the products from that evaluation were cited.…”

Non-formal educator use of evaluation results

“…Beginning in the 1980s, researchers began to broaden the concept of use to include activities such as learning from the process of conducting the evaluation, increasing stakeholder awareness of program impacts and outcomes and thinking about the evaluation (McCormick, 1997). More recently, Kirkhart (2000) and Henry and Mark (2003) have called for consideration of a broader construct of use to include evaluation influence. Patton (2008) now defines use as "how real people in the real world apply evaluation findings and experience" (p. 37).…”