Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations

Abstract

Collaborative evaluations, in which program stakeholders participate extensively, typically are conducted for the primary purpose of enhancing the use of evaluation findings, and non-collaborative evaluations, in which stakeholders do not participate extensively, typically are conducted for the primary purpose of generating valid findings. This article shows how the gap can be bridged between these two types of evaluations. The article synthesizes, and elaborates on, a small body of recent research that showed that considerable interaction with stakeholders during the evaluation of small education programs helped achieve the raison d'etre of non-collaborative evaluations-that is, to enhance validity. It is also shown that theories and methods of stakeholder participation for the purpose of enhancing validity, which by and large have been ignored in the collaborative evaluation literature, can improve the quality of collaborative studies.

Do machine learning methods used in data mining enhance the potential of decision support systems? A review for the urban water sector

“…There is also an apparent gap between experts and/or researchers and stakeholders and/or policy makers, which often leads to an inadequate application of models and other support tools in the decision-making process [20]. Early involvement of stakeholders in the development and application of support tools can help bridge that gap and spread the application of model-based tools in the decision-making process [21]. In terms of the adaptive capacity of a system, a broad range of perspectives can facilitate adaptation by recognising new challenges and needs for institutional change [19].…”

An overview of appreciative inquiry in evaluation

“…By involving stakeholders in the evaluation process, evaluators typically seek to increase the validity of the evaluation data and the use of findings (Brandon, 1998;Cousins and Earl, 1992;Patton, 1997), build evaluation capacity (Compton, Glover-Kudon, Smith, and Avery, 2002;King, 2002;O'Sullivan and O'Sullivan, 1998;Preskill and Torres, 1999), empower individuals and groups to effect social change (Fetterman, 2000;Rebien, 1996), and make evaluation more democratic (Greene, 2000;House and Howe, 1999;MacNeil, 2002;Mathison, 2000). Although the degree of stakeholder participation may vary from one evaluation to another, there is little doubt that evaluators are increasingly involving stakeholders in various phases of evaluation practice (Cousins and Whitmore, 1998;Ryan and Johnson, 2000;Shulha and Cousins, 1997;Torres and Preskill, 2001;Weiss, 1998).…”

On Understanding Evaluation of Tool Support for IS Development

“…For example, Jones and Hughes (2001) stress the importance of stakeholders in arguing that the "views, beliefs and assumptions of stakeholders must be exposed and considered with the IS evaluation process" (p. 200), and Mcauley et al (2002) "extends Farbey et al's (1999) concern that stakeholder analysis should be included as an important evaluation theme" (p. 242) based on a study concerning an outsourcing decision. From evaluation studies in the field of educational programmes, Brandon (1998) concludes that "stakeholder participation can help enhance the likelihood of producing valid evaluation findings." (p. 333) There are many reports in the literature concerning why it is important to seriously consider the impact of stakeholders on evaluations.…”

“…An important part of transparency is direct feedback, and Rehbinder et al (2001) claim that an evolving evaluation framework can act as an effective feedback mechanism. Brandon (1998) found, from studies in the field of educational programmes, that stakeholder participation improves acceptance among stakeholders that are not directly involved in the evaluation process because they know that their colleagues have been involved. There is an important distinguished role of the evaluator identified in the literature, which has been characterised as a "key role" in IS evaluation (Serafeimidis and Smithson, 2003, p. 270).…”