Trust Matters: Distrust in an External Evaluation of a Public Sector Program

Wond, Tracey

doi:10.1080/01900692.2015.1126732

Cited by 5 publications

(6 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the Standards themselves barely mention generalisation, there is adequate justification for use of evaluation to generalise findings within and beyond a programme, as noted in Yin (2014), above. To some extent, credibility depends not only on credentials, but also on the trusted relationship between evaluators and commissioners (Wond, 2017). ‘Engaging in an effective evaluation that involves qualitative inquiry without developing and maintaining trusting relationships is inconceivable’ (Goodyear et al, 2014, p. 258).…”

Section: Literature Reviewmentioning

confidence: 99%

Generalising from qualitative evaluation

Guenther

Falk

2021

Evaluation Journal of Australasia

View full text Add to dashboard Cite

Evaluations are often focused on assessing merit, value, outcome or some other feature of a programme, project, policy or some other object. Evaluation research is then more concerned with the particular rather than the general – even more so, when qualitative methods are used. But does this mean that evaluations should not be used to generalise? If it is possible to generalise from evaluations, under what circumstances can this be legitimately achieved? The authors of this article have previously argued for generalising from qualitative research (GQR), and in this article, they extrapolate the discussion to the field of evaluation. First, the article begins with a discussion of the definitions of generalisability in research, recapping briefly on our arguments for GQR. Second, the differentiation between research and evaluation is explored with consideration of what literature there is to justify generalisation from qualitative evaluation (GQE). Third, a typology derived from the literature is developed, to sort 54 evaluation projects. Fourth, material from a suite of evaluation projects is drawn from to demonstrate how the typology of generalisation applies in the context of evaluations conducted in several fields of study. Finally, we suggest a model for GQE.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Generalising from qualitative evaluation

Guenther

Falk

2021

Evaluation Journal of Australasia

View full text Add to dashboard Cite

show abstract

“…Reasons for the significant turnover in village representatives at the formation of the program were not confirmed, though turnover was probably related to personal circumstances (e.g., competing priorities that hindered their participation), programmatic problems (e.g., dissatisfaction with the level of engagement), or both (P:3 -5). The timing of the evaluation, which occurred near the end of the five-year research agreement, limited opportunities for the evaluator to develop trusted relationships, observe participants, and develop a process to track community engagement and BSP-funded outputs more comprehensively (P:1); this is a common occurrence in evaluation (Wond, 2017).…”

Section: Processesmentioning

confidence: 99%

Knowledge Co-production in Contested Spaces: An Evaluation of the North Slope Borough – Shell Baseline Studies Program

Kettle¹

2019

ARCTIC

View full text Add to dashboard Cite

Supporting the development of trusted and usable science remains a key challenge in contested spaces. This paper evaluates a collaborative research agreement between the North Slope Borough of Alaska and Shell Exploration and Production Company—an agreement that was designed to improve collection of information and management of issues associated with the potential impacts of oil and gas development in the Arctic. The evaluation is based on six categories of knowledge co-production indicators: external factors, inputs, processes, outputs, outcomes, and impacts. Two sources of data were used to assess the indicators: interviews with steering committee members and external science managers (n = 16) and a review of steering committee minutes. Interpretation of the output and outcome indicators suggests that the Baseline Studies Program supported a broad range of research, though there were differences in how groups perceived the relevance and legitimacy of project outcomes. Several input, process, and external variables enabled the co-production of trusted science in an emergent boundary organization and contested space; these variables included governance arrangements, leveraged capacities, and the inclusion of traditional knowledge. Challenges to knowledge co-production on the North Slope include logistics, differences in cultures and decision contexts, and balancing trade-offs among perceived credibility, legitimacy, and relevance. Reinforced lessons learned included providing time to foster trust, developing adaptive governance approaches, and building capacity among scientists to translate community concerns into research questions.

show abstract

“…The under-utilisation and effectiveness of evaluation findings has frustrated both evaluation communities and those who fund them, expounding criticism that evidence-based policy and practice (as a mechanism that evaluation feeds) is ideological, flawed and failing (Parkhurst, 2017). Despite significant public expenditure being committed to construct such evidence bases (see for instance National Audit Office, 2013), the use of evidence to inform policy intervention is sporadic, and there are increasing accounts of the underuse and misuse of such evidence (Weiss, 1993;Wond, 2017). EBPP has endured a great deal of criticism, and despite maturing as a concept, in practice, it has struggled to become fully institutionalised or legitimated in many areas of public policy.…”

Section: Challenges Facing Evaluationmentioning

confidence: 99%

“…The diversity of stakeholders and the varying expectations and interests that stakeholders hold further complicates the evaluation setting (Mohan and Sullivan, 2006;Stufflebeam, 2001). In a program evaluation setting, stakeholders might include program managers, program delivery staff (these two groups form our policy implementers in this study), beneficiaries, policy-makers, commissioners, the evaluator, and academics (McCoy and Hargie, 2001;O'Brien et al, 2010;Stern, 2008;Wond, 2017). Whilst this makes for a complicated environment, Mohan and Sullivan (2006) highlight that stakeholder diversity can be beneficial, particularly to support evaluators to appreciate the varying perceptions that exist (beyond their own).…”

Section: Complex Evaluation Systemsmentioning

confidence: 99%

“…Program evaluators have the potential to influence decision-makers through their findings, for instance by stimulating program improvements, or providing evidence that supports or hinders continuation grant applications (Feinstein, 2002;Pollitt, 2003). This potential influence could lead to tension and influence trust within the program evaluation relationship (Roe, 1982;Wond, 2017): 'sometimes stakeholders are hypersensitive to evaluators because they perceive they have much to gain or lose based on the success of failure of the program' (Donaldson et al, 2002, p.267). Vice versa, evaluation stakeholders can also significantly influence the evaluation and evaluator (Patton, 1997).…”

Section: Complex Evaluation Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation for What Purpose? Findings From Two Stakeholder Groups

Wond

2018

Responsibility and Governance

Self Cite

View full text Add to dashboard Cite

A host of reasons exist for the pursuit of evidence in the public sector, including to support good governance and policy development. As the expectations for program evaluation from policymakers have evolved, so too has evaluation practice and a great deal of experimentalism has ensued. There is a risk that these developments and the complexity inherent in them, may lead to conflicting expectations about why program evaluation is done, or even a loss of purpose. This prompts the meso-level analysis of two types of stakeholders in a governance network, explored in this chapter. This chapter presents the findings of an ongoing study which explores the perceptions of program evaluators and policy implementers towards the purpose of evidence. The findings suggest that program evaluators and policy implementers have divergent expectations of why and how evaluation data might be used. The findings suggest that program evaluators aspire to support change and enhance the policy domains they serve, whereas policy implementers perceive program evaluation as serving a more governance-/managementorientated role. The chapter demonstrates the complexity of both program evaluation and policy and may have implications for the twin pillars of governance and responsibility at the heart of the book. If governance and responsibility are the twin pillars of sustainability then the complex networks of relationships, expectations, values, and outcomes may need to be considered. The findings also have implications for evaluation commissioners and practitioners, demonstrating the need for the purpose and expectations of program evaluation to be agreed early. The use of program evaluation as a symbolic, aesthetic or structural mechanism also emerges, prompting opportunity for further research, for instance, to explore legitimacy and program evaluation.

show abstract

Trust Matters: Distrust in an External Evaluation of a Public Sector Program

Cited by 5 publications

References 39 publications

Generalising from qualitative evaluation

Generalising from qualitative evaluation

Knowledge Co-production in Contested Spaces: An Evaluation of the North Slope Borough – Shell Baseline Studies Program

Evaluation for What Purpose? Findings From Two Stakeholder Groups

Contact Info

Product

Resources

About