What counts as evidence for policy? An analysis of policy actors' perceptions

MacKillop, Eleanor; Downe, James Daniel

doi:10.1111/puar.13567

Cited by 9 publications

(7 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings also speak to ongoing disputes between political and ‘pragmatic’ (functional) perspectives on evidence‐based policymaking (Cairney, 2016; see MacKillop & Downe, 2022; Sanderson, 2002) by revealing the mechanisms that may lead to evaluation as evidence being perceived in either way. An overly positive portrayal of evaluated activities may undermine trust in evaluation results.…”

Section: Discussionsupporting

confidence: 52%

“…Theoretically , we speak to ongoing disputes between political and ‘pragmatic’ (functional) perspectives in debates on evaluation and, more broadly, evidence‐based policymaking (Cairney, 2016; see MacKillop & Downe, 2022; Sanderson, 2002). We do so by revealing the conditions under which the functional or the political perspective on such a technocratic tool like evaluation is likely to emerge.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Institutional design and biases in evaluation reports by international organizations

Eckhard

Jankauskas

Leuschner

2023

Public Administration Review

View full text Add to dashboard Cite

Governments spend hundreds of millions on evaluations to assess the performance of public organizations. In this article, we scrutinize whether variation in the institutional design of evaluation systems leads to biases in evaluation findings. Biases may emerge because influence over evaluation processes could enable the bureaucracy to present its work in a more positive way. We study evaluation reports published by nine international organizations (IOs) of the United Nations system. We use deep learning to measure the share of positive assessments at the sentence level per evaluation report as a proxy for the positivity of evaluation results. Analyzing 1082 evaluation reports, we find that reports commissioned by operative units, as compared to central evaluation units, systematically contain more positive assessments. Theoretically, this link between institutional design choices and evaluation outcomes may explain why policymakers perceive similar tools for evidence‐based policymaking as functional in some organizations, and politicized in others.

show abstract

Section: Discussionsupporting

confidence: 52%

Section: Introductionmentioning

confidence: 99%

Institutional design and biases in evaluation reports by international organizations

Eckhard

Jankauskas

Leuschner

2023

Public Administration Review

View full text Add to dashboard Cite

show abstract

“…This means that a SIB model with stewardship or post‐NPM characteristics could inadvertently undermine measurement, making it impossible to have the methodologically rigorous evaluations called for in the literature (Heinrich & Kabourek, 2019). Perceptions of evaluation reliability are related to factors like political context, policy area, and the reputation of those conducting evaluations (MacKillop & Downe, 2022; Schmidthuber, Willems, & Krabina, 2022), but actual reliability can also be related to service contract conditions.…”

Section: Discussionmentioning

confidence: 99%

The impact of social impact bond financing

Hevenstone

Fraser

Hobi

et al. 2023

Public Administration Review

View full text Add to dashboard Cite

Social impact bonds (SIBs), also known as Pay for Success, are an innovation in Payment by Results contracting. Investors finance programs and are repaid based on the “SIB effect,” which includes changes in outcomes attributable to financing. We generate a quantitative estimate of this part of the SIB effect for two active labor market programs in the Netherlands and Switzerland. Comparing program impacts within providers using SIB and non‐SIB contracts suggests financing has positive impacts on public benefit receipt, employment, and income. Qualitative research suggests this is because SIB contracts increased pressure for all involved parties, leading to the institutionalization of selection and greater resources for SIB‐financed services. Contracts with high pressure, like SIBs, may compromise both performance requirements and the potential to measure performance. We examine the implications of these findings in relation to agency and stewardship theories and highlight the significance of SIBs as multilateral as opposed to bilateral contracts.

show abstract

“…There is no agreement over how to define it, even though this matters for what types of evidence are used in policy decisions and what kinds of evidence are ignored. Existing research has shown that there are three dominant discourses for understanding evidence which can be identified from the literature (MacKillop and Downe, 2023). First, an evidence-based policy-making (EBPM) discourse which views evidence as the solution to poorly designed policies.…”

mentioning

confidence: 99%

Do policy actors have different views of what constitutes evidence in policymaking?

Piddington,

MacKillop,

Downe

2024

Policy &Amp; Politics

View full text Add to dashboard Cite

The role played by evidence in policymaking is hotly disputed and there is no agreement over how evidence is defined. This article examines whether policy actors have different views of what counts as evidence and which factors influence these perceptions (for example, professional background, length of service, organisation setting, cultures of evidence)? In addressing this question, we contribute to the growing research focus on the context of evidence use. Q methodology – a mixed method approach to study people’s attitudes towards a topic – is used in interviewing 67 policy actors and comparing two countries, Scotland and Wales, to find out whether there are different cultures of evidence. In both countries, we identified four distinct profiles of attitudes towards evidence: the evidence-based policymaking (EBPM) Idealist, the Pragmatist, the Inclusive, and the Political. Our research highlights important differences between the two contexts, with a greater leaning towards EBPM views of evidence in Wales, and more pragmatism in defining evidence in Scotland. We illustrate how different cultures of evidence coexist in a same context and highlight their similarities and differences. We also contribute to the understanding of the value of Q methodology research by showing that it can be used to compare two datasets collected in different countries.

show abstract

What counts as evidence for policy? An analysis of policy actors' perceptions

Cited by 9 publications

References 48 publications

Institutional design and biases in evaluation reports by international organizations

Institutional design and biases in evaluation reports by international organizations

The impact of social impact bond financing

Do policy actors have different views of what constitutes evidence in policymaking?

Contact Info

Product

Resources

About