Whom Do Bureaucrats Believe? A Randomized Controlled Experiment Testing Perceptions of Credibility of Policy Research

Abstract

More than ever before, analysts in government have access to policy-relevant research and advocacy, which they consume and apply in their role in the policy process. Academics have historically occupied a privileged position of authority and legitimacy, but some argue this is changing with the rapid growth of think tanks and research-based advocacy organizations. This article documents the findings from a randomized controlled survey experiment using policy analysts from the British Columbia provincial government in Canada to systematically test the source effects of policy research in two subject areas: minimum wage and income-splitting tax policy. Subjects were asked to read research summaries of these topics and then assess the credibility of each article, but for half of the survey respondents the affiliation/ authorship of the content was randomly reassigned. The experimental findings lend evidence to the hypothesis that academic research is perceived to be substantially more credible than think tank or advocacy organization research, regardless of its content. That increasingly externalized policy advice systems are not a pluralistic arena of policy research and advice, but instead subject to powerful heuristics that bureaucrats use to sift through policy-relevant information and advice, demands added nuance to both location and content-based policy advisory system models.KEY WORDS: expert credibility, policy advisory systems, social science experimentation 384 0190-292X V C 2016 Policy Studies Organization

How Transparency and Reproducibility Can Increase Credibility in Policy Analysis: A Case Study of the Minimum Wage Policy Estimate

“…(Harvard Magazine, 2016) When high credibility cannot be obtained by directly assessing the intrinsic quality of a report, the reader has to rely on reputation as a proxy for quality. This is consistent with the evidence found by Doberstein (2016) where the same policy brief receives drastically different credibility scores, depending on the reputation of the authoring institution placed on the cover. The problem of low credibility in policy analysis is important because the output of these analyses is among the main inputs that policy makers use as evidence when designing policy.…”

“…If the quality of a report is not observable we should expect policy makers to put a large value on reputation as a proxy of quality. Doberstein (2016) finds evidence that the credibility of the same policy brief varies drastically when only the name of the authoring institution is replaced. This large reputational premium is evidence of how difficult it is to critically examined a given analysis on the basis of its own merits (sound reasoning, methodology, data and execution).…”

“…2016<-"51135-2016-01-Economic%20Projections.xlsx" late.2015<-"51135-2015-08-EconomicProjections.xlsx" early.2015<-"51135-2015-01-EconomicProjections.xlsx" late.2014<-"51135-2014-08-EconomicProjections.xlsx" early.2014<-"51135-2014-02-EconomicProjections.xlsx" early.2013<-"51135-2013 …”

A Framework for Open Policy Analysis

“…Moreover, without openness in policy analysis, the credibility of a report largely rests on the reputation of the analysts (Doberstein, 2017), in contrast to the dictum from the scientific principle of universalism that the strength of a claim should rest on the quality of the evidence rather than who is making the claim (Merton, 1973).…”

Understanding evidence and behavioral responses: Future directions in evidence‐based policy‐making

“…Through an experiment conducted with policy analysts in British Columbia, Doberstein () demonstrates that the source of evidence (for example, academic vs. think tanks) acts as a cue to shaping one's view of the credibility of the information at hand. This is an important finding in that it shows that assessing the credibility of evidence partly rests on an assessment of the provenance of the authority presenting the evidence.…”