Evaluating and Using Observational Evidence: The Contrasting Views of Policy Makers and Epidemiologists

Abstract

BackgroundCurrently, little is known about the types of evidence used by policy makers. This study aimed to investigate how policy makers in the health domain use and evaluate evidence and how this differs from academic epidemiologists. By having a better understanding of how policy makers select, evaluate, and use evidence, academics can tailor the way in which that evidence is produced, potentially leading to more effective knowledge translation.MethodsAn exploratory mixed-methods study design was used. Quantitative measures were collected via an anonymous online survey (n = 28), with sampling from three health-related government and non-government organizations. Semi-structured interviews with policy makers (n = 20) and epidemiologists (n = 6) were conducted to gather qualitative data.ResultsPolicy makers indicated systematic reviews were the preferred research resource (19%), followed closely by qualitative research (16%). Neither policy makers nor epidemiologists used grading instruments to evaluate evidence. In the web survey, policy makers reported that consistency and strength of evidence (93%), the quality of data (93%), bias in the evidence (79%), and recency of evidence (79%) were the most important factors taken into consideration when evaluating the available evidence. The same results were found in the qualitative interviews. Epidemiologists focused on the methodology used in the study. The most cited barriers to using robust evidence, according to policy makers, were political considerations (60%), time limitations (55%), funding (50%), and research not being applicable to current policies (50%).ConclusionThe policy maker’s investigation did not report a systematic approach to evaluating evidence. Although there was some overlap between what policy makers and epidemiologists identified as high-quality evidence, there was also some important differences. This suggests that the best scientific evidence may not routinely be used in the development of policy. In essence, the policy-making process relied on other jurisdictions’ policies and the opinions of internal staff members as primary evidence sources to inform policy decisions. Findings of this study suggest that efforts should be directed toward making scientific information more systematically available to policy makers.

Natural experiment methodology for research: a review of how different methods can support real-world research

“…historically there was no evidence to support the effectiveness of policies around smoke-free restaurants and bars until researchers evaluated the impact of such policies within communities that were early adopters of such policies). Considering policy makers often rely on the evidence derived from other similar jurisdictions to inform their own policy development, evidence derived from natural experiments can be highly desirable (O'Donoughue-Jenkins, Kelly, Cherbuin, & Anstey, 2016). As such, robust evaluations of ongoing natural experiments have a lot to offer the scientific community across multiple disciplines more broadly.…”

“…For example, in the school-based tobacco control literature, evidence has shown that tobacco retailer density surrounding a school is associated with youth smoking behaviour (Chan & Leatherdale, 2011), but the impact of tobacco retailer density surrounding a school has not been controlled for in the RCTs exploring the effectiveness of school-based tobacco prevention programming (Thomas, McLellan, & Perera, 2013). Although natural experimental studies are more susceptible to bias and confounding (internal validity) (Craig et al, 2012), when designed appropriately, it is possible to maintain robust internal validity while also generating evidence with robust external validity that decision-makers and stakeholders often find more meaningful (Fong et al, 2006;Green, 2006;Leatherdale, 2012;O'Donoughue-Jenkins et al, 2016).…”

“…Second is improving the strength of evidence about policy effectiveness. Government policy makers are concerned about policy effectiveness in real-world conditions (O'Donoughue-Jenkins et al, 2016) and stakeholders commonly report that when it comes to developing policy, they require the best available evidence as opposed to the most rigorous evidence possible (Brownson et al, 2009). Although conclusions regarding policy impact derived from natural experimental evaluations should be drawn cautiously (Craig et al, 2011), when appropriately designed and implemented, natural experiment evaluations of policy can provide robust evidence of policy impact.…”

A Knowledge Brokering Framework for Integrated Landscape Management

“…For example, a survey of decision makers in the health sector found that systematic reviews are a particularly well-favoured source of evidence, followed closely by quantitative research. Evidence strength and consistency, data quality, bias and publication date were all cited as key factors affecting the influence of a study on policy (O'Donoughue Jenkins et al, 2016). The weight attached to research evidence depends to a great extent on the design of studies.…”

“…Time, ease of access, costs and institutional support are cited as common constraints by policymakers in various sectors across many countries (DFID, 2013;Oliver et al, 2014;Ntshotsho et al, 2015;O'Donoughue Jenkins et al, 2016;Rose et al, 2017;Waqa et al, 2017). The relevance of research evidence is also a barrier to its use (Oliver et al, 2014;O'Donoughue Jenkins et al, 2016). Sometimes studies are not well-aligned with the actual policy options that are being proposed or do not cover a wide enough range of outcomes.…”