Reviewing the evidence: reflections from experience

Abstract

English Evidence-based policy (EBP), along with the ‘systematic review’, has recently emerged as a prominent strand within social science and public policy research. A number of articles have heralded this emergence but the vast majority are theoretical and concerned with how EBP research could be practised. This article reflects on the author’s experiences of identifying and reviewing evidence on the effectiveness of labour market interventions aimed at people with a disability or a chronic illness, to demonstrate some of the methodological challenges and practical problems that arise from the actual conduct of EBP research.

Real world reviews: a beginner's guide to undertaking systematic reviews of public health policy interventions

“…For example, in the two systematic reviews of shift work interventions, although 27 different databases were searched, 40 of the 66 included studies were found in either Medline or Embase, and half (13) of the others were found by citation follow-up (box 1). 17 18 Similarly, in the systematic review of welfare to work interventions, only three were located by electronic searches, with the rest found by specialist website searches and citation follow-up 14 21. Evidence from other reviews also supports the value of website searches and citation follow-up in locating new studies 36…”

“…It should define the intervention of interest, the population receiving the intervention, the outcome(s) of interested, and the study designs deemed worthy of inclusion 11. However, the use of systematic reviews in evaluating public health and social interventions has highlighted problems with this approach as policy or practice questions may well be broader and the answer may cover more than one intervention 21. For example, in the welfare to work review (box 1) the policy (or intervention) under review was ‘welfare to work’; however, this covers a multitude of different types of intervention (such as antidiscrimination legislation, vocational rehabilitation, return to work credits, etc), which may operate on the outcome in different ways 14.…”

“…Supplemental searches (hand search, websites, citation follow-up, or expert contact) are essential and may in some cases even further negate the need for a broad ‘purist’ search of the electronic databases, especially in systematic reviews of social interventions in which the grey literature is likely to be a prominent source of studies 21. For example, in the two systematic reviews of shift work interventions, although 27 different databases were searched, 40 of the 66 included studies were found in either Medline or Embase, and half (13) of the others were found by citation follow-up (box 1).…”

Supply chain management research impact: an evidence‐based perspective

“…However, despite the criticisms, evidence-based approaches are now central to medical research and practice, and RCTs remain the gold standard, with qualitative research evidence still being viewed by many in the medical profession as poorer quality research. Evidence-based policy research became established in the late 1990s, seeking to provide evidence about the operation, impact, outcome and effectiveness of public policy interventions, for the purpose of informing public policy formulation and implementation (Bambra, 2005). Policies should be effective in their use of rationed resources, particularly time and finance (Davies and Smith, 1999;Macintyre and Petticrew, 2000;Campbell Collaboration, 2001).…”

Systematic reviews in social policy: to go forward, do we first need to look back?

“…Victora et al (2004) make a similar case in respect of evidence for public health interventions, arguing that evaluations of interventions in a variety of everyday settings accompanied by a plausible rationale for why the outcomes observed occurred, would better inform policy and practice than studies with the highest internal validity. Bambra (2005) highlights the difficulties inherent in attributing impact to specific policies (and thus their potential impact in other contexts) when the environment in which these policies are evaluated is already shaped by other policies that are themselves evolving. In a similar vein, Wallace et al (2006) point to the significant social and economic changes that can take place over time and which may confound external validity.…”