Evaluating the evidence in evidence-based policy and practice: Examples from systematic reviews of literature

Abstract

The full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-prot purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. AbstractWith the push for evidence-informed policy and practice, schools and policy makers are now increasingly encouraged and supported to use and enagage with research evidence. This means that consumers of research will now need to be discerning in judging the quality of research evidence that will inform their decisions. This paper evaluates the quality of evidence behind some well-known education programmes using examples from previous reviews of over 5,000 studies on a range of topics. It shows that much of the evidence is weak, and fundamental flaws in research are not uncommon. This is a serious problem if teaching practices and important policy decisions are made based on such flawed evidence. Lives may be damaged and opportunities missed. The aim of this paper is to show how widespread this problem is and to suggest ways by which the quality of education research may be improved. For example, funders of research and research bodies need to insist on quality research and fund only those that meet the minimum quality criteria. Journal editors and reviewers need to be cognizant of fundamental flaws in research and reject such submissions. One way to do this is to encourage submission of the research design and research protocol prior to acceptance, so acceptance or rejection is based on the design and not on the outcomes. This helps prevent publication bias and biased reporting. Individual researchers can improve quality by making it their moral responsibility to be truthful and transparent.Keywords: Evidence-based policy and practice, assessing trustworthiness of research evidence, systematic reviews INTRODUCTIONThis paper addresses the question of the validity of evidence that informs policy and practice and argues that more needs to be done to ensure better quality control of published research.

Socioeconomic status and school absenteeism: A systematic review and narrative synthesis

“…Due to the vast heterogeneity of studies regarding study focus (primary vs. secondary), SES and school absenteeism measures, design and statistical methodology, we refrained from using meta‐analytic techniques as this may lead to misleading conclusions (Cheung & Slavin, 2016; See, 2018). Instead, we adopted a narrative synthesis for summarising our findings.…”

A systematic review of the impact of technology-mediated parental engagement on student outcomes

“…Previous reviews also tended to summarise the findings of existing research or average the effect sizes of the individual studies being synthesised, a practice Slavin called, "muddling meta-analysis" (Slavin, 2020a). Such reviews may give misleading conclusions because weak studies often report big effect sizes (See, 2018). Starkey et al (2018), for example, reviewed studies that measured the educational value of home internet access, and only provided various classifications of the studies but did not evaluate the strength of the evidence.…”

The post-truth tyrannies of an evidence-based hegemony

“…Education departments in the United States and the United Kingdom have also transitioned their policymaking and its associated rhetoric to evidence-based practice (Haskins & Margolis, 2014;See, 2018). In the United States, the No Child Left Behind Act recommended the use of practices informed by scientifically based research (Slavin, 2016).…”