Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?

Abstract

There are a growing number of large-scale educational Randomized Controlled Trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the EEF (UK) and the NCEE (US) which evaluated interventions aimed at improving academic achievement in K-12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations (SDs). These sat within relatively large confidence intervals (mean width 0.30 SDs) which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects. Rigorous Large-Scale Educational RCTs are Often Uninformative: Should We Be Concerned? Large-scale Randomized Controlled Trials (RCTs) are now regularly used to evaluate educational interventions. For example, the US-based National Center for Educational Evaluation and Regional Assistance (NCEE) started funding large-scale RCTs in 2002, and the UK-based Education Endowment Foundation (EEF) has funded more than 160 since 2012. This trend is not limited to these two countries: in recent years funding organizations in the European Union (e.g., European Schoolnet), Japan (e.g., Nippon Foundation), Australia (e.g., Social Ventures), Switzerland (e.g., Jacob's Foundation), Brazil (e.g., Lemann Foundation) and Bangladesh (e.g., BRAC) have also prioritized RCTs in education. Evaluating the efficacy of educational programs before implementation is important to avoid wasting resources. In medicine, there are many instances where RCTs have shown that promising treatments were ineffective or harmful (Sibbald & Roland, 1998). However, conducting large-scale RCTs is expensive. For example, the EEF spends around £500,000 per trial (EEF, 2015a). Given the growing number of large-scale RCTs in education, and their expense, it is important to reflect on how informative this new research focus has been. To our knowledge, no study has systematically evaluated this recent trend. In this paper we use empirical data from two prominent educational funding bodies to evaluate the typical effects produced by large-scale educational RCTs. Our aim is to provide an empirical basis for discussions of the field's efforts to build rigorous scientific evidence. Randomized Control Trials RCTs are widely regarded as the 'gold standard' for measuring the efficacy of interventions (Pocock, 1983). In their simplest form, participants are randomly assigned to an experimental group which receives the intervention, or a control group that receives an alternative treatment or possibly no treatment. The effectiveness of the intervention is then determined by comparing the outcomes between groups. RCTs are highly regarded because, compared with other types of studies (e.g., case studies), they ensure that the groups are probabilistically identical at the outset and that any difference in outcome are...

Uncovering Individualised Treatment Effect: Evidence from Educational Trials

“…Our re-analysis shows that, only 6 out of the 84 outcomes are statistically significant for FSM students, which means, consistent with the literature on overall effect sizes of EEF studies 22 , conventional regression analyses focusing on ATEs often produces results that are non-actionable, even when participants' lived experiences show otherwise 23 . One reason, in addition to those given elsewhere 7,22 , this may be the case is that we conduct the same analysis across all studies, thereby holding all analyses to the same standards. Arguably, of course, our analyses following some conventional approaches to subgroup analyses are not necessarily the "correct" ones.…”

“…With ITEs from the re-trained RF, we visualise in Figure 2 (B) the proportions of FSM and Non-FSM students at different ITE thresholds, which are set according to what we know from the literature and understand about the trial under investigation. For instance, either the overall effect size or that for FSM students reported by the original evaluation team is 0.01 23 , and the average of all the EEF trials funded to date is about 0.05 22 . Other values in the figure reflect the distribution of our predicted ITEs.…”

Education Trials and Tribulations

“…A study by Hugues Lortie‐Forgues of the University of York and Matthew Inglis of Loughborough University suggests it may be more the latter than the former 6 . The researchers evaluated 98 EEF trials and 56 NCEE trials and reported that the average effect size of these trials – that is, the mean standardised difference between the control and the intervention groups – was close to zero (specifically, 0.06).…”

Open Science 2.0: Large-Scale Collaborative Education Research

“…Evidence suggests that the majority of education randomized control trial results (the "gold standard" of research) do not have sufficient precision to assess whether the intervention was effective (Lortie-Forgues & Ingles, 2019). Worse yet, replication has been a low priority.…”