What exactly do RCT findings tell us in education research?

Abstract

This article addresses issues related to whether null randomised control trial (RCT) findings can by themselves be a secure indicator of programme failure. This is done by drawing on the findings of the evaluation of the Integrated Group Reading (IGR) programme using a number of teacher case studies. The case studies illustrate how the same intervention can be implemented differently in local circumstances, with different outcomes. The different ways in which IGR was implemented reflect how teachers experienced the pressures of the national curriculum, their attitudes to the IGR approach to reading, the school ethos and the resources and support available-and point to how IGR use might be enhanced to result in more significant reading gains. The article argues that in addition to the statistical findings, evaluators ought to pay attention to the context in which a programme is implemented, especially when it comes to complex interventions trialled in real classrooms. It is also concluded that it is preferable to avoid asking whether a programme works or not for all, and under any circumstances. A focus on the different ways that programmes work under different circumstances, and when implemented by different people, is a more useful perspective. This might not provide the certainty that policy-makers would likely opt for, but it captures better the complexity associated with teaching programme evaluation.

The validity and reliability of observational assessment tools available to measure fundamental movement skills in school-age children: A systematic review

“…The TGMD was the assessment tool with the most consistently positive evidence in favour of validity and reliability. However, it is important to consider the suitability of observational assessment tools for use in schools, alongside the evidence for the psychometric properties of measures [158]. Recent research by Klingberg et al established a framework to evaluate the feasibility of implementing FMS assessments in schools [22].…”

“…AST) need more research to be done to establish psychometric properties. Currently, schools would have to choose an assessment tool based on either feasibility or strong psychometric evidence alone, however, it is known from educational research that there needs to be a trade-off between the two for school-based initiatives to be implemented consistently, and effective [158].…”

Experiment’s persistent failure in education inquiry, and why it keeps failing

“…Karlsson and Bergmark (2015) too have commented on the problematics of ‘compared with what?’ The delivery of education interventions is far more complex and open to variation than the administration of a pill. Koutsouris and Norwich (2018), for example, show how the same ‘experimentally tested’ intervention was implemented wholly differently in differing local circumstances (see also Goodman et al ., 2018). Their findings validate Tobin’s (2005) and Biesta’s (2015) observations that teachers are co‐constructors of change, not identical delivery agents.…”

Developing and validating a school-based screening tool of Fundamental Movement Skills (FUNMOVES) using Rasch analysis

“…It is important to consider the feasibility of FUNMOVES for use in schools alongside the results demonstrating structural validity, as it is known from educational research that there needs to be a trade-off between feasibility and validity/reliability in order for school-based initiatives to be implemented consistently and effectively [61]. Klingberg et al outlined seven criteria that assessments should meet in order to be feasible for use in schools [34].…”