Sources of bias in outcome assessment in randomised controlled trials: a case study

Ainsworth, Hannah; Hewitt, Catherine; Higgins, Steve; Wiggins, Andy; Torgerson, David; Torgerson, Carole

doi:10.1080/13803611.2014.985316

Cited by 10 publications

(26 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The effect size (in the form of standardised mean difference) for the PIM test was 0.33 and for the SENT-R-B test was 1.11. Ainsworth et al (2015) is predicated on the assumption that these numbers are estimates for the effectiveness of the intervention, should be expected to be similar and, therefore, that the apparently large difference needs explanation. Ainsworth et al (2015) suggests three alternative explanations for the difference in effect size: the timing of the tests, their treatment inherence and the nature of masking.…”

Section: Contextmentioning

confidence: 99%

“…Ainsworth et al (2015) is predicated on the assumption that these numbers are estimates for the effectiveness of the intervention, should be expected to be similar and, therefore, that the apparently large difference needs explanation. Ainsworth et al (2015) suggests three alternative explanations for the difference in effect size: the timing of the tests, their treatment inherence and the nature of masking. The paper presents various arguments as evidence in favour or against each alternative, concluding that "the evidence from this study suggests that the difference in effect size between the primary and secondary outcome is probably due to lack of blinding and nonindependence of teachers administering the tests" (p.12), albeit maintaining that the only way to be sure of this explanation would be to conduct an RCT comparing masked and nonmasked assessor conditions.…”

Section: Contextmentioning

confidence: 99%

“…In education, it is likely to be impossible to mask many key groups of people involved in a study, but it could be argued that we should still do the best we can and mask wherever possible and it is important to understand the potential consequences which may result when the mask slips. Ainsworth et al (2015) details an interesting study in which two different outcome measures were used: one in which the administrators and markers were masked to treatment assignment and another where they were not. The two different assessments were associated with very different standardised effect sizes: the unmasked assessor condition had an effect size more than three times that of the masked assessor condition.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Unmasking the unasked: correcting the record about assessor masking as an explanation for effect size differences

Simpson

2018

Educational Research and Evaluation

View full text Add to dashboard Cite

Ainsworth et al.'s paper "Sources of bias in outcome assessment in randomised controlled trials: a case study" examines alternative accounts for a large difference in effect size between two outcomes in the same intervention evaluation. It argues that the probable explanation relates to masking: only one outcome measure was administered by those aware of participants' treatment assignment. This paper shows this conclusion is not substantiated by the evidence: the original paper fails to exclude alternative explanations and what it takes as positive evidence for the preferred explanation is actually negative. While accepting the importance of masking in RCTs, this paper concludes that the original question was based on a misconception about effect sizes: seen correctly as a measure of whole study design, the question of effect size difference between different outcome measures does not need asking.

show abstract

Section: Contextmentioning

confidence: 99%

Section: Contextmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Unmasking the unasked: correcting the record about assessor masking as an explanation for effect size differences

Simpson

2018

Educational Research and Evaluation

View full text Add to dashboard Cite

show abstract

“…It is the interaction of the virus with the body's physiology and the treatment. Also, being a randomised intervention, there is a claim that the results have more general applicability and a high degree of validity (Ainsworth et al 2015). This works for things like viruses because they do not have a mind of their own but are of the same 'mind' (e.g.…”

Section: My Virus Has a Mind Of Its Own -On The Ontological Question mentioning

confidence: 99%

“…For example … the Bush government in the USA attempted to construct 'empirically randomised control trials' as the 'gold standard' for assessing educational research and for evaluating all research applications … The situation in the UK has been similar. (Lingard & Gale 2010: 33) The reasons for doing so are (1) ostensibly 'grounded in political and professional concerns about underachievement and educational equity' (Chapter 8) and (2) for purposes of efficiency and effectiveness, so that 'time and money were not wasted on irrelevant or ineffective strategies' (Chapter 8); (3) both pursued in the belief that much education research is unscientific and based in values, opinion and 'bias' that have a disregard for evidence (Hammersley 2005;Ainsworth et al 2015;Pring 2015). In fact, advocates of RCTs at government level have called for their use in educational research precisely because they are presumed to yield knowledge superior and more reliable than other methods (e.g.…”

Section: Doing a Bradbury -On The Ethical Questions Of Distinction Anmentioning

confidence: 99%

What’s not to like about RCTs in education?

Gale¹

2017

Mobilising Teacher Researchers

View full text Add to dashboard Cite

Mobile device use in the primary school classroom and impact on pupil literacy and numeracy attainment: A systematic review

Dorris,

Winter,

O'Hare

et al. 2024

Campbell Systematic Reviews

View full text Add to dashboard Cite

BackgroundInvestment in mobile devices to support primary or elementary education is increasing and must be informed by robust evidence to demonstrate impact. This systematic review of randomised controlled trials sought to identify the overall impact of mobile devices to support literacy and numeracy outcomes in mainstream primary classrooms.ObjectivesThe aim of this systematic review was to understand how mobile devices are used in primary/elementary education around the world, and in particular, determine how activities undertaken using mobile devices in the primary classroom might impact literacy and numeracy attainment for the pupils involved. Within this context, mobile devices are defined as tablets (including iPads and other branded devices), smartphones (usually those with a touchscreen interface and internet connectivity) and handheld games consoles (again usually with touchscreen and internet‐enabled). The interventions of interest were those aimed at improving literacy and/or numeracy for children aged 4–12 within the primary/elementary school (or equivalent) classroom.Specifically, the review aimed to answer the following research questions: What is the effect of mobile device integration in the primary school classroom on children's literacy and numeracy outcomes? Are there specific devices which are more effective in supporting literacy and numeracy? (Tablets, smartphones, or handheld games consoles) Are there specific classroom integration activities which moderate effectiveness in supporting literacy and numeracy? Are there specific groups of children for whom mobile devices are more effective in supporting literacy and numeracy? (Across age group and gender). Do the benefits of mobile devices for learning last for any time beyond the study? What is the quality of available evidence on the use of mobile devices in primary/elementary education, and where is further research needed in this regard?An Expert Advisory Group supported the review process at key stages to ensure relevance to current practice.Search MethodsThe search strategy was designed to retrieve both published and unpublished literature, and incorporated relevant journal and other databases with a focus on education and social sciences. Robust electronic database searches were undertaken (12 databases, including APA PsychInfo, Web of Science, ERIC, British Education Index and others, and relevant government and other websites), as well as a hand‐search of relevant journals and conference proceedings. Contact was also made with prominent authors in the field to identify any ongoing or unpublished research. All searches and author contact took place between October and November 2020. The review team acknowledges that new studies will likely have emerged since and are not captured at this time. A further update to the review in the future is important and would build on the evidence reflected here.Selection CriteriaThe review included children within mainstream primary/elementary/kindergarten education settings in any country (aged 4–12), and interventions or activities initiated within the primary school classroom (or global equivalent) that used mobile devices (including tablets, smartphones, or hand‐held gaming devices) to intentionally support literacy or numeracy learning. In terms of study design, only Randomised Controlled Trials were included in the review.Data Collection and AnalysisA total of 668 references were identified through a robust search strategy including published and unpublished literature. Following duplicate screening, 18 relevant studies, including 11,126 participants, 14 unique interventions, and 46 relevant outcome measures were synthesised using Robust Variance Estimation and a random effects meta‐analysis model. Risk of Bias assessment was undertaken by three reviewers using the ROB2 tool to assess the quality of studies, with 13 studies rated as having some concerns, and 5 as having high risk of bias. Qualitative data was also extracted and analysed in relation to the types of interventions included to allow a comparison of the key elements of each.Main ResultsA positive, statistically significant combined effect was found (Cohen's d = 0.24, CI 0.0707 to 0.409, p < 0.01), demonstrating that in the studies and interventions included, children undertaking maths or literacy interventions using mobile devices achieved higher numeracy or literacy outcomes than those using an alternative device (e.g., a laptop or desktop computer) or no device (class activities as usual). However these results should be interpreted with caution given the risk of bias assessment noted above (5 studies rated high risk of bias and 13 rated as having some concerns). As the interventions and classroom circumstances differed quite widely, further research is needed to understand any potential impact more fully.Sensitivity analysis aimed to identify moderating factors including age or gender, screen size, frequency/dosage of intervention exposure, and programme implementation features/activities (based on Puentedura's [2009] SAMR model of technology integration). There were too few studies identified to support quantitative analysis of sufficient power to draw robust conclusions on moderating factors, and insufficient data to determine impact beyond immediate post‐test period. Sensitivty analysis was also undertaken to exclude the five studies identified as having a high risk of bias, to identify any impact they may have on overall findings.Authors' ConclusionsOverall, this review demonstrates that for the specific interventions and studies included, mobile device use in the classroom led to a significant, positive effect on literacy and numeracy outcomes for the children involved, bringing positive implications for their continued use in primary education. However given the concerns on risk of bias assessment reported above, the differing circumstances, interventions and treatment conditions and intensities, the findings must be interpreted with caution. The review also supports the need for further robust research to better understand what works, under what circumstances, and for whom, in the use of mobile devices to support learning.

show abstract

Sources of bias in outcome assessment in randomised controlled trials: a case study

Cited by 10 publications

References 14 publications

Unmasking the unasked: correcting the record about assessor masking as an explanation for effect size differences

Unmasking the unasked: correcting the record about assessor masking as an explanation for effect size differences

What’s not to like about RCTs in education?

Mobile device use in the primary school classroom and impact on pupil literacy and numeracy attainment: A systematic review

Contact Info

Product

Resources

About