Phantom effects in school composition research: consequences of failure to control biases due to measurement error in traditional multilevel models

Abstract

The main objective of this study was to quantify the impact of failing to account for measurement error on school compositional effects. Multilevel structural equation models were incorporated to control for measurement error and/or sampling error. Study 1, a large sample of English primary students in Years 1 and 4, revealed a significantly negative compositional effect associated with school-average achievement that became more negative after controlling for measurement error. Study 2, a large study of Cypriot primary students in Year 4, showed a small, positive but statistically significant compositional effect that became non-significant after controlling for measurement error. Our findings have important methodological, substantive, and theoretical implications for on-going debates on the school compositional effects on students' outcomes, because nearly all previous research has been based on traditional approaches to multilevel models, which are positively biased, due to the failure to control for measurement error.

English Value‐Added Measures: Examining the Limitations of School Performance Measurement

“…This particular problem highlights the ‘fragility’ of causal inferences based on VA measures (Marsh et al ., , p. 282) introduced earlier. It might be that grammar schools are more effective; this may be a compositional effect stemming from the grouping of higher ability pupils or it could be a ‘phantom’ effect arising from poor quality data or unobserved non‐school factor bias (Harker & Tymms, , Televantou et al ., ). As Harker and Tymms (, p. 195) note, ‘the really worrying thing is that the researcher can never be sure about what has been found’.…”

Reframing the English grammar schools debate

“…the effect of having more-able peers over and above individual ability) are consistently found to be very small (see Gibbons and Telhaj 2012, which summarises previous studies in Table 1) and more recent studies are less likely to pick one up at all (Yeung and Nguyen-Hoang 2016). Studies have also found that measurement error can lead to 'phantom' compositional effects, inflating value-added scores for schools with high-ability intakes (Harker and Tymms 2004;Televantou et al 2015).…”

Do peer effects mediate the association between family socio‐economic status and educational achievement?

“…For example, some researchers have argued that school compositional effects are phantom effects related to sampling error (Pokropek ; Televantou et al. ).…”

“…There is also some debate in the literature regarding the size of school SES effects with some researchers arguing that school SES effects are either small or non‐existent (Marks ; Pokropek ; Televantou et al. ), whereas other researchers argue that the SES of the school population is associated with levels of academic achievement net of individual factors (Rothman & McMillan ; Sirin ; McConney & Perry ; Organisation for Economic Co‐operation and Development (OECD) ; Perry & McConney ; Buckingham et al. ; Lim et al.…”