Stability of School Building Accountability Scores and Gains

What does the National High School Exam (ENEM) tell Brazilian society?

“…There are data showing that few schools have results that are consistent (across different students) and stable (through time) (Thomas and others, 1997). Various studies have calculated the correlation coefficient of the school effect in different years, reporting coefficients ranging from zero (Linn and Haug, 2002) to around 0.6 (Mandeville, 1988); while (Luyten, 1994) finds correlation coefficients that are always between 0.35 and 0.65 for primary schools and between 0.70 and 0.95 in the case of high schools. Here again, it is worth asking to what extent the enem microdata confirm these results.…”

Precision and Volatility in School Accountability Systems

“…In particular, Kane and Staiger (2002) and Linn and Haug (2002) argued that year-to-year changes in scores for groups of students are extremely unstable, and therefore, prone to misuse and misinterpretation. Using state-based data, Kane and Staiger (2002) illustrated the volatility of test score measures as tools for measuring change at the school level and suggested implications that were severe and alarming:…”

“…

AbstractThis paper contrasts the positions of Kane and Staiger (2002) and Linn and Haug (2002) with that of Rogosa (2002Rogosa ( , 2003a on the volatility in test scores as measures of school growth. In particular, these researchers disagree on whether school growth can be measured reliably in school accountability systems.

…”

A Multilevel, Longitudinal Analysis of Middle School Math and Language Achievement

“…However, the use of different cohorts of students to measure school progress or school improvement is problematic for evaluative purposes. Recent investigations of the successive cohort approach demonstrate that estimates of year-to-year gains in proficiency are affected in large part by sampling variation, measurement error, and unique, non-persistent factors that are not associated with school size or school practice (Linn & Haug, 2002;Kane & Staiger, 2002). The lack of systematic variation in the successive cohort change score puts states at risk of assessing school performance on the basis of fluctuations in student cohorts or test administration conditions instead of actual changes in student performance (Linn & Haug, 2002).…”

“…Evidence that school performance cannot be estimated without bias when student test scores are aggregated at a single point in time or with precision when successive student cohorts are compared has led a number of authors to argue for the use of longitudinal analyses of individual student performance as a more direct and accurate estimate of school effects (Barton & Coley, 1998;Bryk & Raudenbush, 1988;Linn & Haug, 2002). For example, Goldstein (1991, p. 14), describing school effectiveness studies in Britain, stated that "...It is now recognised...that intake achievement is the single most important factor affecting subsequent achievement, and that the only fair way to compare schools is on the basis of how much progress pupils make during their time at school."…”