Volatility in School Test Scores: Implications for Test-Based Accountability Systems

Kane, Thomas J.; Staiger, Douglas O.

doi:10.1353/pep.2002.0010

Cited by 138 publications

(145 citation statements)

References 5 publications

Supporting

Mentioning

133

Contrasting

Unclassified

Order By: Relevance

“…Over the last decade, a growing number of VAMs have been developed, and different model specifications have been explored, with the dual goal of understanding and improving the quality of teacher value-added scores (Ballou, 2005;Ballou et al, 2004;Briggs & Domingue, 2011;Buddin, 2010;Harris & Sass, 2010;Harris, Sass, & Semykina, 2012;Harris, 2011;Hill, et al, 2011;Kane & Staiger, 2002a;2002b;McCaffrey et al, 2004;McCaffrey, Sass, Lockwood, Milhaly, 2009;Papay, 2011;Rothstein, 2007;Schochet & Chiang, 2010). Yet, understanding which data and model specifications have a larger effect on teacher VAEs compared to others has been challenging because the number of model specifications and their possible combinations is too large to be fully investigated in any single study.…”

Section: Literature Reviewmentioning

confidence: 99%

Value-added Teacher Estimates as Part of Teacher Evaluations: Exploring the Effects of Data and Model Specifications on the Stability of Teacher Value-added Scores

Kersting

Chen

Stigler

2013

EPAA

View full text Add to dashboard Cite

If teacher value-added estimates (VAEs) are to be used as indicators of individual teacher performance in teacher evaluation and accountability systems, it is important to understand how much VAEs are affected by the data and model specifications used to estimate them. In this study we explored the effects of three conditions on the stability of VAEs and evaluated their relative impact. We varied how we accounted for differences among students in their prior learning; whether we epaa aapeValue-added teacher estimates as part of teacher evaluation 2 estimated VAEs from single or multiple cohorts of students; and the number of students contributing to the VAE for each teacher. Using data from one of the largest school districts in the nation, we created a single, complete data set and used it to estimate several sets of VAEs for each of the 3651 5th grade mathematics teachers. We found that approximately two thirds of teachers' were stable in that they remained in the same performance group across all conditions. We also found that differences in number of students used for VAEs accounted for up to one third of teacher reclassifications into different performance groups; single versus multiple cohort models accounted for about one fifth; and different methods for controlling for student prior learning accounted for about one sixth (16%) of teacher reclassifications. We relate our findings to characteristics of our data and discuss implications for educational policy. Keywords: Value-added Analysis, Value-added Models; Value-added Estimates; Stability; Teacher Evaluations; Accountability.Las estimaciones de valor agregado docente como parte de las evaluaciones docentes: Explorando los efectos de los datos y las especificaciones de los modelos en la estabilidad de las puntuaciones de valor agregado docentes Resumen: Si estimaciones de valor añadido docente (VADs) van a ser utilizadas como indicadores de rendimiento de los docentes en cada evaluación de los profesores y en los sistemas rendición de cuentas, es importante entender cómo los VADs son afectados por los datos recogidos y las especificaciones de los modelos utilizados para estimarlos. En este estudio hemos explorado los efectos de tres condiciones para la estabilidad de VADs y evaluar sus impactos relativos. Hemos variado cómo se estimaron las diferencias entre los estudiantes en sus aprendizajes previos; hemos estimado VADs de una o múltiples promociones de estudiantes; y el número de estudiantes que contribuyen a la de cada uno de los VADs de los profesores. Utilizando los datos de uno de los mayores distritos escolares de la nación, hemos creado un conjunto único de datos completo y lo utilizamos para estimar varios conjuntos de único para cada uno de los 3651 profesores de matemáticas de 5to. Grado. Encontramos que aproximadamente dos terceras partes de los maestros se mantuvieron estables y que permanecieron en el mismo grupo de rendimiento a lo largo de todas las condiciones. También encontramos que las diferencias en el número de alumnos u...

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Value-added Teacher Estimates as Part of Teacher Evaluations: Exploring the Effects of Data and Model Specifications on the Stability of Teacher Value-added Scores

Kersting

Chen

Stigler

2013

EPAA

View full text Add to dashboard Cite

show abstract

“…Por exemplo, segundo Kane et al (2002), um problema estatístico diretamente associado a essas imprecisões decorre da chamada volatilidade dos resultados das avaliações. Os autores a ela se referem assim:…”

Section: Também Observam Queunclassified

Volatilidade Dos Resultados De Proficiência E Seu Impacto Sobre as Metas Do Ideb Nas Escolas Públicas De Minas Gerais

Pontes

Soares

2017

Educ. rev.

View full text Add to dashboard Cite

RESUMO: Este artigo tem dois objetivos principais: mensurar a volatilidade dos resultados das avaliações de Língua Portuguesa e Matemática nas escolas públicas mineiras; e estimar o possível impacto dessas flutuações nos resultados do cumprimento das metas do Ideb. Para tal, analisa-se um banco com dados provenientes da Prova Brasil e do Sistema Mineiro de Avaliação da Educação Básica (Simave). O método estatístico empregado é um modelo longitudinal linear hierárquico que tem a escola como nível de interesse. Entre as principais conclusões obtidas estão: o fato de que as flutuações das notas médias das escolas são bastante acentuadas, mesmo quando se leva em conta a sua tendência de crescimento; e a constatação de que o impacto sobre o cumprimento das metas do Ideb, devido a tais flutuações, pode ser considerável, a ponto de facilitar ou dificultar o sucesso das escolas quanto ao seu cumprimento. Palavras-chave: Ideb. Regressão linear hierárquica. Volatilidade dos resultados educacionais. VOLATILITY OF PROFICIENCY RESULTS AND ITS IMPACT ON IDEB GOALS IN THE PUBLIC SCHOOLS OF MINAS GERAISABSTRACT: This paper has two main purposes. The first one is to obtain a volatility measure of Portuguese and Mathematics mean scores in Minas Gerais State public schools. The second one is to estimate the probable impact of these fluctuations on results of respective Ideb goals. The analysis is based on data obtained from Prova Brasil and Simave (Sistema Mineiro de Avaliação da Educação Básica) tests. The used statistical technique is a longitudinal hierarchical model with the schools as the level of interest. One of the main conclusions obtained is the fact that fluctuation in school mean scores is considerable, even * http://dx

show abstract

“…

AbstractThis paper contrasts the positions of Kane and Staiger (2002) and Linn and Haug (2002) with that of Rogosa (2002Rogosa ( , 2003a on the volatility in test scores as measures of school growth. In particular, these researchers disagree on whether school growth can be measured reliably in school accountability systems.

…”

mentioning

confidence: 93%

“…In particular, Kane and Staiger (2002) and Linn and Haug (2002) argued that year-to-year changes in scores for groups of students are extremely unstable, and therefore, prone to misuse and misinterpretation. Using state-based data, Kane and Staiger (2002) illustrated the volatility of test score measures as tools for measuring change at the school level and suggested implications that were severe and alarming:…”

Section: Introductionmentioning

confidence: 99%

Precision and Volatility in School Accountability Systems

Way¹

2006

ETS Research Report Series

View full text Add to dashboard Cite

This paper contrasts the positions of Kane and Staiger (2002) and Linn and Haug (2002) with that of Rogosa (2002Rogosa ( , 2003a on the volatility in test scores as measures of school growth. In particular, these researchers disagree on whether school growth can be measured reliably in school accountability systems. The different positions of these authors are examined in some detail. In addition, several issues related to their debate are examined in the context of the No Child Left Behind (NCLB) legislation, which has imposed a particular structure on the accountability systems currently in place in many states. Finally, some possibilities and speculations about the future of school accountability systems are provided in relation to the NCLB requirements and their impact.

show abstract

Volatility in School Test Scores: Implications for Test-Based Accountability Systems

Cited by 138 publications

References 5 publications

Value-added Teacher Estimates as Part of Teacher Evaluations: Exploring the Effects of Data and Model Specifications on the Stability of Teacher Value-added Scores

Value-added Teacher Estimates as Part of Teacher Evaluations: Exploring the Effects of Data and Model Specifications on the Stability of Teacher Value-added Scores

Volatilidade Dos Resultados De Proficiência E Seu Impacto Sobre as Metas Do Ideb Nas Escolas Públicas De Minas Gerais

Precision and Volatility in School Accountability Systems

Contact Info

Product

Resources

About