Adjusting for measurement error in the value added model: evidence from Portugal

Abstract

The multilevel value added approach to measuring school effectiveness is now widely used. We propose a method to adjust for measurement error to investigate the extent to which this changes school effect estimates. It is applied to longitudinal data collected in the region of Cova da Beira (NUT III) for 1st, 3rd, 5th, 7th and 8th grades. Three different variance component models are considered, depending on the predictor variables included. Assuming measurement error occurs in explanatory and/or response variables, corrections are made for different values of the coefficient of reliability. Moreover, models are fitted under the assumption of either independent or correlated measurement errors.

On the stability of value added indicators

“…There it is shown that the correlation between the school value added indicators resulting from each pair of models is larger than 0.96. Ferrão and Goldstein (2009) report correlations larger than 0.88 for the value added estimates obtained between TVA and the model which also incorporated student background variables such as: gender; whether or not the student went to kindergarten before entering primary school; whether he/she is a special needs student; if he/she was not promoted to the next grade at least once. Evidence tends to support the consistency of VA indicators across different models.…”

“…The traditional value added model (Ferrão and Goldstein 2009) is a two-level random intercept model where students are nested within classes or schools and the response variable is the student's score in Maths (3EMat score at the end of academic year) and the independent variables are prior achievement (3EMat score at the beginning) and socioeconomic status (SES), taking into account the social heterogeneity of Portuguese pupils. This is in line with Sammons et al (1997, p. 43) who demonstrated that prior achievement is the most important factor required to control intake differences in measuring 'value added', and they also showed that the inclusion of socioeconomic factors in the analysis is highly relevant.…”

“…However, results suggest a reasonable stability of indicators which allows us to identify schools or classes that repeatedly are at the extreme positions at a value added scale. We present results using a multilevel model of school effectiveness that is usually known as the traditional value added (TVA) model (Ferrão and Goldstein 2009). That model is separately fitted for each academic year.…”

Estatística Educacional E Política Pública: A Propósito Dos Modelos De Valor Acrescentado

“…O princípio da transparência recomenda que qualquer que seja a abordagem de VA usada, os métodos estatísticos e o software devem ser completamente descritos, testados e documentados (MORGANSTEIN & WASSERSTEIN, 2014). Embora os estudos de consistência e da correlação forte entre as estimativas obtidas por modelos com diferentes especificações sejam encorajadores quanto ao seu uso como critério de avaliação educacional (FERRÃO, 2010;FERRÃO & GOLDSTEIN, 2009), é necessário ter em mente que a classificação da escola (ou do professor) por meio da aplicação do modelo pode depender da escolha do modelo. De acordo com Everson (2016), a inclusão ou a exclusão de covariáveis no modelo deve depender antes de tudo da resposta a algumas questões de natureza filosófica, pois na comparação inerente ao processo de avaliação, nem sempre se sabe com quem um professor ou os alunos de uma escola devem ser comparados.…”

Avaliação educacional e modelos de valor acrescentado: tópicos de reflexão

“…Por ora, vale mencionar os novos desafi os de investigação científi ca e de desenvolvimento metodológico na classe de modelos de valor acrescentado que derivam dessas difi culdades, tais como a linha de investigação que visa o ajuste ou a correcção do erro da medida nos modelos de valor acrescentado (e.g. Ferrão & Goldstein, 2009;Goldstein, Kounali & Robinson, 2008;Ladd & Walsh, 2002;Woodhouse, Yang & Goldstein, 1996).…”