Contrasting Classical and Machine Learning Approaches in the Estimation of Value-Added Scores in Large-Scale Educational Data

Abstract

There is no consensus on which statistical model estimates school value-added (VA) most accurately. To date, the two most common statistical models used for the calculation of VA scores are two classical methods: linear regression and multilevel models. These models have the advantage of being relatively transparent and thus understandable for most researchers and practitioners. However, these statistical models are bound to certain assumptions (e.g., linearity) that might limit their prediction accuracy. Machine learning methods, which have yielded spectacular results in numerous fields, may be a valuable alternative to these classical models. Although big data is not new in general, it is relatively new in the realm of social sciences and education. New types of data require new data analytical approaches. Such techniques have already evolved in fields with a long tradition in crunching big data (e.g., gene technology). The objective of the present paper is to competently apply these "imported" techniques to education data, more precisely VA scores, and assess when and how they can extend or replace the classical psychometrics toolbox. The different models include linear and non-linear methods and extend classical models with the most commonly used machine learning methods (i.e., random forest, neural networks, support vector machines, and boosting). We used representative data of 3,026 students in 153 schools who took part in the standardized achievement tests of the Luxembourg School Monitoring Program in grades 1 and 3. Multilevel models outperformed classical linear and polynomial regressions, as well as different machine learning models. However, it could be observed that across all schools, school VA scores from different model types correlated highly. Yet, the percentage of disagreements as compared to multilevel models was not trivial and real-life implications for individual schools may still be dramatic depending on the model type used. Implications of these results and possible ethical concerns regarding the use of machine learning methods for decision-making in education are discussed.

Value-Added Scores Show Limited Stability over Time in Primary School

“…To solve this difficulty in comparing schools, researchers have drawn on VA scores that control for student composition factors (e.g., students' language background, SES, and prior achievement) and single out the "net effect" of school effectiveness (21). VA scores can be calculated in different ways in terms of variables and choice of statistical models (9,10,22). However, the underlying idea, which originated from economics (23), is the same for all of these statistical models: When controlling for all available background variables and prior achievement, all gains in achievement that are left are likely to be due to teacher or school effectiveness (for an in-depth literature review of research on teacher and school VA scores, please see 9,17,23).…”

“…and tackled, such as creating a consensus on how to best estimate VA scores (9,10,17). The most widely used models are the multilevel and linear regression models, with the former outperforming the latter (19,20).…”

Educating Software and AI Stakeholders About Algorithmic Fairness, Accountability, Transparency and Ethics

“…Our society is challenged by the increasing proliferation of opaque algorithmic systems, that apply black box machine learning algorithms, making use of large volumes of data to facilitate decision making in a wide variety of sensitive applications. We can see this phenomena in every aspect of our lives, including education (see for instance a work about estimating school value-added scores (Levy et al 2190;Prinsloo 2020) and a recent survey about the application of machine learning in education (Kučak et al 2018)). These systems are based on complex machine learning algorithms that are trained over large amounts of data in order to provide their prediction or classification of new cases.…”

La Comprensión Lectora Avanzada a través de las Disciplinas: Variables Instruccionales y Psicoeducativas

“…Como consecuencia de lo anterior, en la actualidad, son diversos los análisis focales comparativos realizados en torno a las competencias, entre las que se incluyen la comprensión lectora avanzada (Hershkovitz y Alexandron, 2020; Levy et al, 2020;Wang, 2020). Estas investigaciones se centran, en su mayoría, en el análisis de datos masivos obtenidos de pruebas de organismos internacionales procedentes de diferentes países.…”

“…Igualmente, otros estudios examinan las variables psicoeducativas relacionadas con la CLA tales como el academic engagement (Durón-Ramos et al, 2020) o el compromiso organizacional y su relación con la eficacia docente y la autoestima colectiva y su influjo en los resultados académicos (Chung, 2019) o el e-learning engagement (Gao et al, 2021); la motivación de logro y para el aprendizaje como variables claves para la consecución de resultados y mejoras en las competencias (Lai et al, 2018;Rodríguez et al, 2019); las atribuciones y las estrategias de afrontamiento ante las dificultades en el aprendizaje de competencias u otros (Morales, 2018;Song et al, 2020); los estilos de aprendizaje dentro de modelos diversos que median en los resultados en las competencias a partir de los cambios que se introducen en las metodologías docentes y en los sistemas de evaluación (Gutiérrez, 2018); la inteligencia emocional como papel mediador en el compromiso con la mejora de competencias (Merino et al, 2018); la autorregulación en la lectura y la teoría de la autorregulación incluyendo la automotivación como factores nucleares en el despliegue de las competencias (Cheng y Lin, 2020;Li et al, 2018;Qi, 2021;Suárez et al, 2019); las habilidades emocionales inter e intrapersonales de los estudiantes que juegan un papel relevante mediando en la pericia con las competencias (Almerich et al, 2018;Morales, 2018); el pensamiento crítico y creativo que introduce flexibilidad y eficiencia en el dominio de competencias (Almerich et al, 2018); el uso de los social media como arma poderosísima y cada vez más esencial en la conquista del dominio de las competencias genéricas y específicas incluyendo todo el campo de la competencia digital y la digital literacy pasando de un uso de las herramientas para el ocio a un uso cada vez más educativo (García-Martín y García, 2013); los resultados académicos (Doménech et al, 2017;Karagiannopoulou et al, 2018); la orientación a metas académicas, las actitudes y la ansiedad ante la lectura (Nootens et al, 2019;Zhou, 2017); el papel de los conocimientos previos (Dong et al, 2020); el clima y la mentalidad académica (Kearney et al, 2020); la autoeficacia creativa (He et al, 2020); la autoeficacia y aceptación de la tecnología móvil en un modelo estructural complejo (Levy et al, 2020); incluyendo el papel crec...…”