How the Predictors of Math Achievement Change over Time: A Longitudinal Machine Learning Approach

Abstract

Researchers have focused extensively on understanding the factors influencing students' academic achievement over time. However, existing longitudinal studies have often examined only a limited number of predictors at one time, leaving gaps in our knowledge about how these predictors collectively contribute to achievement beyond prior performance and how their impact evolves during students' development. To address this, we employed machine learning to analyze longitudinal survey data from 3,425 German secondary school students spanning Years 5 to 9. Our objectives were twofold: to model and compare the predictive capabilities of 105 predictors on math achievement and to track changes in their importance over time. We first predicted standardized math achievement scores in Years 6 to 9 using the variables assessed in the previous year ("next year prediction"). Second, we examined the utility of the variables assessed in Year 5 at predicting future math achievement at varying time lags (1-4 years ahead) --- "varying lag prediction". In the next year prediction analysis, prior math achievement was the strongest predictor, gaining importance over time. In the varying lag prediction analysis, the predictive power of Year 5 math achievement waned with longer time lags. In both analyses, additional predictors, including IQ, grades, motivation and emotion, cognitive strategies, classroom/home environments, and demographics (including SES), exhibited relatively smaller yet consistent contributions, underscoring their distinct roles in predicting math achievement over time. These findings have implications for both future research and educational practices, which are discussed in detail.

Machine Learning Meets Psychological Data: Challenges and Future Directions

“…Yet, while machine learning models are highly adept at dealing with time series data (e.g., Hochreiter and Schmidhuber (1997); Tuli et al (2022)), there are limited approaches optimized for data with low T 10 . Instead, the problem is typically converted to a more traditional machine learning problem, for instance, by treating each time point as a different prediction problem (e.g., see Lavelle-Hill et al (2023b) Another challenge is that, by heavily relying on subjective self-reported data, many measurements in psychology research are deemed ordinal (e.g., Likert scales). In traditional statistics, while researchers typically analyze such data as continuous, models have nonetheless been developed to model this data type (i.e., ordinal regression (Bürkner and Vuorre, 2019)).…”

“…Machine learning methods are seen as an appropriate tool to analyze large (and often unstructured) data (Faraway and Augustin, 2018). These methods can find complex (e.g., non-linear) patterns in data and have enabled accurate predictions of a number of different behaviors, for example, suicide attempts (Walsh et al, 2017); environmental behaviors (Lavelle-Hill et al, 2020), life outcomes (e.g., education achievement (Lavelle-Hill et al, 2023b); marital hardship (Salganik et al, 2020)), psychological constructs (e.g., personality traits (Youyou et al, 2015;Stachl et al, 2020)), developmental trajectories (Van Lissa et al, 2023;Van Lissa, 2022), clinical diagnoses (e.g., Alzheimer's disease (Lei et al, 2022)) as well as treatment outcomes (Chekroud et al, 2016). Contrary to the traditional statistical approach, which usually focuses on a single theoretically formalized statistical model, (supervised) machine learning follows an "algorithmic modeling" approach (Breiman, 2001b), which makes no assumptions about the underlying data generating mechanism, and seeks to find the function that best predicts the outcome from a set of possible predictors.…”

“…Let's take the example of a psychologist who wants to use supervised machine learning to analyze large-sample survey data to predict academic performance (e.g., Lavelle-Hill et al (2023b)). It sounds like a straightforward problem, but there are many practical issues that need to be addressed.…”