Large-scale diagnostic assessment in first-year university students: pre- and transpandemic comparison

Abstract

COVID-19 has disrupted higher education globally, and there is scarce information about the “learning loss” in university students throughout this crisis. The goal of the study was to compare scores in a large-scale knowledge diagnostic exam applied to students admitted to the university, before and during the pandemic. Research design was quasi-experimental with static group comparisons, taking advantage of the pandemic “natural experiment,” to assess knowledge in students admitted to the National Autonomous University of Mexico. Four student cohorts were analyzed: 2017 and 2018 (prepandemic, paper-and-pencil exams), 2020 and 2021 (transpandemic, online exams). The same instruments were applied in each pair of cohorts (2017–2021; 2018–2020) to decrease instrumentation threat. Propensity score matching was used to create balanced comparable groups. 35,584 matched students from each of the 2018 and 2020 cohorts were compared and 31,574 matched students from each of the 2017–2021 cohorts. Reliability and point biserial correlation coefficients were higher in the transpandemic online applications. Knowledge scores were 2.3 to 7.1% higher in the transpandemic assessments, Spanish scores in the 2018–2020 comparison were 1.3% lower, and English results in 2021 were 7.1% lower than in 2017. Before the pandemic, there was a 3.1% higher test performance in men; this gap decreased to 0.34% during the pandemic. There was no documented learning loss in this large student population, with an increase in knowledge in the pandemic cohorts. Some influence in scores due to the online testing modality cannot be ruled out. Longitudinal follow-up is required to continue evaluating the impact of the pandemic in learning.

Perspectives and Opportunities for Learning Analytics Integration

“…Early alerts, timely feedback, and quality evaluations are examples of desired services that LA could facilitate in Latin America (Hilliger et al, 2020b); these could help to reduce school dropout and educational lag in the region (Ferreyra et al, 2017), challenges that have accentuated learning gaps due to the COVID-19 pandemic (Sánchez-Mendiola et al, 2023). Recognition of these phenomena can be attributed to the framework developed by the LALA Project (Tsai & Gašević, 2017).…”

“…Different efforts to promote LA have been made at Mexican institutions, with the development of applications like predictive models (Talamás-Carvajal & Ceballos Cancino, 2023), prevention of student dropout (Alvarado-Uribe et al, 2022), as well as other initiatives regarding prediction of student performance (Acosta-Gonzaga & Ramirez-Arellano, 2020;Monteverde-Suárez et al, 2021;Rincon-Flores et al, 2022). Similarly, there are studies of diagnostic assessment with large-scale standardized exams (Sánchez-Mendiola et al, 2023). Experiences related to these publications are a foundation for the continuous development of LA.…”

“…This characteristic is a feature previously recognized in institutional analytics (Siemens & Gašević, 2012), which, while desired, is still far from the expectations of an integrated learning analytics system. Other examples, like curricular analytics (Hilliger et al, 2019), recognition of school trajectories (Canales Sánchez et al, 2022), or opportunities to recognize the magnitude of the effects of the COVID-19 pandemic on first-year students (Sánchez-Mendiola et al, 2023), can provide other purposes and inputs for institutional decision-making. Nonetheless, many of these efforts are still in the research stage, especially in the case of Latin America and the Caribbean (Cechinel et al, 2020).…”

“…This practice does not promote well-being. Students suggest that adequate feedback using anonymized data and behaviours, accompanied by strategies that encourage student collaboration and role-differentiation, may improve the learning and assessment process by incorporating the principles of assessment-for-learning (Sánchez-Mendiola et al, 2023).…”