Real-world effectiveness of a social-psychological intervention translated from controlled trials to classrooms

Abstract

Social-psychological interventions have raised the learning and performance of students in rigorous efficacy trials. Yet, after they are distributed “in the wild” for students to self-administer, there has been little research following up on their translational effectiveness. We used cutting-edge educational technology to tailor, scale up, and track a previously-validated Strategic Resource Use intervention among 12,065 college students in 14 STEM and Economics classes. Students who self-administered this “Exam Playbook” benefitted by an average of 2.17 percentage points (i.e., a standardized effect size of 0.18), compared to non-users. This effect size was 1.65 percentage points when controlling for college entrance exam scores and 1.75 [−1.88] for adding [dropping] the Exam Playbook in stratified matching analyses. Average benefits differed in magnitude by the conduciveness of the class climate (including peer norms and incentives), gender, first-generation status, as well as how often and how early they used the intervention. These findings on how, when, and who naturally adopts these resources address a need to improve prediction, translation, and scalability of social-psychological intervention benefits.

Preparing Educators and Students at Higher Education Institutions for an AI-Driven World

“…Another example is the University of Michigan's use of ECoach, an AI-powered personalized adaptive learning (PAL) model that provides students with scaffolded feedback, resources, and study strategies, helping them to navigate their courses more effectively (University of Michigan n.d.). An evaluation of ECoach found that it improved student performance and reduced achievement gaps, which may be relevant to improving outcomes for first-generation students and students from lowerincome backgrounds (Chen et al 2022). Despite the availability of these higher education-specific systems tailored for learners and adaptable to specific courses, publicly available GenAI tools, such as CoPilot, ChatGPT, and Gemini mean that students are also able to access an additional layer of highly flexible and customizable GenAI tools at any time.…”