The validity and utility of activity logs as a measure of student engagement

Motz, Benjamin; Quick, Joshua; Schroeder, Noah L.; Zook, Jordon; Gunkel, Matthew

doi:10.1145/3303772.3303789

Cited by 35 publications

(35 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Such measures are commonly viewed through the broader framework of self-regulated learning, such that the quality and extent of activity can provide a composite measure of a student's motivation, engagement, time management, learning strategies, study behavior, and more (Roll & Winne, 2015;Winne, 2017). In general, studies find that these objective measures of a student's LMS activity are positively associated with engagement and achievement (Cerezo et al, 2016;Conijn et al, 2017;Joksimović et al, 2015;You, 2016;Yu & Jo, 2014), even when controlling for the amount of work assigned within a course (Motz et al, 2019). Given these patterns, college instructors, facing the reality of transitioning all their course materials online in a short period of time, would have been well-justified in prioritizing the development of activities to help their students stay engaged during the period of remote instruction.…”

Section: A Pandemic Of Busywork: Increased Online Coursework Following the Transition To Remote Instruction Is Associated With Reduced Acmentioning

confidence: 99%

A Pandemic of Busywork: Increased Online Coursework Following the Transition to Remote Instruction is Associated with Reduced Academic Achievement

Motz

Quick

Wernert

et al. 2021

OLJ

View full text Add to dashboard Cite

Under normal circumstances, when students invest more effort in their schoolwork, they generally show evidence of improved academic achievement. But when universities abruptly transitioned to remote instruction in Spring 2020, instructors assigned rapidly-prepared online learning activities, disrupting the normal relationship between effort and outcomes. In this study, we examine this relationship using data observed from a large-scale survey of undergraduate students, from logs of student activity in the online learning management system, and from students’ estimated cumulative performance in their courses (n = 4,636). We find that there was a general increase in the number of assignments that students were expected to complete following the transition to remote instruction, and that students who spent more time and reported more effort carrying out this coursework generally had lower course performance and reported feeling less successful. We infer that instructors, under pressure to rapidly put their course materials online, modified their courses to include online busywork that did not constitute meaningful learning activities, which had a detrimental effect on student outcomes at scale. These findings are discussed in contrast with other situations when increased engagement does not necessarily lead to improved learning outcomes, and in comparison with the broader relationship between effort and academic achievement.

show abstract

Section: A Pandemic Of Busywork: Increased Online Coursework Following the Transition To Remote Instruction Is Associated With Reduced Acmentioning

confidence: 99%

A Pandemic of Busywork: Increased Online Coursework Following the Transition to Remote Instruction is Associated with Reduced Academic Achievement

Motz

Quick

Wernert

et al. 2021

OLJ

View full text Add to dashboard Cite

show abstract

“…But despite this range of efforts, there are no published examples of scalable interventions aimed at direct support of students' adherence to assigned learning activities. The relative scarcity of intervention todols for helping students stay on top of the college workload is particularly problematic, considering that contemporary e-learning environments provide more autonomy for students to complete (or to fail to complete) their coursework [17], and where completion of this coursework is a primary predictor of student engagement and success [18].…”

Section: Intervention Science In Educationmentioning

confidence: 99%

“…The goal of the current study is to assess the benefit of proactive educative nudges for reducing missed assignments. Adherence to class assignments is, unsurprisingly, a principle predictor of student success and engagement in college [17,18], and considering that assignments represent one of the most heavily utilized features of learning management systems [27], this is a fertile area for improving student success at scale.…”

Section: The Current Studymentioning

confidence: 99%

Automated Educative Nudges to Reduce Missed Assignments in College

Motz¹,

Mallon²,

Quick³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

As institutions of higher education increasingly utilize online learning management systems, college students are asked to submit more assignments online. Under this regime, when most assignments are posted and submitted online, it is possible to know if a student is missing a submission for an imminent deadline, and to intervene proactively to reduce missed assignments and improve student outcomes. Toward this goal, we designed and evaluated a scalable targeted intervention: a mobile app that would deploy push notifications when students were missing submissions for assignments with imminent deadlines. Results from two experimental pilots demonstrate that this intervention system significantly decreased missed assignments compared with control notifications about instructor announcements to the class (in Experiment 1), and improved assignment adherence and course grades compared with courses that were not using the app (in Experiment 2). We discuss the benefits and theoretical implications of this behavioral guide rail, a purely informative proactive intervention to mitigate risk in advance of a negative outcome.

show abstract

“…In general, college students are completing and submitting an increasing volume of their assignments online, a trend broadly driven by higher education's increasing adoption of online learning management systems (Pomerantz & Brooks, 2017), and recently accelerated by the transition to remote instruction due to the COVID-19 pandemic (Motz et al, in press). Unsurprisingly, submitting to these assignments is a dominant factor in student success (Kauffman, 2015) and engagement (Motz et al, 2019), and these relationships are linked to students' self-regulation and motivation (Bembenutty & White, 2013;Kitsantas & Zimmerman, 2009).…”

Section: The Efficacy Of Automated Praise At Facilitating Behavior Chmentioning

confidence: 99%

The Influence of Automated Praise on Behavior and Performance

Motz¹,

Canning²,

Green³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

To be successful in college, students need to turn in their coursework, and a known strategy for improving submission rates is to present positive feedback when students turn in their assignments. In the current study, we test competing theoretical approaches to positive feedback in a large scale field experiment (n=1,766). Using a mobile app, we assigned college students to receive occasional notifications immediately upon submitting online assignments that either praised them for having submitted their coursework, or that highlighted the value of submitting coursework for academic success, or to a no-treatment control group. We find that only praise messages improved submission rates and course performance, suggesting that drawing attention to the feedback-eliciting task is necessary and sufficient for facilitating change at scale.

show abstract

The validity and utility of activity logs as a measure of student engagement

Cited by 35 publications

References 32 publications

A Pandemic of Busywork: Increased Online Coursework Following the Transition to Remote Instruction is Associated with Reduced Academic Achievement

A Pandemic of Busywork: Increased Online Coursework Following the Transition to Remote Instruction is Associated with Reduced Academic Achievement

Automated Educative Nudges to Reduce Missed Assignments in College

The Influence of Automated Praise on Behavior and Performance

Contact Info

Product

Resources

About