How to Build More Generalizable Models for Collaboration Quality? Lessons Learned from Exploring Multi-Context Audio-Log Datasets using Multimodal Learning Analytics

Chejara, Pankaj; Prieto, Luis P.; Rodríguez‐Triana, María Jesús; Kasepalu, Reet; Ruiz‐Calleja, Adolfo; Shankar, S.

doi:10.1145/3576050.3576144

Cited by 7 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these kinds of settings, the use of audio and video data has the potential to offer insights into collaborative learning processes. For example, the number of conversation turns, and speaking time have been found to be associated with collaboration quality in prior work (Chejara, Prieto, Rodriguez‐Triana, Kasepalu, et al., 2023; Chejara et al., 2021; Martinez‐Maldonado et al., 2013; Praharaj et al., 2021). Similarly, Storch, 2001 in their discourse analysis of collaborative pairs found differences in high/low collaborative groups in terms of their usage of personal pronouns.…”

Section: Study Setupmentioning

confidence: 93%

“…and threshold selection. We then identified high‐performing modelling pipelines for our final model development (Chejara, Prieto, Rodriguez‐Triana, Kasepalu, et al., 2023). We trained a Random Forest algorithm on the data.…”

Section: Methodsmentioning

confidence: 99%

“…MMLA researchers have utilized a variety of data sources (such as audio, video, eye gaze and skin‐conductance) for modelling collaboration (Pugh et al., 2022; Reilly & Schneider, 2019). From the collected data, features ranging from speaking time, turn‐taking and joint‐visual attention to facial action units and emotions have been extracted for understanding and modelling collaboration (Cai et al., 2020; Chejara, Prieto, Rodríguez‐Triana, Ruiz‐Calleja, Kasepalu, et al., 2023; Martinez‐Maldonado et al., 2013; Reilly & Schneider, 2019).…”

Section: Related Workmentioning

confidence: 99%

“…For conducting the collaborative learning activity and data collection, we used a web‐based application, CoTrack (Chejara, Kasepalu, et al., 2023). CoTrack uses an open‐source collaborative text editor, Etherpad, to allow group members to draft the solution to a given task in collaboration.…”

Section: Study Setupmentioning

confidence: 99%

See 3 more Smart Citations

How well do collaboration quality estimation models generalize across authentic school contexts?

Chejara,

Kasepalu,

Prieto

et al. 2023

Brit J Educational Tech

Self Cite

View full text Add to dashboard Cite

Multimodal learning analytics (MMLA) research has made significant progress in modelling collaboration quality for the purpose of understanding collaboration behaviour and building automated collaboration estimation models. Deploying these automated models in authentic classroom scenarios, however, remains a challenge. This paper presents findings from an evaluation of collaboration quality estimation models. We collected audio, video and log data from two different Estonian schools. These data were used in different combinations to build collaboration estimation models and then assessed across different subjects, different types of activities (collaborative‐writing, group‐discussion) and different schools. Our results suggest that the automated collaboration model can generalize to the context of different schools but with a 25% degradation in balanced accuracy (from 82% to 57%). Moreover, the results also indicate that multimodality brings more performance improvement in the case of group‐discussion‐based activities than collaborative‐writing‐based activities. Further, our results suggest that the video data could be an alternative for understanding collaboration in authentic settings where higher‐quality audio data cannot be collected due to contextual factors. The findings have implications for building automated collaboration estimation systems to assist teachers with monitoring their collaborative classrooms. Practitioners notesWhat is already known about this topic Multimodal learning analytics researchers have established several features as potential indicators for collaboration quality, e.g., speaking time or joint visual attention. The current state of the art has shown the feasibility of building automated collaboration quality models. Recent research has provided preliminary evidence of the generalizability of developed automated models across contexts different in terms of given task and subject. What does this paper add This paper offers collaboration indicators for different types of collaborative learning activities in authentic classroom settings. The paper includes a systematic investigation into collaboration quality automated model's generalizability across different tasks, types of tasks and schools. This paper also offers a comparison between different modalities' potential to estimate collaboration quality in authentic settings. Implications for practice The findings inform the development of automated collaboration monitoring systems for authentic classroom settings. This paper provides evidence on across‐school generalizability capabilities of collaboration quality estimation models.

show abstract

Section: Study Setupmentioning

confidence: 93%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Study Setupmentioning

confidence: 99%

See 2 more Smart Citations

How well do collaboration quality estimation models generalize across authentic school contexts?

Chejara,

Kasepalu,

Prieto

et al. 2023

Brit J Educational Tech

Self Cite

View full text Add to dashboard Cite

show abstract

“…They collected and analysed video data, audio data, and log data, and achieved the accuracy with a mean squared error of 0.13. However, the generalizability [7] and pedagogical value [31] of these predictions are also considered limited. Third, existing research illustrated the possibility of applying learning analytics to provide learning support and feedback for learners [41] and teachers [33] in collaborative learning.…”

Section: Background 21 Analytics Of Collaborative Learningmentioning

confidence: 99%

Harnessing Transparent Learning Analytics for Individualized Support through Auto-detection of Engagement in Face-to-Face Collaborative Learning

Zhou,

Suraworachet,

Cukurova

2024

Proceedings of the 14th Learning Analytics and Knowledge Conference

View full text Add to dashboard Cite

Using learning analytics to investigate and support collaborative learning has been explored for many years. Recently, automated approaches with various articial intelligence approaches have provided promising results for modelling and predicting student engagement and performance in collaborative learning tasks. However, due to the lack of transparency and interpretability caused by the use of "black box" approaches in learning analytics design and implementation, guidance for teaching and learning practice may become a challenge. On the one hand, the black box created by machine learning algorithms and models prevents users from obtaining educationally meaningful learning and teaching suggestions. On the other hand, focusing on group and cohort level analysis only can make it dicult to provide specic support for individual students working in collaborative groups. This paper proposes a transparent approach to automatically detect student's individual engagement in the process of collaboration. The results show that the proposed approach can reect student's individual engagement and can be used as an indicator to distinguish students with dierent collaborative learning challenges (cognitive, behavioural and emotional) and learning outcomes. The potential of the proposed collaboration analytics approach for scaolding collaborative learning practice in face-to-face contexts is discussed and future research suggestions are provided. CCS CONCEPTS• Applied computing ! Education; Collaborative learning; • Human-centered computing ! Collaborative and social computing; Empirical studies in collaborative and social computing.

show abstract

Exploring Indicators for Collaboration Quality and Its Dimensions in Classroom Settings Using Multimodal Learning Analytics

Chejara,

Prieto,

Rodríguez-Triana

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

How to Build More Generalizable Models for Collaboration Quality? Lessons Learned from Exploring Multi-Context Audio-Log Datasets using Multimodal Learning Analytics

Cited by 7 publications

References 23 publications

How well do collaboration quality estimation models generalize across authentic school contexts?

How well do collaboration quality estimation models generalize across authentic school contexts?

Harnessing Transparent Learning Analytics for Individualized Support through Auto-detection of Engagement in Face-to-Face Collaborative Learning

Exploring Indicators for Collaboration Quality and Its Dimensions in Classroom Settings Using Multimodal Learning Analytics

Contact Info

Product

Resources

About