Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

No abstract

Monitoring and guiding multiple groups of students in face-to-face collaborative work is a demanding task which could possibly be alleviated with the use of a technological assistant in the form of learning analytics. However, it is still unclear whether teachers would indeed trust, understand, and use such analytics in their classroom practice and how they would interact with such an assistant. The present research aimed to find out what the perception of in-service secondary school teachers is when provided with a dashboard based on audio and digital trace data when monitoring a collaborative learning activity. In a vignette study, we presented twenty-one in-service teachers with videos from an authentic collaborative activity, together with visualizations of simple collaboration analytics of those activities. The teachers perceived the dashboards as providers of useful information for their everyday work. In addition to assisting in monitoring collaboration, the involved teachers imagined using it for picking out students in need, getting information about the individual contribution of each collaborator, or even as a basis for assessment. Our results highlight the need for guiding dashboards as only providing new information to teachers did not compel them to intervene and additionally, a guiding dashboard could possibly help less experienced teachers with data-informed assessment.

Orchestrating collaborative learning (CL) is difficult for teachers as it involves being aware of multiple simultaneous classroom events and intervening when needed. Artificial intelligence (AI) technology might support the teachers’ pedagogical actions during CL by helping detect students in need and providing suggestions for intervention. This would be resulting in AI and teacher co-orchestrating CL; the effectiveness of which, however, is still in question. This study explores whether having an AI assistant helping the teacher in orchestrating a CL classroom is understandable for the teacher and if it affects the teachers’ pedagogical actions, understanding and strategies of coregulation. Twenty in-service teachers were interviewed using a Wizard-of-Oz protocol. Teachers were asked to identify problems during the CL of groups of students (shown as videos), proposed how they would intervene, and later received (and evaluated) the pedagogical actions suggested by an AI assistant. Our mixed-methods analysis showed that the teachers found the AI assistant useful. Moreover, in multiple cases the teachers started employing the pedagogical actions the AI assistant had introduced to them. Furthermore, an increased number of coregulation methods were employed. Our analysis also explores the extent to which teachers’ expertise is associated with their understanding of coregulation, e.g., less experienced teachers did not see coregulation as part of a teacher’s responsibility, while more experienced teachers did.

The estimation of collaboration quality using manual observation and coding is a tedious and difficult task. Researchers have proposed the automation of this process by estimation into few categories (e.g., high vs. low collaboration). However, such categorical estimation lacks in depth and actionability, which can be critical for practitioners. We present a case study that evaluates the feasibility of quantifying collaboration quality and its multiple sub-dimensions (e.g., collaboration flow) in an authentic classroom setting. We collected multimodal data (audio and logs) from two groups collaborating face-to-face and in a collaborative writing task. The paper describes our exploration of different machine learning models and compares their performance with that of human coders, in the task of estimating collaboration quality along a continuum. Our results show that it is feasible to quantitatively estimate collaboration quality and its sub-dimensions, even from simple features of audio and log data, using machine learning. These findings open possibilities for in-depth automated quantification of collaboration quality, and the use of more advanced features and algorithms to get their performance closer to that of human coders.