Towards automatic annotation of collaborative problem‐solving skills in technology‐enhanced environments

Abstract

Background Collaborative problem solving (CPS) is important for success in the 21st century, especially for teamwork and communication in technology‐enhanced environments. Measurement of CPS skills has emerged as an essential aspect in educational assessment. Modern research in CPS relies on theory‐driven measurements that are usually carried out as manual annotations over recorded logs of collaborative activities. However, manual annotation has limited scalability and is not conductive towards CPS assessments at scale. Objective We explore possibilities for automated annotation of actions in collaborative‐teams, especially chat messages. We evaluate two approaches that employ machine learning for automated classification of CPS events. Method Data were collected from engineering, physics and electronics students' participation in a simulation‐based task on electronics concepts, in which participants communicated via text‐chat messages. All task activities were logged and time stamped. Data have been manually classified for the CPS skills, using an ontology that includes both social and cognitive dimensions. In this article, we describe computational linguistic methods for automatically classifying the CPS skills from logged data, with a view towards automating CPS assessments. Results We applied two machine learning methods to our data. A Naïve Bayes classifier has been previously used in CPS research, but it is only moderately successful on our data. We also present a k‐nearest‐neighbours (kNN) classifier that uses distributional semantic models for measuring text similarity. This classifier shows strong agreement between automated and human annotations. The study also demonstrates that automatic spelling correction and slang normalization of chat texts are useful for accurate automated annotation. Implications Our results suggest that a kNN classifier can be very effective for accurate annotation of CPS events. It achieves reasonably strong results even when trained on only half of the available data. This shows a promise towards reduction of manual data annotation for CPS measurement.

Exploring Automated Classification Approaches to Advance the Assessment of Collaborative Problem Solving Skills

“…In particular, some profiles showed more consistency (e.g., Low Collaborators (71.4%)) than others (e.g., Super Socials (55.6%), Active Collaborators (55.9%)). These results are reasonably in line with prior automated annotation work for CPS which has shown a general, but not complete agreement when compared to human annotation, which is often considered the ground truth ( Flor and Andrews-Todd 2022 ; Flor et al 2016 ; Hao et al 2017 ; Pugh et al 2021 ; Rosé et al 2008 ; Stewart et al 2019 ). Refinement of our automated annotation algorithms is still ongoing, striving for even better agreement with human annotations.…”

“…Automated classification of the data was approached as a hybrid, rule-based and machine-learning classification process ( Flor and Andrews-Todd 2022 ). All the events in the Three Resistor Activity were automatically logged, with multiple information fields.…”

“…Recent research has shown that the interpretation of CPS data can be automated using natural language processing (NLP) approaches ( Flor et al 2016 ; Flor and Andrews-Todd 2022 ; Hao et al 2017 ; Pugh et al 2021 ; 2022 ; Stewart et al 2019 ). As previously noted, one approach for CPS assessment is to have humans interact with an artificial agent to solve a problem.…”

“…We seek to explore how well the approaches align in terms of generating the same profiles and how comparable student classifications into identified profiles are across the annotation approaches. Based on recent work ( Flor and Andrews-Todd 2022 ) and the extent to which automated approaches are becoming more sophisticated, our expectation was that the same profiles would emerge across both annotation methods. However, given that automated approaches are not typically 100% accurate in identifying CPS skills, we believed there would be differences in how students were categorized into profiles across the two annotation approaches.…”

Artificial intelligence on the advance to enhance educational assessment: Scientific clickbait or genuine gamechanger?

“…log and text data) and demonstrate the value of their joint analysis at the output level. Flor and Andrews‐Todd (2022) classified actions consisting of log events from learner–environment interactions and learner‐generated chat messages during a collaborative problem‐solving task. Although they did not directly align the streams of log and chat message data, they used a multi‐class ontology with the ultimate goal of replacing human annotation.…”