Towards a fuller picture: Triangulation and integration of the measurement of self‐regulated learning based on trace and think aloud data

Abstract

Background: Many learners struggle to productively self-regulate their learning. To support the learners' self-regulated learning (SRL) and boost their achievement, it is essential to understand the cognitive and metacognitive processes that underlie SRL.To measure these processes, contemporary SRL researchers have largely utilized think aloud or trace data, however, not without challenges.Objectives: In this paper, we present the findings of a study that investigated how concurrent analysis and integration of think aloud and trace data could advance the measurement of SRL and assist in better understanding the mechanisms of SRL processes, especially those details that remain obscured by observing each data channel individually.Methods: We concurrently collected think aloud and trace data generated by 44 university students in a laboratory setting and analysed those data relative to the same timeline.Results: We found that the two data channels could be interchangeably used to measure SRL processes for only 17.18% of all the time segments identified in a learning task. Moreover, SRL processes for around 45% of all the time segments could be detected via either trace data or think aloud data. For another 27.17% of all the time segments, different SRL processes were detected in both data channels.Conclusions: Our results largely suggest that the two data collection methods can be used to complement each other in measuring SRL. In particular, we found that think aloud and trace data could provide different perspectives on SRL. The integration of the two methods further allowed us to reveal a more complex and more comprehensive temporal associations among SRL processes compared to using a single data collection method. In future research, the integrated measurement of SRL can be used to improve the detection of SRL processes and provide a fuller picture of SRL.

Using Large Language Models to Detect Self-Regulated Learning in Think-Aloud Protocols

“…Being able to measure SRL in think-aloud at scale is particularly relevant given the two approaches-behavioral log data and thinkaloud protocols-appear to complement each other rather than being substitutive, as noted in [19]. In their study, they show that SRL behaviors, such as orientation and elaboration, were more likely to be detected in behavioral log data than in think-aloud protocols.…”

“…Think-aloud protocols (TAPs) are an important approach for investigating SRL during problem-solving and have been widely used in prior research [9,26,28,37]. Previous studies not only demonstrate TAPs as a valid approach to measure SRL but also underlines its importance in capturing SRL in a more comprehensive way, complementing behavioral log data in SRL measurement [19]. However, TAPs require researchers to manually code students' verbalizations, which presents challenges in scalability.…”

“…In their study, they show that SRL behaviors, such as orientation and elaboration, were more likely to be detected in behavioral log data than in think-aloud protocols. This could potentially be due to the differences in students' ability to articulate their thoughts in the form of useful data, particularly when students are experiencing high cognitive demands during a think-aloud activity, in which they are learning while verbalizing their thoughts [19]. In contrast, SRL behaviors, such as planning and monitoring, were found to be more easily identified through think-aloud protocols.…”

Combining Dialog Acts and Skill Modeling: What Chat Interactions Enhance Learning Rates During AI-Supported Peer Tutoring?

“…This finding implies that future work should (a) incorporate information from ITS log data (i.e., whether a Tutor chat recommendation would lead to a correct attempt) into the prediction of the BERT model and (b) include training data examples of contextually incorrect constructive messages by Tutors into its sample. Indeed, prior work found similar integrations of log data into natural language for inference of self-regulated to be effective [20]. Similarly, nascent work found that the exclusion of problem-solving context in tutoring systems likely limits the accuracy of classifiers of selfregulation process stages based on natural language [72].…”

Using Think-Aloud Data to Understand Relations between Self-Regulation Cycle Characteristics and Student Performance in Intelligent Tutoring Systems

“…We collected think-aloud from students working with ITS for stoichiometry chemistry, a domain that, to the best of our knowledge, has not been investigated via fine-grained SRL codes. As suggested in prior work [19,42], think-aloud protocols provide valid measures on students' use of SRL because they capture an immediate window into students' cognitive and metacognitive processes as opposed to predefined categories in self-report questionnaires. The rich accounts of SRL in think-aloud data are especially useful when limited inferences can be drawn from behavioral logs.…”