A Deep Transfer Learning Approach to Modeling Teacher Discourse in the Classroom

Abstract

Teachers, like everyone else, need objective reliable feedback in order to improve their effectiveness. However, developing a system for automated teacher feedback entails many decisions regarding data collection procedures, automated analysis, and presentation of feedback for reflection. We address the latter two questions by comparing two different machine learning approaches to automatically model seven features of teacher discourse (e.g., use of questions, elaborated evaluations). We compared a traditional openvocabulary approach using n-grams and Random Forest classifiers with a state-of-the-art deep transfer learning approach for natural language processing (BERT). We found a tradeoff between data quantity and accuracy, where deep models had an advantage on larger datasets, but not for smaller datasets, particularly for variables with low incidence rates. We also compared the models based on the level of feedback granularity: utterance-level (e.g., whether an utterance is a question or a statement), class session-level proportions by averaging across utterances (e.g., question incidence score of 48%), and session-level ordinal feedback based on pre-determined thresholds (e.g., question asking score is medium [vs. low or high]) and found that BERT generally provided more accurate feedback at all levels of granularity. Thus, BERT appears to be the most viable approach to providing automatic feedback on teacher discourse provided there is sufficient data to fine tune the model.

Can Multi-Label Classifiers Help Identify Subjectivity? A Deep Learning Approach to Classifying Cognitive Presence in MOOCs

“…Google has provided the BERT models pre-trained by the large corpora (Wikipedia and Book Corpus) for developers and researchers to fine-tune and apply in their personalised tasks. In learning analytics fields, the BERT model shows great potential to analyse teacher discourse in online classrooms automatically (Jensen et al, 2021). The fine-tuning BERT model was applied to analyse cognitive presence in discussion messages from a computer science MOOC and for-credit course, which reported the promising F 1 scores of 0.95 (Hosmer & Lee, 2021;Lee et al, 2022).…”

Computationally Identifying Funneling and Focusing Questions in Classroom Discourse

“…Prior work in computationally analyzing classroom discourse has employed a variety of techniques to automatically detect teacher discourse variables. Recent advances in natural language processing has led to a larger presence of work applying neural methods with varying levels of success in detecting classroom discourse variables, such as semantic content, instructional talk, and elaborated evaluation (Jensen et al, 2021;Song et al, 2021). For unsupervised approaches, Demszky et al (2021a), which is also most similar to our work in terms of approach and dataset, propose an unsupervised measure of teachers' uptake of students' contributions, and we use their sample in our annotation for funneling and focusing.…”

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

“…Prior work in computationally analyzing classroom discourse has employed a variety of techniques to automatically detect teacher discourse variables. Recent advances in natural language processing has led to a larger presence of work applying neural methods with varying levels of success in detecting classroom discourse variables, such as semantic content, instructional talk, and elaborated evaluation (Jensen et al, 2021;. For unsupervised approaches, Demszky et al (2021a), which is also most similar to our work in terms of approach and dataset, propose an unsupervised measure of teachers' uptake of students' contributions, and we use their sample in our annotation for funneling and focusing.…”

“…Similarly, ) used a chunking approach where documents were broken down into multiple chunks, and the activations were then combined to perform the tasks. Another recent example is the BERT-Seq model for classifying Collaborative Problem Solving (Pugh et al, 2021). The BERT-Seq model uses a special input representation that combines embeddings from adjacent utterances as contextual cues for the model.…”