Identifying Teacher Questions Using Automatic Speech Recognition in Classrooms

Abstract

We investigate automatic question detection from recordings of teacher speech collected in live classrooms. Our corpus contains audio recordings of 37 class sessions taught by 11 teachers. We automatically segment teacher speech into utterances using an amplitude envelope thresholding approach followed by filtering non-speech via automatic speech recognition (ASR). We manually code the segmented utterances as containing a teacher question or not based on an empirically-validated scheme for coding classroom discourse. We compute domain-independent natural language processing (NLP) features from transcripts generated by three ASR engines (AT&T, Bing Speech, and Azure Speech). Our teacher-independent supervised machine learning model detects questions with an overall weighted F 1 score of 0.59, a 51% improvement over chance. Furthermore, the proportion of automatically-detected questions per class session strongly correlates (Pearson's r = 0.85) with human-coded question rates. We consider our results to reflect a substantial (37%) improvement over the state-of-the-art in automatic question detection from naturalistic audio. We conclude by discussing applications of our work for teachers, researchers, and other stakeholders.

Predicting Specificity in Classroom Discussion

“…Chen et al (2014) developed a tool for teacher self-assessment of classroom discussion through the analysis of the frequency of participation of students in the discussion, and teacherstudent turn patterns. Blanchard et al (2016) developed a system for detecting teacher questions from classroom discussion recordings. These works, however, do not take into account the actual student discussion content.…”

A Systematic Review of Machine Learning for Assessment and Feedback of Treatment Fidelity

“…In this context, the machine learning methods were designed to automatically assign codes from the behavioural coding measures to overt interactions recorded in the dataset (e.g., words/utteran- (Atkins et al, 2014;Can et al, 2015;Can et al, 2012;Can et al, 2016;Cao et al, 2020;Carcone et al, 2019, Study 1;Chakravarthula et al, 2015;Chen et al, 2019;Gibson et al, 2017;Gibson et al, 2016;Gupta et al, 2014;Hasan et al, 2019;Hasan et al, 2018;Imel et al, 2015;Perez-Rosas et al, 2017;Perez-Rosas et al, 2019;Singla et al, 2018;Tanana et al, 2016;Xiao et al, 2012;Xiao et al, 2015;Xiao, Can, et al, 2016;Xiao, Huang, et al, 2016) Medical care, provider-patient clinical interactions (Carcone et al, 2019, Study 2;Park et al, 2019) Education (teachers) (Blanchard et al, 2016a;Blanchard et al, 2016b;Donelly et al, 2017;Donnely et al, 2016a;Donnelly et al, 2016b;Samei et al, 2014;Samei et al, 2015;Song et al, 2020;Suresh et al, 2019;Wang et al, 2014) Counselling, (counsellors), (Althoff et al, 2016;Flemotomos et al, 2018;Gallo et al, 2015;Gaut et al, 2017;Goldberg et al, 2020…”

“…Because few studies examined the performance of methods when transferred to other similar settings-for example, with similar predictors and outcomes but different participants-we are unable to ascertain whether any particular method predicted new data better than others. There were three studies that compared the performance of methods but did not report the predictive performance of all the tested methods and only chose the best performing method (Blanchard et al, 2016a(Blanchard et al, , 2016bDonnelly et al, 2017). Only one study developed a Support Vector Machine method in psychotherapy and applied it on new data from another context (i.e., medicine; Carcone et al, 2019).…”

Text mining in education

“…( Araki et al, 2016) Online assignments NLP Question/content generation The paper proposed a question generation approach that engages learners through the use of specific inference steps over multiple sentences requiring more semantic understanding of text. (Blanchard et al, 2016) Online assignments NLP Question/content generation The paper centers on a fully automated process for predicting instructor questions in a noisy real-world classroom environment. Online assignments…”