Discourse Annotation of Non-native Spontaneous Spoken Responses Using the Rhetorical Structure Theory Framework

Abstract

The availability of the Rhetorical Structure Theory (RST) Discourse Treebank has spurred substantial research into discourse analysis of written texts; however, limited research has been conducted to date on RST annotation and parsing of spoken language, in particular, nonnative spontaneous speech. Considering that the measurement of discourse coherence is typically a key metric in human scoring rubrics for assessments of spoken language, we initiated a research effort to obtain RST annotations of a large number of non-native spoken responses from a standardized assessment of academic English proficiency. The resulting inter-annotator κ agreements on the three different levels of Span, Nuclearity, and Relation are 0.848, 0.766, and 0.653, respectively. Furthermore, a set of features was explored to evaluate the discourse structure of non-native spontaneous speech based on these annotations; the highest performing feature showed a correlation of 0.612 with scores of discourse coherence provided by expert human raters.

Using Rhetorical Structure Theory to Assess Discourse Coherence for Non-native Spontaneous Speech

“…This study obtained manual RST annotations on a corpus of 1440 spoken responses, where 600 of them were obtained in our previous work (Wang et al, 2017a), and the additional 840 responses were annotated more recently. All the responses were drawn from a large-scale, highstakes standardized assessment of English for nonnative speakers, the TOEFL R Internet-based Test (TOEFL R iBT), which assesses English communication skills for academic purposes (ETS, 2012).…”

“…This study used the same annotation guidelines as in our previous work Wang et al (2017a), which is a modified version of the tagging reference manual from the RST Discourse Treebank ). According to these guidelines, annotators segment a transcribed spoken response into Elementary Discourse Unit (EDU) spans of text (corresponding to clauses or clauselike units), and indicate rhetorical relations between non-overlapping spans which typically consist of a nucleus (the most essential information in the rhetorical relation) and a satellite (supporting or background information).…”

“…Among the 600 annotations obtained in Wang et al (2017a), 120 responses from 6 test questions (5 responses from each score level for each question) were double annotated. The standard evaluation method of F1 scores on three levels (span, nuclearity, and relation) (Marcu, 2000b) was used to evaluate the human agreement, where 3 Downloaded from http://www.isi.edu/ licensed-sw/RSTTool/index.html the F1 scores were calculated globally by comparing the two annotators' labels from all samples, i.e., a micro-averaged F1 score.…”

“…It can represent a document's discourse structure using a hierarchical tree in which nodes are recursively linked with rhetorical relations and labeled with nucleus or satellite tags to depict the importance of the child nodes in a relation. In our previous study (Wang et al, 2017a), RST-based discourse annotations were obtained on a corpus of 600 spontaneous spoken responses provided by non-native English speakers in the context of an English speaking proficiency assessment. In this paper, we continued this line of research, and made further contributions as follows:…”

“…• A larger annotated corpus consisting of 1440 non-native spontaneous spoken responses was obtained using an annotation scheme based on the RST framework. In addition to the previously annotated 600 responses (Wang et al, 2017a), annotations on additional 840 responses were obtained to enlarge the data set that can be used to train an automatic RST parser. When comparing the annotations from two independent human experts on 120 responses, the resulting micro-averaged F1 scores on the three different levels of span, nuclearity, and relation 1 are 86.8%, 72.2%, and 58.2%, respectively.…”