How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT

Abstract

This article evaluates purported progress over the past years in RST discourse parsing. Several studies report a relative error reduction of 24 to 51% on all metrics that authors attribute to the introduction of distributed representations of discourse units. We replicate the standard evaluation of 9 parsers, 5 of which use distributed representations, from 8 studies published between 2013 and 2017, using their predictions on the test set of the RST-DT. Our main finding is that most recently reported increases in RST discourse parser performance are an artefact of differences in implementations of the evaluation procedure. We evaluate all these parsers with the standard Parseval procedure to provide a more accurate picture of the actual RST discourse parsers performance in standard evaluation settings. Under this more stringent procedure, the gains attributable to distributed representations represent at most a 16% relative error reduction on fully-labelled structures.

Using Rhetorical Structure Theory to Assess Discourse Coherence for Non-native Spontaneous Speech

“…The human agreement results are 86.8%, 72.2%, and 58.2%, according to the span, nuclearity, and relation levels respectively. This level of agreement is similar to the inter-annotator agreement rates on the RST Discourse Treebank, i.e., 88.3% on span, 77.3% on nuclearity, and 64.7% on relation, respectively (Joty et al, 2015;Morey et al, 2017).…”

“…In this paper, all the parsers we built were evaluated with the micro F1 score. When using the gold standard syntax trees and EDU segmentations, the F1 scores on three levels of span, nuclearity, and relation can reach 84.1%, 69.6%, and 56.5% respectively, which are close to state-of-the-art accuracy, as reported in Morey et al (2017).…”

“…A wide range of techniques have been applied to this task, and document-level discourse parsers are available (Marcu, 2000a;Sagae, 2009;Hernault et al, 2010;Joty et al, 2013;Li et al, 2014;Ji and Eisenstein, 2014;Li et al, 2016;Liu and Lapata, 2017;Braud et al, 2017;Wang et al, 2017c). Morey et al (2017) replicated the same evaluation procedure on 9 recent parsers, and indicated that the recent gains in discourse parsing can be attributed to the distributed representations.…”

Discourse-Driven Argument Mining in Scientific Abstracts

“…The SciDTB annotations use 17 coarse-grained relation types and 26 finegrained relations. Polynary discourse relations in RST are binarized in SciDTB following a criteria similar to the "right-heavy" transformation used in other works that represent discourse structures as dependency trees [19,28,13], which makes it particularly suitable as input of sequence tagging algorithms.…”

SciDTB: Discourse Dependency TreeBank for Scientific Abstracts

“…The second scenario is that several EDUs are of equal importance in a polynary relation. For this case, we link each former EDU to its neighboring EDU with the same relation, forming a relation chain similar to "right-heavy" binarization transformation in (Morey et al, 2017).…”