Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality

Abstract

Genre and domain are well known covariates of both manual and automatic annotation quality. Comparatively less is known about the effect of sentence types, such as imperatives, questions or fragments, and how they interact with text type effects. Using mixed effects models, we evaluate the relative influence of genre and sentence types on automatic and manual annotation quality for three related tasks in English data: POS tagging, dependency parsing and coreference resolution. For the latter task, we also develop a new metric for the evaluation of individual regions of coreference annotation. Our results show that while there are substantial differences between manual and automatic annotation in each task, sentence type is generally more important than genre in predicting errors within our data.

Exploring a Multi-Layered Cross-Genre Corpus of Document-Level Semantic Relations

“…We see three directions for future research in this space. This type of quantitative characterization of semantic relations could be extended to other genres [2][3][4]. Alternatively, additional semantic or pragmatic relations could be annotated at both the sentence and document level [9].…”

“…Multi-layered corpora are corpora that are annotated for multiple, mutually independent layers of natural language information on the same text [1][2][3][4][5]. While the contents of one layer may not be directly and immediately inferred from the contents of another, there may nonetheless be some correlation between elements of one layer and another.…”

AMALGUM -- A Free, Balanced, Multilayer English Web Corpus

“…One reviewer has asked how StanfordNLP compares to other available libraries, such as Spacy (https://spacy.io/). While we do not have up to date numbers for Spacy, which was not featured in the recent CoNLL shared task on Universal Dependencies parsing, the most recent numbers reported in(Zeldes and Simonson, 2016) do not suggest that it would outperform StanfordNLP.…”

What to do about non-standard (or non-canonical) language in NLP

“…Interest in this question re-emerged recently. For example, focusing on annotation difficulty, Zeldes and Simonson (2016) remark "that domain adaptation may be folding in sentence type effects", motivated by earlier findings by Silveira et al (2014) who remark that "[t]he most striking difference between the two types of data [Web and newswire] has to do with imperatives, which occur two orders of magnitude more often in the EWT [English Web Treebank]." A very recent paper examines word order properties and their impact on parsing taking a control experiment approach (Gulordava and Merlo, 2016).…”