Linguistic Annotation

Palmer, Martha; Xue, Nianwen

doi:10.1002/9781444324044.ch10

Cited by 8 publications

(3 citation statements)

References 708 publications

(1,111 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regarding the use of IAA in corpus annotation tasks, and more specifically the task of linguistic annotation, we refer to Palmer and Xue (2005) and Pustejovsky and Stubbs (2013). Both provide extensive theoretical descriptions of how to perform 4 Some examples of correlation coefficients are Kendall's τ , Pearson's r, Spearman's ρ and Goodman and Kruskall's Gamma.…”

Section: Related Workmentioning

confidence: 99%

Agreement is overrated: A plea for correlation to assess human evaluation reliability

Amidei¹,

Piwek²,

Willis³

2019

Proceedings of the 12th International Conference on Natural Language Generation

View full text Add to dashboard Cite

Inter-Annotator Agreement (IAA) is used as a means of assessing the quality of NLG evaluation data, in particular, its reliability. According to existing scales of IAA interpretationsee, for example, Lommel et al. (2014), Liu et al. (2016), Sedoc et al. (2018) and Amidei et al. (2018a)-most data collected for NLG evaluation fail the reliability test. We confirmed this trend by analysing papers published over the last 10 years in NLG-specific conferences (in total 135 papers that included some sort of human evaluation study). Following Sampson and Babarczy (2008), Lommel et al. (2014), Joshi et al. (2016) and Amidei et al. (2018b), such phenomena can be explained in terms of irreducible human language variability. Using three case studies, we show the limits of considering IAA as the only criterion for checking evaluation reliability. Given human language variability, we propose that for human evaluation of NLG, correlation coefficients and agreement coefficients should be used together to obtain a better assessment of the evaluation data reliability. This is illustrated using the three case studies.

show abstract

Section: Related Workmentioning

confidence: 99%

Agreement is overrated: A plea for correlation to assess human evaluation reliability

Amidei¹,

Piwek²,

Willis³

2019

Proceedings of the 12th International Conference on Natural Language Generation

View full text Add to dashboard Cite

show abstract

“…The scheme represents the theoretical backbone of the evaluation process. The guidelines that supplement the scheme provide additional information, often with examples, making clear the scheme usage (Palmer and Xue, 2010). In the original evaluation, the guidelines were minimal.…”

Section: Introductionmentioning

confidence: 99%

Proceedings of the 9th International Natural Language Generation conference

Kok

2016

View full text Add to dashboard Cite

“…For readable and instructive discussions of using corpora (including treebanks) for linguistic research, seeMeurers (2005),Meurers and Müller (2009) andKübler and Zinsmeister (2015) Palmer and Xue (2013). contains a concise and useful discussion of treebank development from the viewpoint of corpus developers.3 In the case of NPCMJ, the annotation manual is available at http://www.…”

mentioning

confidence: 99%

Probing the nature of an island constraint with a parsed corpus: A case study on the Coordinate Structure Constraint in Japanese

Kubota

Kubota²

2019

LILT

View full text Add to dashboard Cite

This paper presents a case study of the use of the NINJAL Parsed Corpus of Modern Japanese (NPCMJ) for syntactic research. NPCMJ is the first phrase structure-based treebank for Japanese that is specifically designed for application in linguistic (in addition to NLP) research. After discussing some basic methodological issues pertaining to the use of treebanks for theoretical linguistics research, we introduce our case study on the status of the Coordinate Structure Constraint (CSC) in Japanese, showing that NPCMJ enables us to easily retrieve examples that support one of the key claims of Kubota and Lee (2015): that the CSC should be viewed as a pragmatic, rather than a syntactic constraint. The corpus-based study we conducted moreover revealed a previously unnoticed tendency that was highly relevant for further clarifying the principles governing the empirical data in question. We conclude the paper by briefly discussing some further methodological issues brought up by our case study pertaining to the relationship between linguistic research and corpus development.

show abstract

Linguistic Annotation

Cited by 8 publications

References 708 publications

Agreement is overrated: A plea for correlation to assess human evaluation reliability

Agreement is overrated: A plea for correlation to assess human evaluation reliability

Proceedings of the 9th International Natural Language Generation conference

Probing the nature of an island constraint with a parsed corpus: A case study on the Coordinate Structure Constraint in Japanese

Contact Info

Product

Resources

About