Generating Referring Expressions in Context: The GREC Task Evaluation Challenges

Abstract

Abstract. Until recently, referring expression generation (reg) research focused on the task of selecting the semantic content of definite mentions of listener-familiar discourse entities. In the grec research programme we have been interested in a version of the reg problem definition that is (i) grounded within discourse context, (ii) embedded within an application context, and (iii) informed by naturally occurring data. This paper provides an overview of our aims and motivations in this research programme, the data resources we have built, and the first three sharedtask challenges, grec-msr'08, grec-msr'09 and grec-neg'09, we have run based on the data.

Models and empirical data for the production of referring expressions

“…Viethen & Dale, 2007;van Deemter, Gatt, van der Sluis, & Power, 2012). Recent work in the context of a series of nlg shared tasks, in which participants are required to design algorithms that are developed and tested against a common dataset to enable comparison, has shown that results from these two perspectives may diverge significantly Belz, Kow, Viethen, & Gatt, 2010). For instance, an algorithm's choice of content for referential descriptions may be very similar to the choices humans make, as shown by its degree of match to corpus data, but this does not imply that the resulting description will be easily resolved by human listeners.…”

Assessing the Trade-Off between System Building Cost and Output Quality in Data-to-Text Generation

“…What we have found again and again in our evaluation experiments (see also Gatt & Belz [12], and Belz et al [6], both in this volume), is that different types of evaluation do not necessarily agree with each other, and that we should not regard any single one of them as an objective measure of quality, but rather as assessing one particular aspect of systems. If we want our wind forecasts to be similar to the corpus forecasts, then BLEU and NIST can probably give us a fair indication of that type of similarity; if we are interested in how readable and clear human readers (think they) find our forecasts, then we should look at the Clarity and Readability scores.…”

“…The two automatic metrics used in the evaluations, NIST 5 and BLEU, 6 in the METEO domain [3]. BLEU-x is an n-gram based string comparison measure, originally proposed by Papineni et al [19] for evaluation of MT systems.…”

Proceedings of the 9th International Natural Language Generation conference

“…This tasks aimed at the unique identification of the referent and brevity of the referring expression. Slightly different, the GREC challenges Belz et al, 2009;Belz et al, 2010) propose the generation of referring expressions in a discourse context. The GREC tasks use a corpus created from Wikipedia abstracts on geographic entities and people and with two referring expression annotation schemes, reference type and word strings.…”