Report on the first NLG Challenge on Generating Instructions in Virtual Environments (GIVE)

Abstract

We describe the first installment of the Challenge on Generating Instructions in Virtual Environments (GIVE), a new shared task for the NLG community. We motivate the design of the challenge, describe how we carried it out, and discuss the results of the system evaluation.

Referring Expression Generation under Uncertainty: Algorithm and Evaluation Framework

“…That is, this methodology does not necessarily assess how well a generated RE actually allows a target to be picked out by a hearer -the presumed purpose of REG algorithms (Van Deemter and Gatt, 2009). Recently, there has been a shift towards task-based evaluations (e.g., (Byron et al, 2009;Koller et al, 2010;Viethen and Dale, 2006)), in which algorithms are compared by how well they allow some task to be achieved.…”

Empirical Methods in Natural Language Generation

“…These kinds of shared tasks, including the availability of benchmark data sets and standardized evaluation procedures, have proven to be an important impetus on developments in other areas of NLP, and already a similar effect can be observed for the various NLG shared tasks ("generation challenges") for referring expression generation [1], for generation of references to named entities in text [2] and for instruction giving in virtual environments [3]. These generation challenges not only resulted in new-generation research, but also in a better understanding of evaluation and evaluation metrics for generation algorithms.…”

“…The selected chapters are mostly thoroughly revised and extended versions of original research that was presented at the 12th European Workshop on Natural Language Generation (ENLG 2009) …”

Validating the web-based evaluation of NLG systems

“…In the GIVE scenario (Byron et al, 2009), users try to solve a treasure hunt in a virtual 3D world that they have not seen before. The computer has complete information about the virtual world.…”

“…The GIVE Challenge (Byron et al, 2009) is a recent shared task which takes a third approach to NLG evaluation: By connecting NLG systems to experimental subjects over the Internet, it achieves a true task-based evaluation at a much lower cost. Indeed, the first GIVE Challenge acquired data from over 1100 experimental subjects online.…”