Validating the web-based evaluation of NLG systems

Abstract

The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internetbased approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.

User preferences can drive facial expressions: evaluating an embodied conversational agent in a recommender dialogue system

“…Such effects have been found in previous studies; for example, Krahmer and Swerts (2004) found that Dutch and Italian subjects perceived the role of prosodically linked eyebrow movements differently. However, Koller et al (2009) have recently replicated in a lab-based study the results from an online study of a set of natural-language generation systems. This indicates that the results from Internet-based evaluation of generated output can be reliable despite the diverse subject pool; however, future lab-based experiments may be advisable to confirm this for this particular task.…”

A Flexible Dual Task Paradigm for Evaluating an Embodied Conversational Agent: Modality Effects and Reaction Time as an Index of Cognitive Load

“…Taxonomies [e.g., 1] and frameworks [e.g., 2] have been proposed often emphasizing the need to distinguish features of user, agent, and task. It is more common now for evaluation of ECAs, or component models such as natural language generation or text to speech (TTS) synthesis systems, to consist of both objective and subjective measures [2][3][4][5][6][7]. There are still instances, however, where data are collected in the absence of the manipulation of specific variables (comparison conditions) or without a control condition [e.g.…”