Reasoning about Pragmatics with Neural Listeners and Speakers

Andreas, Jacob; Klein, Dan

doi:10.18653/v1/d16-1125

Cited by 110 publications

(159 citation statements)

References 14 publications

(15 reference statements)

Supporting

Mentioning

159

Contrasting

Order By: Relevance

“…Pragmatic approaches in this category (Frank and Goodman, 2012;Andreas and Klein, 2016;Vedantam et al, 2017;Cohn-Gordon et al, 2018) derive pragmatic behavior by producing outputs that distinguish the input i from an alternate distractor input (or inputs). We construct a distractor ı for a given input i in a task-dependent way.…”

Section: Distractor-based Pragmaticsmentioning

confidence: 99%

See 1 more Smart Citation

Pragmatically Informative Text Generation

Shen¹,

Fried²,

Andreas³

et al. 2019

Proceedings of the 2019 Conference of the North

Self Cite

View full text Add to dashboard Cite

We improve the informativeness of models for conditional text generation using techniques from computational pragmatics. These techniques formulate language production as a game between speakers and listeners, in which a speaker should generate output text that a listener can use to correctly identify the original input that the text describes. While such approaches are widely used in cognitive science and grounded language learning, they have received less attention for more standard language generation tasks. We consider two pragmatic modeling methods for text generation: one where pragmatics is imposed by information preservation, and another where pragmatics is imposed by explicit modeling of distractors. We find that these methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations.

show abstract

Section: Distractor-based Pragmaticsmentioning

confidence: 99%

“…Our work builds on a line of learned Rational Speech Acts (RSA) models (Monroe and Potts, 2015;Andreas and Klein, 2016), in which generated strings are selected to optimize the behav- Human-written A cheap coffee shop in riverside with a 5 out of 5 customer rating is Fitzbillies. Fitzbillies is family friendly and serves English food.…”

Section: Introductionmentioning

confidence: 99%

Pragmatically Informative Text Generation

Shen¹,

Fried²,

Andreas³

et al. 2019

Proceedings of the 2019 Conference of the North

Self Cite

View full text Add to dashboard Cite

show abstract

“…The main difference between BISON and existing captioning scores is in how they rank the ability of humans to generate captions: all three systems outperform humans in terms of nearly all captioning scores, but they all perform substantially worse than humans in terms of BISON accuracy 5 . Unless one believes that current image captioning systems actually exhibit super-human performance, this suggests that measuring the BISON score of a system provides a more realistic assessment of the capabilities of modern image captioning systems compared to humans.…”

Section: Resultsmentioning

confidence: 98%

“…Unless one believes that current image captioning systems actually exhibit super-human performance, this suggests that measuring the BISON score of a system provides a more realistic assessment of the capabilities of modern image captioning systems compared to humans. 5 Please note that the accuracy of humans on the BISON task is 100% by definition due to the way the COCO-BISON dataset was collected.…”

Section: Resultsmentioning

confidence: 99%

Evaluating Text-to-Image Matching using Binary Image Selection (BISON)

Misra

Maaten

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Providing systems the ability to relate linguistic and visual content is one of the hallmarks of computer vision. Tasks such as text-based image retrieval and image captioning were designed to test this ability, but come with evaluation measures that have high variance or are difficult to interpret. We study an alternative task for systems that match text and images: given a text query, the system is asked to select the image that best matches the query from a pair of semantically similar images. The system's accuracy on this Binary Image SelectiON (BISON) task is interpretable, eliminates the reliability problems of retrieval evaluations, and focuses on the system's ability to understand fine-grained visual structure. We gather a BISON dataset that complements the COCO dataset and use it to evaluate modern text-based image retrieval and image captioning systems. Our results provide novel insights into the performance of these systems. * This work was performed while Hexiang Hu was at Facebook.Text query: Plates filled with carrots and beets on a white table.

show abstract

“…Meanwhile, the tasks during of such interaction may be essential during learning too (Goodman and Frank 2016). Recent experiments (Rohlfing 2016;Andreas and Klein 2016) and also proposed that language learning should be posited in the context of task-directed behaviours.…”

Section: Language Understanding For Robot Systemsmentioning

confidence: 99%

Sensorimotor input as a language generalisation tool: a neurorobotics model for generation and generalisation of noun-verb combinations with sensorimotor inputs

et al. 2018

View full text Add to dashboard Cite

The paper presents a neurorobotics cognitive model explaining the understanding and generalisation of nouns and verbs combinations when a vocal command consisting of a verb-noun sentence is provided to a humanoid robot. The dataset used for training was obtained from object manipulation tasks with a humanoid robot platform; it includes 9 motor actions and 9 objects placing placed in 6 different locations), which enables the robot to learn to handle real-world objects and actions. Based on the multiple time-scale recurrent neural networks, this study demonstrates its generalisation capability using a large data-set, with which the robot was able to generalise semantic representation of novel combinations of noun-verb sentences, and therefore produce the corresponding motor behaviours. This generalisation process is done via the grounding process: different objects are being interacted, and associated, with different motor behaviours, following a learning approach inspired by developmental language acquisition in infants. Further analyses of the learned network dynamics and representations also demonstrate how the generalisation is possible via the exploitation of this functional hierarchical recurrent network.

show abstract

Reasoning about Pragmatics with Neural Listeners and Speakers

Cited by 110 publications

References 14 publications

Pragmatically Informative Text Generation

Pragmatically Informative Text Generation

Evaluating Text-to-Image Matching using Binary Image Selection (BISON)

Sensorimotor input as a language generalisation tool: a neurorobotics model for generation and generalisation of noun-verb combinations with sensorimotor inputs

Contact Info

Product

Resources

About