A Corpus of Natural Language for Visual Reasoning

Abstract

We present a new visual reasoning language dataset, containing 92,244 pairs of examples of natural statements grounded in synthetic images with 3,962 unique sentences. We describe a method of crowdsourcing linguistically-diverse data, and present an analysis of our data. The data demonstrates a broad set of linguistic phenomena, requiring visual and set-theoretic reasoning. We experiment with various models, and show the data presents a strong challenge for future research.

A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context

“…To elicit this property, we sampled all attributes of the entities uniformly at random, with the only restriction that the entities cannot be too close to each other. As the previous work with similar idea confirmed (Suhr et al 2017), we found rich varieties of linguistic phenomena, including cardinalities ("three gray dots"), existentials ("There is another small dark .."), universals ("all of the other dots are larger"), coordinations and negations ("further to the right and not as far down").…”

“…• In a categorical context (Bordes and Weston 2016;He et al 2017;Lewis et al 2017), information can be expressed by symbolic natural language without ambiguity. For example, there could be little ambiguity in describing categorical properties, such as discrete color (red, blue and yellow).…”

“…Second of all, to enable training of recent end-to-end dialogue systems with high flexibility (Bordes and Weston 2016;Lewis et al 2017), we collected a largescale dataset of 6,760 human dialogues with over 32K utterances through crowdsourcing on Amazon Mechanical Turk. During the dataset collection, we managed to fulfill three essential requirements of natural language corpora: interpretability, linguistic/strategic variety and reliability.…”

“…Recently, a new line of data-driven dialogue systems, which we refer to as end-to-end dialogue systems, has been gaining attention in both task-oriented (Bordes and Weston 2016;Lewis et al 2017) and non-task-oriented domains (Vinyals and Le 2015;Serban et al 2016). In this approach, dialogue state and utterance generation are learned directly from large raw corpora with little prior constraints, so they are more suitable for complex common grounding where flexibility is a requirement.…”

“…In this approach, dialogue state and utterance generation are learned directly from large raw corpora with little prior constraints, so they are more suitable for complex common grounding where flexibility is a requirement. However, few existing tasks focus on the difficulty of common grounding and most are based on either fully-observable or categorical context (De Vries et al 2017;Bordes and Weston 2016;Lewis et al 2017) where difficult common grounding is not required. A dataset closest to our setting is the Mutu-alFriends dataset (He et al 2017), which is based on the task of finding a mutual friend from private lists of friends.…”