We study the problem of jointly reasoning about language and vision through a navigation and spatial reasoning task. We introduce the TOUCHDOWN task and dataset, where an agent must first follow navigation instructions in a real-life visual urban environment, and then identify a location described in natural language to find a hidden object at the goal position. The data contains 9,326 examples of English instructions and spatial descriptions paired with demonstrations. Empirical analysis shows the data presents an open challenge to existing methods, and qualitative linguistic analysis shows that the data displays richer use of spatial reasoning compared to related resources. The environment and data are available at https://touchdown.ai. the dumpster has a blue tarp draped over the end closest to you. touchdown is on the top of the blue tarp on the dumpster.LINGUNET The model correctly predicts the location of Touchdown, putting most of the predicted distribution (green) on the top-left of the dumpster at the center. 3TEXT2CONV The model incorrectly predicts the location of Touchdown to the top of the car on the far right. While some of the probability mass is correctly placed on the dumpster, the pixel with the highest probability is on the car. 3CONCATCONV The model correctly predicts the location of Touchdown. The distribution is heavily concentrated at a couple of nearby pixels. 3CONCAT The prediction is similar to CONCATCONV.3 Figure 9. Three of the models are doing fairly well. Only TEXT2CONV fails to predict the location of Touchdown.turn to your right and you will see a green trash barrel between the two blue benches on the right. click to the base of the green trash barrel to find touchdown.LINGUNET The model accurately predicts the green trash barrel on the right as Touchdown's location. 41TEXT2CONV The model predicts successfully as well. The distribution is focused on a smaller area compared to LIN-GUNET closer to the top of the object. This possibly shows a learned bias towards placing Touchdown on the top of objects that TEXT2CONV is more suceptible to. 41CONCATCONV The model prediction is correct. The distribution is focused on fewer pixels compared to LINGUNET. 41CONCAT The model prediction is correct. Similar to CONCATCONV, it focuses on a few pixels. 41 Figure 10. All the models predict the location of Touchdown correctly. Trash can is a relatively common object that workers use to place Touchdown in the dataset . on your right is a parking garage, there is a red sign with bikes parked out in front of the garage, the bear is on the red sign.LINGUNET The model predicted the location of Touchdown correctly to the red stop sign on the right side. 59TEXT2CONV The model predicts the location of Touchdown correctly. 59CONCATCONV The model predicts the location of Touchdown correctly. 59CONCAT The model predicts the location of Touchdown correctly. Figure 11. All the models predict the location of Touchdown correctly. Reference to a red sign are relatively common in the data (Figure 8) potentially sim...

We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge. * Contributed equally. † Work done as an undergraduate at Cornell University. 1 In parts of this paper, we use the term compositional differently than it is commonly used in linguistics to refer to reasoning that requires composition. This type of reasoning often manifests itself in highly compositional language.The left image contains twice the number of dogs as the right image, and at least two dogs in total are standing.One image shows exactly two brown acorns in back-to-back caps on green foliage.

We present a new visual reasoning language dataset, containing 92,244 pairs of examples of natural statements grounded in synthetic images with 3,962 unique sentences. We describe a method of crowdsourcing linguistically-diverse data, and present an analysis of our data. The data demonstrates a broad set of linguistic phenomena, requiring visual and set-theoretic reasoning. We experiment with various models, and show the data presents a strong challenge for future research.

We propose a context-dependent model to map utterances within an interaction to executable formal queries. To incorporate interaction history, the model maintains an interaction-level encoder that updates after each turn, and can copy sub-sequences of previously predicted queries during generation. Our approach combines implicit and explicit modeling of references between utterances. We evaluate our model on the ATIS flight planning interactions, and demonstrate the benefits of modeling context and explicit references.

We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets. * Work done during an internship at Google. Advising (Finegan-Dollak et al., 2018) NL: For EECS 478, how many credits is it? SQL: select distinct credits from course where department ='EECS' and number = 478; GeoQuery (Zelle and Mooney, 1996) NL: How many people live in mississippi? SQL: select population from state where state name = 'mississippi'; ATIS (Hemphill et al., 1990; Dahl et al., 1994) NL: Flights from Phoenix to Milwaukee SQL: select distinct T1.flight id from airport service as T2, airport service as T3, city as T4, city as T5, flight as T1 where T4.city code = T2.city code and T4.city name = 'Phoenix' and T5.city code = T3.city code and T5.city name = 'Milwaukee' and T1.from airport = T2.airport code and T1.to airport = T3.airport code; Spider (Yu et al., 2018) NL: List the emails of the professionals who live in the state of Hawaii or the state of Wisconsin.

We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accomplish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this scenario, and learn to map user instructions to system actions. We introduce a learning approach focused on recovery from cascading errors between instructions, and modeling methods to explicitly reason about instructions with multiple goals. We evaluate with a new evaluation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities.

No abstract

We propose a learning approach for mapping context-dependent sequential instructions to actions. We address the problem of discourse and state dependencies with an attention-based model that considers both the history of the interaction and the state of the world. To train from start and goal states without access to demonstrations, we propose SESTRA, a learning algorithm that takes advantage of singlestep reward observations and immediate expected reward maximization. We evaluate on the SCONE domains, and show absolute accuracy improvements of 9.8%-25.3% across the domains over approaches that use high-level logical representations.