IQA: Visual Question Answering in Interactive Environments

Abstract

We introduce Interactive Question Answering (IQA), the task of answering questions that require an autonomous agent to interact with a dynamic visual environment. IQA presents the agent with a scene and a question, like: "Are there any apples in the fridge?" The agent must navigate around the scene, acquire visual understanding of scene elements, interact with objects (e.g. open refrigerators) and plan for a series of actions conditioned on the question. Popular reinforcement learning approaches with a single controller perform poorly on IQA owing to the large and diverse state space. We propose the Hierarchical Interactive Memory Network (HIMN), consisting of a factorized set of controllers, allowing the system to operate at multiple levels of temporal abstraction. To evaluate HIMN, we introduce IQUAD V1, a new dataset built upon AI2-THOR [35], a simulated photo-realistic environment of configurable indoor scenes with interactive objects. 1 IQUAD V1 has 75,000 questions, each paired with a unique scene configuration. Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1. For sample questions and results, please view our video: https://youtu.be/pXd3C-1jr98.

Embodied Question Answering in Photorealistic Environments With Point Cloud Perception

“…Gandhi et al [19] collect a dataset of drone crashes and train self-supervised agents to avoid obstacles. A number of new challenging tasks have been proposed including instruction-based navigation [6,7], target-driven navigation [2,4], embodied/interactive question answering [1,9], and task planning [5].…”

“…We empirically show that point cloud representations are more effective for navigation in this task. Moreover, contrary to [1,9] that use synthetic environments, we extend the task to real environments sourced from [16]. 3D Representations and Architectures.…”

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

“…These approaches have been extended to the video domain as well [20,34,42]. Recently, [15,10] address the problem of question answering in an interactive environment. None of these approaches, however, is designed for leveraging external knowledge so they cannot handle the cases that the image does not represent the full knowledge to answer the question.…”

Interactive Language Learning by Question Answering

“…Several embodied or visual question answering datasets have been presented recently to address some of the problems of interest in our work, such as those of Brodeur et al (2017); Das et al (2017); Gordon et al (2017). In contrast with these, our purely text-based environment circumvents challenges inherent to modelling interactions between separate data modalities.…”