AI2-THOR: An Interactive 3D Environment for Visual AI

Kolve, Eric; Mottaghi, Roozbeh; Han, Winson; Eli, VanderBilt,; Weihs, Luca; Herrasti, Alvaro; Matt, Deitke,; Ehsani, Kiana; Gordon, Daniel; Zhu, Yuke; Kembhavi, Aniruddha; Gupta, Abhinav; Farhadi, Ali

doi:10.48550/arxiv.1712.05474

Cited by 186 publications

(321 citation statements)

References 0 publications

Supporting

Mentioning

321

Contrasting

Order By: Relevance

“…However, they often require many hand-designed rules or scale inadequately to more complex tasks and environments. Recently, many efforts have been put into creating more realistic environments with the goal to further advances in this area [32,40,41,19,37,1]. At the same time, by leveraging the better representation power of neural architectures, a number of works have looked into creating instruction-following agents that can perform manipulation [24,25], navigation [11,47,26], or both [42,14,12].…”

Section: Related Workmentioning

confidence: 99%

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Huang¹,

Abbeel²,

Pathak³

et al. 2022

Preprint

View full text Add to dashboard Cite

Can world knowledge learned by large language models (LLMs) be used to act in interactive environments? In this paper, we investigate the possibility of grounding high-level tasks, expressed in natural language (e.g. "make breakfast"), to a chosen set of actionable steps (e.g. "open fridge"). While prior work focused on learning from explicit step-by-step examples of how to act, we surprisingly find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans without any further training. However, the plans produced naively by LLMs often cannot map precisely to admissible actions. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions. Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline. The conducted human evaluation reveals a trade-off between executability and correctness but shows a promising sign towards *Equal advising. Correspondence to Wenlong Huang

show abstract

Section: Related Workmentioning

confidence: 99%

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Huang¹,

Abbeel²,

Pathak³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Thanks to the recent progress of physics simulation (Todorov et al 2012;Coumans and Bai 2016;Erez et al 2015), it has drawn increasing interest to build full-physics robot simulation environment (Urakami et al 2019;Tunyasuvunakool et al 2020;Zhu et al 2020;James et al 2020;Mu et al 2021). Compared to robot simulation with abstract action (Kolve et al 2017;Savva et al 2019;, full-physic robot simulation supports low-level policy learning that could be transferred to real world. Therefore, we integrate our depth simulator with state-ofthe-art robot simulator SAPIEN (Xiang et al 2020), targeting at vision-based robot policy learning.…”

Section: Robot Simulationmentioning

confidence: 99%

Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

Zhang¹,

Chen²,

Xiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we focus on the simulation of active stereovision depth sensors, which are popular in both academic and industry communities. Inspired by the underlying mechanism of the sensors, we designed a fully physics-grounded simulation pipeline, which includes material acquisition, ray tracing based infrared (IR) image rendering, IR noise simulation, and depth estimation. The pipeline is able to generate depth maps with material-dependent error patterns similar to a real depth sensor. We conduct extensive experiments to show that perception algorithms and reinforcement learning policies trained in our simulation platform could transfer well to real world test cases without any fine-tuning. Furthermore, due to the high degree of realism of this simulation, our depth sensor simulator can be used as a convenient testbed to evaluate the algorithm performance in the real world, which will largely reduce the human effort in developing robotic algorithms. The entire pipeline has been integrated into the SAPIEN simulator and is open-sourced to promote the research of vision and robotics communities.

show abstract

“…Recent progress in Embodied Artificial Intelligence, spans both simulation environments Kolve et al (2017); Li et al (2021); Savva et al (2019); Gan et al (2020); Puig et al (2018) and sophisticated tasks Das et al (2018); Anderson et al (2018); Shridhar et al (2020). Our work is most closely related to research in language-guided task completion, Neural SLAM, and exploration.…”

Section: Related Workmentioning

confidence: 99%

“…We focus on the ALFRED challenge Shridhar et al (2020), where an agent is asked to follow human instructions to complete long-horizon household tasks in indoor scenes (simulated in AI2Thor Kolve et al (2017)). Each task in ALFRED consists of several subgoals for either navigation (moving in the environment) or object interactions (interacting with at least one object).…”

Section: Problem Formulationmentioning

confidence: 99%

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Jia¹,

Lin²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent years have witnessed an emerging paradigm shift toward embodied artificial intelligence, in which an agent must learn to solve challenging tasks by interacting with its environment. There are several challenges in solving embodied multimodal tasks, including long-horizon planning, vision-and-language grounding, and efficient exploration. We focus on a critical bottleneck, namely the performance of planning and navigation. To tackle this challenge, we propose a Neural SLAM approach that, for the first time, utilizes several modalities for exploration, predicts an affordance-aware semantic map, and plans over it at the same time. This significantly improves exploration efficiency, leads to robust long-horizon planning, and enables effective vision-and-language grounding. With the proposed Affordance-aware Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement over prior published work on the ALFRED benchmark and set a new state-of-the-art generalization performance at a success rate of 23.48% on the test unseen scenes.

show abstract

AI2-THOR: An Interactive 3D Environment for Visual AI

Cited by 186 publications

References 0 publications

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Contact Info

Product

Resources

About