RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

Matt, Deitke,; Han, Winson; Herrasti, Alvaro; Kembhavi, Aniruddha; Kolve, Eric; Mottaghi, Roozbeh; Salvador, Jordi; Schwenk, Dustin; Eli, VanderBilt,; Wallingford, Matthew; Weihs, Luca; Yatskar, Mark; Farhadi, Ali

doi:10.1109/cvpr42600.2020.00323

Cited by 166 publications

(159 citation statements)

References 42 publications

Supporting

Mentioning

156

Contrasting

Order By: Relevance

“…We further test our model's generalization capability to objects and rooms of novel types. In Table 2, we report quantitative evaluations where we train our systems on one type of AI2THOR rooms and test over the other types, as well as additional results of testing over a totally out-of-distribution RoboTHOR [Deitke et al, 2020] test dataset (see Fig. A.7).…”

Section: Results and Analysismentioning

confidence: 99%

IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

Li¹,

Mo²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Building embodied intelligent agents that can interact with 3D indoor environments has received increasing research attention in recent years. While most works focus on single-object or agent-object visual functionality and affordances, our work proposes to study a new kind of visual relationship that is also important to perceive and model -inter-object functional relationships (e.g., a switch on the wall turns on or off the light, a remote control operates the TV). Humans often spend little or no effort to infer these relationships, even when entering a new room, by using our strong prior knowledge (e.g., we know that buttons control electrical devices) or using only a few exploratory interactions in cases of uncertainty (e.g., multiple switches and lights in the same room). In this paper, we take the first step in building AI system learning inter-object functional relationships in 3D indoor environments with key technical contributions of modeling prior knowledge by training over large-scale scenes and designing interactive policies for effectively exploring the training scenes and quickly adapting to novel test scenes. We create a new benchmark based on the AI2Thor and PartNet datasets and perform extensive experiments that prove the effectiveness of our proposed method. Results show that our model successfully learns priors and fast-interactive-adaptation strategies for exploring inter-object functional relationships in complex 3D scenes. Several ablation studies further validate the usefulness of each proposed module.

show abstract

Section: Results and Analysismentioning

confidence: 99%

IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

Li¹,

Mo²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Another dimension of laboratory automation is the increasing use of robots and robotic equipment[ 3 , 16 , 25 ]. A synergistic combination of human–machine collaboration across a spectrum of operations provides a realistic use case for robotics, particularly in the post-pandemic era [25] .…”

Section: Discussionmentioning

confidence: 99%

“…With recent developments, particularly in the role of game simulations for training machine learning algorithms, AI technologies have made significant strides in modeling and learning complex, multicomponent systems[ 15 , 16 , 17 , 18 ]. Simulation models, which capture various layers and constraints, play a pivotal role prior to the building of actual systems, and they require significant cost and time resources.…”

Section: Introductionmentioning

confidence: 99%

AI-driven laboratory workflows enable operation in the age of social distancing

Marescotti

Narayanamoorthy²,

Bonjour

et al. 2022

SLAS Technology

View full text Add to dashboard Cite

“…Previous research works using reinforcement learning have studied the problem on simplified block worlds (Janner et al, 2018;Bisk et al, 2018), which could be far from being realistic. The recent interests on embodied artificial intelligence (embodied AI) have contributed to several realistic simulation environments, such as Gibson (Xia et al, 2018), Habitat (Savva et al, 2019), RoboTHOR (Deitke et al, 2020), and ALFRED (Shridhar et al, 2020). However, because of physical constraints in a real environment, gap between a simulation environment and a real world still exists (Deitke et al, 2020;Shridhar et al, 2021).…”

Section: Language-based Human-robot Interactionmentioning

confidence: 99%

“…The recent interests on embodied artificial intelligence (embodied AI) have contributed to several realistic simulation environments, such as Gibson (Xia et al, 2018), Habitat (Savva et al, 2019), RoboTHOR (Deitke et al, 2020), and ALFRED (Shridhar et al, 2020). However, because of physical constraints in a real environment, gap between a simulation environment and a real world still exists (Deitke et al, 2020;Shridhar et al, 2021). Researchers have also explored the idea of finding a mapping between vision signals of a real robot and language signals directly (Blukis et al, 2020), but this mapping requires detailed annotated data and it is usually expensive to obtain physical interaction data.…”

Section: Language-based Human-robot Interactionmentioning

confidence: 99%

Meta-Reinforcement Learning for Mastering Multiple Skills and Generalizing across Environments in Text-based Games

Zhao¹,

Sun²,

Ma³

2021

Proceedings of the 1st Workshop on Meta Learning and Its Applications to Natural Language Processing

View full text Add to dashboard Cite

Text-based games can be used to develop taskoriented text agents for accomplishing tasks with high-level language instructions, which has potential applications in domains such as human-robot interaction. Given a text instruction, reinforcement learning is commonly used to train agents to complete the intended task owing to its convenience of learning policies automatically. However, because of the large space of combinatorial text actions, learning a policy network that generates an action word by word with reinforcement learning is challenging. Recent research works show that imitation learning provides an effective way of training a generation-based policy network. However, trained agents with imitation learning are hard to master a wide spectrum of task types or skills, and it is also difficult for them to generalize to new environments. In this paper, we propose a meta-reinforcement learning based method to train text agents through learning-to-explore. In particular, the text agent first explores the environment to gather task-specific information and then adapts the execution policy for solving the task with this information. On the publicly available testbed ALFWorld, we conducted a comparison study with imitation learning and show the superiority of our method.

show abstract

RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

Cited by 166 publications

References 42 publications

IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

AI-driven laboratory workflows enable operation in the age of social distancing

Meta-Reinforcement Learning for Mastering Multiple Skills and Generalizing across Environments in Text-based Games

Contact Info

Product

Resources

About