Visual Room Rearrangement

Weihs, Luca; Matt, Deitke,; Kembhavi, Aniruddha; Mottaghi, Roozbeh

doi:10.1109/cvpr46437.2021.00586

Cited by 55 publications

(30 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To train policies, we use the AI2-iTHOR [32] simulator where agents can navigate: A N = {move forward, turn left/right 90 • , look up/down 30 • }, and interact with objects: A I = {take, put, open, close, toggle-on, toggle-off, slice}. Our action space of size |A|= 110 is the union of all navigation actions and valid object interactions following [62]. We use all 30 kitchen scenes from AI2-iTHOR, split into training (25) and testing (5) sets.…”

Section: Methodsmentioning

confidence: 99%

Shaping embodied agent behavior with activity-context priors from egocentric video

Nagarajan,

Grauman

2021

Preprint

View full text Add to dashboard Cite

Complex physical tasks entail a sequence of object interactions, each with its own preconditions-which can be difficult for robotic agents to learn efficiently solely through their own experience. We introduce an approach to discover activitycontext priors from in-the-wild egocentric video captured with human worn cameras. For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e.g., a knife and cutting board brought together with a tomato are conducive to cutting). We encode our video-based prior as an auxiliary reward function that encourages an agent to bring compatible objects together before attempting an interaction. In this way, our model translates everyday human experience into embodied agent skills. We demonstrate our idea using egocentric EPIC-Kitchens video of people performing unscripted kitchen activities to benefit virtual household robotic agents performing various complex tasks in AI2-iTHOR, significantly accelerating agent learning. Project page: http://vision.cs.utexas.edu/projects/ego-rewards/

show abstract

Section: Methodsmentioning

confidence: 99%

Shaping embodied agent behavior with activity-context priors from egocentric video

Nagarajan,

Grauman

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Similarly, not many datasets provide information about object associations between two recordings. The datasets can be separated into synthetic frames-wise annotated datasets ( Park et al, 2021 ; Weihs et al, 2021 ) and real-world datasets where the 3D map is annotated ( Wald et al, 2019 ; Langer et al, 2020 ). Based on the task definitions from Batra et al( Batra et al, 2020 ), Weihs et al ( Weihs et al, 2021 ) introduced a new dataset with object rearrangements in a virtual environment for studying how robots explore their environment.…”

Section: Related Workmentioning

confidence: 99%

“…This does not represent the real world that must consider objects that appear, and are therefore unknown, or disappear. Today most approaches assume a given and fixed set of objects, e.g., Bore et al (2018 ) and Weihs et al (2021 ). To develop more general methods, the task of open-world object detection is recently defined by Joseph et al (2021 ).…”

Section: Introductionmentioning

confidence: 99%

Where Does It Belong? Autonomous Object Mapping in Open-World Settings

2022

View full text Add to dashboard Cite

Detecting changes such as moved, removed, or new objects is the essence for numerous indoor applications in robotics such as tidying-up, patrolling, and fetch/carry tasks. The problem is particularly challenging in open-world scenarios where novel objects may appear at any time. The main idea of this paper is to detect objects from partial 3D reconstructions of interesting areas in the environment. In our pipeline we first identify planes, consider clusters on top as objects, and compute their point-pair-features. They are used to match potential objects and categorize them robustly into static, moved, removed, and novel objects even in the presence of partial object reconstructions and clutter. Our approach dissolves heaps of objects without specific object knowledge, but only with the knowledge acquired from change detection. The evaluation is performed on real-world data that includes challenges affecting the quality of the reconstruction as a result of noisy input data. We present the novel dataset ObChange for quantitative evaluation, and we compare our method against a baseline using learning-based object detection. The results show that, even with a targeted training set, our approach outperforms the baseline for most test cases. Lastly, we also demonstrate our method’s effectiveness in real robot experiments.

show abstract

“…[52] tackle 'interactive navigation', where the robot can bump into and push objects during navigation, but does not have an arm. Some works [56][57][58] abstract away gross motor control entirely by using symbolic interaction capabilities (e.g. a 'pick up X' action) or a 'magic pointer' [9].…”

Section: Related Workmentioning

confidence: 99%

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Szot¹,

Clegg²,

Undersander³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack -data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850× real-time) on an 8-GPU node, representing 100× speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies. Figure 1: A mobile manipulator (Fetch robot) simulated in Habitat 2.0 performing rearrangement tasks in a ReplicaCAD apartment -(left) opening a drawer before picking up an item from it, and (right) placing an object into the bowl after navigating to the table. Best viewed in motion at https://sites.google.com/view/habitat2. Preprint. Under review.

show abstract

Visual Room Rearrangement

Cited by 55 publications

References 32 publications

Shaping embodied agent behavior with activity-context priors from egocentric video

Shaping embodied agent behavior with activity-context priors from egocentric video

Where Does It Belong? Autonomous Object Mapping in Open-World Settings

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Contact Info

Product

Resources

About