Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Abstract

We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack -data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850× real-time) on an 8-GPU node, representing 100× speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies. Figure 1: A mobile manipulator (Fetch robot) simulated in Habitat 2.0 performing rearrangement tasks in a ReplicaCAD apartment -(left) opening a drawer before picking up an item from it, and (right) placing an object into the bowl after navigating to the table. Best viewed in motion at https://sites.google.com/view/habitat2. Preprint. Under review.

PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision

“…Hypersim [16] and OpenRooms [17] develop simulators for indoor object detection. Robotic simulators include AI-2THOR [18], Habitat [19,20], NVIDIA Isaac Sim [21], and iGibson [22] focus largely on embodied AI tasks. More generic tools for object detection dataset generation include BlenderProc [23], BlendTorch [24], NVISII [25], and the Unity Perception package [7].…”

Autonomous Exploration Development Environment and the Planning Algorithms

“…These simulators support various conditions such as lighting and weather changes, moving objects such as pedestrians, and incident scenes. For indoor navigation, iGibson [15], [16], Sapien [17], AI2Thor [18], Virtual Home [19], ThreeDWorld [20], MINOS [21], House3D [22] and CHALET [23] use synthetic scenes, while reconstructed scenes are also available in iGibson, AI Habitat [24], [25] and MINOS [21]. Compared to datasets, simulation environments have the advantage of providing access to ground truth data, e.g.…”

“…Our environment models are both indoor and outdoor containing complex topology to stress exploration and navigation algorithms. In addition to our environment models, we support the photorealistic house models from Matterport3D [3] and provide interface to AI Habitat [24], [25]. Both are widely used by the robotics and computation vision societies.…”

“…Our system also supports the photorealistic house models from Matterport3D [3]. Users are provided with scan data and RGB, depth, and semantic images rendered by AI Habitat [24], [25]. Users have the option of running AI Habitat side by side with our system or in post-processing.…”

LUMINOUS: Indoor Scene Generation for Embodied AI Challenges

“…Embodied artificial intelligence (EAI) has attracted significant attention, both in advanced deep learning models and algorithms [1,2,3,4] and the rapid development of simulated platforms [5,6,7,8,9]. Many open challenges [10,11,12,13] have been proposed to facilitate EAI research. A critical bottleneck in existing simulated platforms [10,12,8,5,14] is the limited number of indoor scenes that support vision-and-language navigation, object interaction, and complex household tasks.…”