Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

Abstract

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense perpixel semantic instance segmentations for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. Together, these features make our dataset well-suited for geometric learning problems that require direct 3D supervision, multi-task learning problems that require reasoning jointly over multiple input and output modalities, and inverse rendering problems. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, annotation effort, and computation time. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a state-of-the-art natural language processing model. All the code we used to generate our dataset is available online.

Recognizing Scenes from Novel Viewpoints

“…Our experiments validate ViewSeg's contributions on two challenging datasets, Hypersim [52] and Replica [62]. We substantially outperform alternate framings of the problem that build on the state-of-the-art: image-based NVS [70] followed by semantic segmentation [8], which tests whether NVS is sufficient for the task; and lifting semantic segmentations [8] to 3D and differentiably rendering, like [65], which tests the value of using implicit functions to tackle the problem.…”

“…We experiment on Hypersim [52] and Replica [62]. Both provide posed views of complex scenes with over 30 object types and under varying conditions of occlusion and lighting.…”

“…Hypersim [52] is a dataset of 461 complex scenes. Camera trajectories across scenes result in 77,400 images with camera poses, masks for 40 semantic classes [58], along with true depth maps.…”

“…For each Replica scene, we simulate an agent circling around the center of the 3D scene and render the observations. Note that this is unlike Hypersim [52], where camera trajectories are extracted by the authors apriori. We use the same camera intrinsics and resolution as Hypersim: the horizontal field of view is 60 • and the image resolution is 1024×768.…”

“…where c ∈ R 3 is the RGB color, σ ∈ R is the occupancy, and s ∈ R |C| is the distribution over semantic classes C. Hypersim [52] provides annotations for 40 semantic classes.…”