From 3D scene geometry to human workspace

Gupta, Abhinav; Satkin, Scott; Efros, Alexei A.; Hebert, Martial

doi:10.1109/cvpr.2011.5995448

Cited by 214 publications

(180 citation statements)

References 17 publications

Supporting

Mentioning

176

Contrasting

Order By: Relevance

“…Prior works have explored learning based approaches for estimating the depth of the scene [33,36,40,43,65,79,81,94,98] and estimating depth ordering [41,45] from a single image for 3D scene understanding. Recent work has shown that we can also use the objects (clutter) in the scene to aid better depth estimation of the scene [34,37] using affordances.…”

Section: Static Camera Scenariomentioning

confidence: 99%

Putting the User in the Loop for Image-Based Modeling

et al. 2014

View full text Add to dashboard Cite

Image-based modeling, the task of recovering the 3D structure of an object or a scene using 2D images is one of the primary goals of computer vision. However, 3D reconstruction in practice is a hard task due to the numerous scene irregularities. When a scene is captured from multiple viewpoints, irregularities such as textureless surfaces, specularities, thin structures, etc., make the reconstruction inaccurate. In addition, image sequences or videos of a scene captured in the wild often consist of dynamic or moving objects such as people, which makes the task of image-based modeling extremely hard. In this thesis, our goal is to tackle these problems to obtain a more accurate reconstruction of the scene. In particular, using the intuition that humans easily discern the 3D structure behind the photons, this thesis addresses these problems by putting the user in the loop with the image-based-modeling algorithm.In the first part of the thesis, we focus on image-based modeling of static scenes. We explore putting the user in the loop interacting with the algorithm by providing constraints via simple interactions on the image data, to help overcome the ill-effects of the scene irregularities. We introduce two algorithms within this framework. First, an algorithm where the user drives the process of image-based modeling. Second, a novel active-learning algorithm, which initiates the process of image-based modeling via an unsupervised algorithm and then guides the user to provide constraints when and where needed.In the second part of the thesis, we focus on image-based modeling of dynamic scenes captured by either a dynamic camera or a static camera. We develop algorithms that leverage the dynamic objects in the scene to aid the imagebased modeling algorithm. We propose novel algorithms that use the sparse, yet strong occlusion cues between the moving object and the scene, along with a realistic motion prior for the moving object to recover the depth of the scene.We explore putting the user in the loop within the framework, guided by the algorithm to provide useful depth-order constraints and show that we further improve the solution of the unsupervised algorithm.Finally, we present an end user application: iModel, which puts the user in the loop for object of interest 3D modeling on a mobile device and 3D printing of an object of interest. THESIS COMMITTEE

show abstract

Section: Static Camera Scenariomentioning

confidence: 99%

Putting the User in the Loop for Image-Based Modeling

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Scene Understanding: Geometry, Humans and Objects. Physical aspects of affordances have been recently explored in [10,12,5,16]. For example, interactions between a sitting human pose and the environment are used to identify sittable regions [10], semantic and geometric affordances of large objects such as furniture are extracted by observing people [5,12], and spatial affordances of objects with respect to possible human poses are used for placing and labeling objects [17,16].…”

Section: Related Workmentioning

confidence: 99%

“…recently, physical aspects have been explored in [10,12,16], where they model the functionality-based spatial interactions of humans with their environments. For example, Grabner et al [10] uses the interactions between a sitting human pose and the environment to identify sittable regions, Delaitre et al [5] observed people to extract semantic and geometric affordances for large furniture-like objects, and Jiang, Koppula and Saxena [16] uses the spatial affordances of objects with respect to possible human poses for the task of labeling objects.…”

Section: Introductionmentioning

confidence: 99%

Physically Grounded Spatio-temporal Object Affordances

Koppula

Saxena

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Objects in human environments support various functionalities which govern how people interact with their environments in order to perform tasks. In this work, we discuss how to represent and learn a functional understanding of an environment in terms of object affordances. Such an understanding is useful for many applications such as activity detection and assistive robotics. Starting with a semantic notion of affordances, we present a generative model that takes a given environment and human intention into account, and grounds the affordances in the form of spatial locations on the object and temporal trajectories in the 3D environment. The probabilistic model also allows uncertainties and variations in the grounded affordances. We apply our approach on RGB-D videos from Cornell Activity Dataset, where we first show that we can successfully ground the affordances, and we then show that learning such affordances improves performance in the labeling tasks.

show abstract

“…The ability to infer the geometry of a scene has enabled a variety of applications in both the vision and graphics fields. For example, Gupta et al [8] use coarse geometry estimates to predict what locations in an environment afford various actions. Karsch et al [16] use scene geometry to realistically render additional objects into a scene.…”

Section: Introductionmentioning

confidence: 99%

“…However, because this approach aims to match the exact configuration of objects in an image, with an identical furniture configuration from a library of 3D models, the algorithm does not have the flexibility required to precisely reconstruct the diversity of object configurations found in natural scenes. Additionally, many scene understanding approaches such as [7,8,28] make limiting assumptions regarding the robustness of existing monocular autocalibration algorithms. When this preliminary stage of their pipelines fail, the algorithms are unable to recover and produce a reasonable result.…”

Section: Introductionmentioning

confidence: 99%