Learning to be a depth camera for close-range human capture and interaction

Fanello, Sean; Keskin, Cem; Izadi, Shahram; Kohli, Pushmeet; Kim, David; Sweeney, David; Criminisi, Antonio; Shotton, Jamie; Kang, Sing Bing; Paek, Tim

doi:10.1145/2601097.2601223

Cited by 62 publications

(48 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly to our work, Fanello et al [2] learn a direct mapping between infrared intensity and depth measurements, to predict per-pixel depth values. Relying on infrared intensity fall-off, this requires mounting of IR illuminants and an IR pass filter, and as such, renders the camera unusable for other purposes while also increasing power draw.…”

Section: Spatial Interaction For Wearable Computingmentioning

confidence: 90%

“…Our method is similar to those proposed in [2,13]. In [13] several randomized decision forests (RFs) are combined to enable the recognition of discrete 2D hand shapes or gestures.…”

Section: Spatial Interaction For Wearable Computingmentioning

confidence: 99%

“…After background segmentation we classify the image into three coarse layers of depth [2,13]. For our purposes we are only interested in close range interaction (i.e., arms reach).…”

Section: Coarse Depth Classificationmentioning

confidence: 99%

“…We leverage these and propose a novel machine-learning based algorithm that jointly recognizes gestures and estimates the 3D position of hands. In contrast to previous work [2], where instrumentation of the camera is required to obtain depth information, our technique uses only the built-in RGB camera, and hence is suitable for off-the-shelf, unmodified mobile devices, including compute-limited HWDs.…”

Section: Introductionmentioning

confidence: 99%

“…To achieve this, we take a data-driven machine learning approach similar to those presented in [2,13] but extend this framework to a hybrid classification-regression scheme, capable of learning a mapping from 2D images to 3D hand positions plus gestures.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction

Song

Pece

Sörös

et al. 2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

We present a machine learning technique to recognize gestures and estimate metric depth of hands for 3D interaction, relying only on monocular RGB video input. We aim to enable spatial interaction with small, body-worn devices where rich 3D input is desired but the usage of conventional depth sensors is prohibitive due to their power consumption and size. We propose a hybrid classification-regression approach to learn and predict a mapping of RGB colors to absolute, metric depth in real time. We also classify distinct hand gestures, allowing for a variety of 3D interactions. We demonstrate our technique with three mobile interaction scenarios and evaluate the method quantitatively and qualitatively.

show abstract

Section: Spatial Interaction For Wearable Computingmentioning

confidence: 90%

“…Our method is similar to those proposed in [2,13]. In [13] several randomized decision forests (RFs) are combined to enable the recognition of discrete 2D hand shapes or gestures.…”

Section: Spatial Interaction For Wearable Computingmentioning

confidence: 99%

“…After background segmentation we classify the image into three coarse layers of depth [2,13]. For our purposes we are only interested in close range interaction (i.e., arms reach).…”

Section: Coarse Depth Classificationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction

Song

Pece

Sörös

et al. 2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

A system for the real‐time tracking of operant behavior as an application of 3D camera

Kuroda

2018

J Exper Analysis Behavior

View full text Add to dashboard Cite

The capacity of 3D cameras to measure many different aspects of behavior (e.g., velocity, pattern, and posture) could contribute to the understanding of behavior. The present article describes a system for the real-time tracking of operant behavior, which is applicable to other domains of behavioral science as well. Methods for real-time 3D tracking of animal behavior are described, along with sample C++ programs. A demonstration using one zebrafish as a subject indicated that the present system successfully tracked the 3D motion of the fish. Moreover, the acquisition of a target response (i.e., approach to a corner of the aquarium) was demonstrated with the arrangement of a reinforcement contingency at the corner in the absence of a traditional, salient operandum. The system offers the capacity to characterize more completely ongoing behavior in learning tasks across a range of species than simply performance of discrete operant responses. The system also is capable of tracking multiple individuals simultaneously so it is possible both to study social interactions and arrange contingencies for engaging in social behavior. Other possible applications of 3D cameras are discussed.

show abstract

3D interactive system based on vision computing of direct‐flective cameras

Chen

Huang

2016

J Soc Info Display

View full text Add to dashboard Cite

A bare‐finger 3D interactive technology for portable devices was developed. Using directive‐flective cameras to reform the field of viewing, a blind working range close to the camera is eliminated. Moreover, the algorithm of vision computing, different from skin color detection, is presented to determine the positions of fingertips. The interactive range is workable from 1.5 to 50 cm above the entire surface of the display. The mean position error of less than 1 cm is achieved. This accuracy realizes a camera‐based 3D interactive system allowing for near‐distance functionality. Therefore, floating 3D images can be touched and interacted with, potentially creating more application and intuitive user‐machine interface.

show abstract

Learning to be a depth camera for close-range human capture and interaction

Cited by 62 publications

References 39 publications

Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction

Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction

A system for the real‐time tracking of operant behavior as an application of 3D camera

3D interactive system based on vision computing of direct‐flective cameras

Contact Info

Product

Resources

About