Deep Head Pose: Gaze-Direction Estimation in Multimodal Video

Abstract

In this paper we present a Convolutional Neural Network based model for human head pose estimation in low-resolution multi-modal RGB-D data. We pose the problem as one of classification of human gazing direction. We further fine-tune a regressor based on the learned deep classifier. Next we combine the two models (classification and regression) to estimate approximate regression confidence. We present state-ofthe-art results in datasets that span the range of high resolution Human Robot Interaction (close up faces plus depth information) data to challenging low resolution outdoor surveillance data. We build upon our robust head-pose estimation and further introduce a new visual attention model to recover interaction with the environment. Using this probabilistic model we show that many higher level scene understanding like human-human/scene interaction detection can be achieved. Our solution runs in real-time on commercial hardware.

Single image-based head pose estimation with spherical parametrization and 3D morphing

“…1. According to [24], we can classify existing computer vision based head pose estimation methods into two categories: learning based methods [1][2][3] [16] [25][26][27][28][29][30][31][32][33][34][35][36][37][38] that need large amount of training data and computational resources and geometry based methods [4][5][6][7][8][9][10] [ [39][40][41][42][43][44][45][46][47][48][49] that are fast but with a little lower accuracy, see section II for details. In this paper, as shown in Fig.…”

Interact as You Intend: Intention-Driven Human-Object Interaction Detection

“…For example, Fathi et al [35] built a probabilistic generative model to simultaneously predict the sequence of gaze locations and the respective action label from first person view videos. Mukherjee and Robertson [38] estimated the gaze direction based on the head pose in multimodal videos, and managed to recover human-human/scene interactions. In the image domain, Recasens et al [36] proposed a method to detect the object regions being fixated at by human in the scene.…”

Predicting students’ attention in the classroom from Kinect facial and body features

“…The gaze direction have been detected from combined video and depth signals [24,25] and utilized in the visual attention model to estimate human-to-human interaction. Human gaze has also been used for semantic mapping of human attention in the 3D environment [26].…”