Face-from-Depth for Head Pose Estimation on Depth Images

Abstract

Depth cameras allow to set up reliable solutions for people monitoring and behavior understanding, especially when unstable or poor illumination conditions make unusable common RGB sensors. Therefore, we propose a complete framework for the estimation of the head and shoulder pose based on depth images only. A head detection and localization module is also included, in order to develop a complete end-to-end system. The core element of the framework is a Convolutional Neural Network, called POSEidon + , that receives as input three types of images and provides the 3D angles of the pose as output. Moreover, a Face-from-Depth component based on a Deterministic Conditional GAN model is able to hallucinate a face from the corresponding depth image. We empirically demonstrate that this positively impacts the system performances. We test the proposed framework on two public datasets, namely Biwi Kinect Head Pose and ICT-3DHP, and on Pandora, a new challenging dataset mainly inspired by the automotive setup. Experimental results show that our method overcomes several recent state-of-art works based on both intensity and depth input data, running in real-time at more than 30 frames per second.

Domain Translation with Conditional GANs: from Depth to RGB Face-to-Face

“…We relaxed the structure of the classical hourglass architecture performing less upsampling and downsampling operations in order to preserve the structural coherence between input and output. We found that using the half of feature maps described in [12] at each layer in both Generator and Discriminator networks sped up the training without a significant reduction of qualitative performance.…”

“…A previous work starting from the same depth images has designed an Autoencoder to create gray-level faces from depth, with the final goal of head estimation [6]. An extension of this work is performed in [12] where a GAN is trained for the same final goal. In this paper, we compare our architecture with [9], using a similar dataset, other datasets, and with some probe perceptual tasks.…”

“…Consequently, our architecture implementation follows the FfD implementation in [12]. We relaxed the structure of the classical hourglass architecture performing less upsampling and downsampling operations in order to preserve the structural coherence between input and output.…”

Mercury: A Vision-Based Framework for Driver Monitoring

“…The first input is represented by the raw depth map, while the second is a Motion Image, obtained running the Optical Flow algorithm (Farneback implementation) on a sequence of depth images. The third input is generated by a network called Face-from-Depth [6], that is able to reconstructs gray-level face images starting from the related depth images. All these three inputs are then processed by a regressive Convolutional Neural Network (CNN) [7] that finally outputs the value of the yaw, pitch and roll 3D angles expressed as continuous values.…”

Anomaly Detection for Vision-Based Railway Inspection

“…We propose a deep learning-based framework based on 2 sequential modules. The first one is an autoencoder network [6], i.e. an encoder-decoder architecture whose goal is to reconstruct the input frame, while the second one is a binary classifier network [4], predicting if the input frame contains or not an anomaly (i.e.…”