Multi‐view pose estimation with mixtures of parts and adaptive viewpoint selection

Abstract

Abstract:We propose a new method for human pose estimation which leverages information from multiple views to impose a strong prior on articulated pose. The novelty of the method concerns the types of coherence modelled. Consistency is maximised over the different views through different terms modelling classical geometric information (coherence of the resulting poses) as well as appearance information which is modelled as latent variables in the global energy function. Moreover, adequacy of each view is assessed and their contributions are adjusted accordingly. Experiments on the HumanEva and UMPM datasets show that the proposed method significantly decreases the estimation error compared to single-view results.This paper is a preprint of a paper submitted to IET Computer Vision. If accepted, the copy of record will be available at the IET Digital Library. IntroductionHuman pose estimation is a building block in many industrial applications such as human-computer interaction, motion capture systems, etc. Whereas the problem has been almost solved for easy instances, such as cooperative settings in close distance and depth data without occlusions, other realistic configurations still present a significant challenge. In particular, pose estimation from RGB input in non-cooperative settings remains a difficult problem. Methods range from unstructured and purely discriminative approaches in simple tasks on depth data, which allow real-time performance on low-cost hardware, up to complex methods imposing strong priors on pose. The latter are dominant on the more difficult RGB data but also increasingly popular on depth. These priors are often modelled as kinematic trees (as in the proposed method) or, using inverse rendering as geometric parametric models (see section 2 for related works).In this paper, we leverage the information from multiple (RGB) views to impose a strong prior on articulated pose, targetting applications such as video surveillance from multiple cameras. Activity recognition in this context is frequently preceded by articulated pose estimation, which -in a non-cooperative environment such as surveillance -can strongly depend on the optimal viewpoint. Multi-view methods can often increase robustness w.r.t. occlusions.In the proposed approach, kinematic trees model independent pose priors for each individual viewpoint, and additional terms favour consistency across views. The novelty of our contribution lies in the fact that consistency is not only forced geometrically on the solution, but also in the space of latent variables across views.More precisely, a pose is modelled as mixtures of parts, each of which is assigned to a position. As in classical kinematic trees, deformation terms model relative positions of parts w.r.t. neighbours in the tree. In the lines of [1], the deformations and the appearance terms depend on latent variables which switch between mixture components. This creates a powerful and expressive model with low-variance mixture components which are able to model precise relation...

Spatio-Temporal Action Detection in Untrimmed Videos by Using Multimodal Features and Region Proposals

“…Furthermore, Tome et al [ 22 ] proposed a method to estimate the 3D pose of a human appearing in a video; however, their method was limited to the conventional feature information that could be obtained from RGB images. To supplement this problem, methods were proposed to solve the occlusion problem by using depth-information-added RGB images [ 23 , 24 ] or by using two or more cameras [ 25 ]. The recent methods of human pose detection can be mainly classified into the top–down and bottom–up approaches.…”

Real-Time Detection and Motion Recognition of Human Moving Objects Based on Deep Learning and Multi-Scale Feature Fusion in Video

“…It is mainly divided into top-down and bottom-up detection methods [21]. The top-down detection method is to directly use the existing detector to estimate a single person's pose for each person in the image [22]. The detection time is directly proportional to the number of detections.…”

A generalizable approach for multi-view 3D human pose regression

“…For these reasons, single-and multi-view pose estimation models trained on datasets captured in such controlled laboratory environments do not generalize well to real world data, which is often visually much more complex due to occlusions, clutter and the presence of multiple persons in the scene. Secondly, current multi-view approaches [38,13,34] learn model parameters that are specific to each multi-view camera setup. In other words, to apply these approaches on a new multi-view scenario, it is required to collect new annotated data that includes both multi-view images and their corresponding 3D ground truth poses for the same camera setup.…”