End-to-end Dynamic Matching Network for Multi-view Multi-person 3D Pose Estimation

Huang, Congzhentao; Jiang, Shuai; Li, Yang; Zhang, Ziyue; Traish, Jason; Deng, Changhong; Ferguson, Sam; Xu, Richard Yi Da

doi:10.1007/978-3-030-58604-1_29

Cited by 26 publications

(31 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3. We achieve slightly better results compared to methods [1,2,10,12,54] which do not rely on 3D supervision and achieve comparable performance compared to learning-based methods [18,46] which train the model based on this dataset. Since the test frames lack pose variations compared to the training set, this dataset is considered less challenging than the Association one, which also has a more strict evaluation metric.…”

Section: Quantitative Resultsmentioning

confidence: 90%

“…One of the main challenges for multi-person pose estimation from multiple views is to associate 2D poses from different views with consistent identities. Prior matching-based work [54], is sensitive to imperfect 2D detections due to its local heuristic, and purely learning-based methods [18,46] are prone to overfitting. In contrast, we propose an effective approach to generate initial 3D pose proposals based on a confidence-aware voting technique, operating in the global 3D space of joint candidates that have been triangulated from pairs of 2D noisy detections.…”

Section: D Human Proposal Generationmentioning

confidence: 99%

“…For clarity, we would like to emphasize that our primary goal is to boost the robustness and effectiveness to new, entirely unseen humans and poses. Note that, learning-based methods [18,46] generally require direct 3D supervision via annotated pairs of multi-people, multi-view data.…”

Section: Training Data Comparisonmentioning

confidence: 99%

“…Due to the real-world importance of this problem, several recent approaches have attempted to predict the poses of multiple people, observed from multiple cameras [6,10,18,45,46,54]. Such methods can loosely be categorized into two groups.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Shape-aware Multi-Person Pose Estimation from Multi-View Images

Dong

Song

Chen

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

In this paper we contribute a simple yet effective approach for estimating 3D poses of multiple people from multi-view images. Our proposed coarse-to-fine pipeline first aggregates noisy 2D observations from multiple camera views into 3D space and then associates them into individual instances based on a confidence-aware majority voting technique. The final pose estimates are attained from a novel optimization scheme which links high-confidence multi-view 2D observations and 3D joint candidates. Moreover, a statistical parametric body model such as SMPL is leveraged as a regularizing prior for these 3D joint candidates. Specifically, both 3D poses and SMPL parameters are optimized jointly in an alternating fashion. Here the parametric models help in correcting implausible 3D pose estimates and filling in missing joint detections while updated 3D poses in turn guide obtaining better SMPL estimations. By linking 2D and 3D observations, our method is both accurate and generalizes to different data sources because it better decouples the final 3D pose from the interperson constellation and is more robust to noisy 2D detections. We systematically evaluate our method on public datasets and achieve state-of-the-art performance. The code and video will be available on the project page: https://ait.ethz.ch/projects/2021/multi-human-pose/.

show abstract

Section: Quantitative Resultsmentioning

confidence: 90%

Section: D Human Proposal Generationmentioning

confidence: 99%

Section: Training Data Comparisonmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Shape-aware Multi-Person Pose Estimation from Multi-View Images

Dong

Song

Chen

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…Multi-view multi-person 3D pose estimation aims to localize 3D skeleton joints for each person instance in a scene from multi-view camera inputs. It is a fundamental task that benefits many real-world applications (such as surveillance, sportscast, gaming and mixed reality) and is mainly tackled by reconstruction-based [6,14,4] and volumetric [40] approaches in previous literature, as shown in Fig. 1 (a) and (b).…”

Section: Introductionmentioning

confidence: 99%

Direct Multi-view Multi-person 3D Pose Estimation

Wang¹,

Zhang

Cai³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present Multi-view Pose transformer (MvP) for estimating multi-person 3D poses from multi-view images. Instead of estimating 3D joint locations from costly volumetric representation or reconstructing the per-person 3D pose from multiple detected 2D poses as in previous methods, MvP directly regresses the multi-person 3D poses in a clean and efficient way, without relying on intermediate tasks. Specifically, MvP represents skeleton joints as learnable query embeddings and let them progressively attend to and reason over the multi-view information from the input images to directly regress the actual 3D joint locations. To improve the accuracy of such a simple pipeline, MvP presents a hierarchical scheme to concisely represent query embeddings of multi-person skeleton joints and introduces an inputdependent query adaptation approach. Further, MvP designs a novel geometrically guided attention mechanism, called projective attention, to more precisely fuse the cross-view information for each joint. MvP also introduces a RayConv operation to integrate the view-dependent camera geometry into the feature representations for augmenting the projective attention. We show experimentally that our MvP model outperforms the state-of-the-art methods on several benchmarks while being much more efficient. Notably, it achieves 92.3% AP 25 on the challenging Panoptic dataset, improving upon the previous best approach [40] by 9.8%. MvP is general and also extendable to recovering human mesh represented by the SMPL model, thus useful for modeling multi-person body shapes. Code and models are available at https://github.com/sail-sg/mvp.

show abstract