Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving

Zheng, Jian; Shi, Xinwei; Gorban, Alexander N.; Mao, Junhua; Yang, Song; Qi, Charles R.; Liu, Ting; Chari, Visesh; Cornman, Andre; Zhou, Yin; Li, Congcong; Anguelov, Dragomir

doi:10.1109/cvprw56347.2022.00494

Cited by 28 publications

(17 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The workflow of E 3 Pose is shown in Figure 9(a). A key innovation of our proposed E 3 Pose system is to predict future 3D poses and then project the prediction results onto each individual camera view to obtain the 2D bounding boxes to enable the IoU calculation. This is because the IoU must be calculated before 2D poses in the scene of interest are estimated.…”

Section: Discussion On the Choice Of Methodsmentioning

confidence: 99%

“…The studio is designed to simulate and capture social activities of multiple people. We use the same set of training and testing sequences captured by the same set of five HD cameras (3,6,12,13,23) as in [10,37] for evaluation.…”

Section: Evaluation 51 Experiments Setupmentioning

confidence: 99%

“…Multi-human 3D pose estimation is an important yet challenging computer vision problem that has a wide range of applications such as action recognition [1], sports analysis [2] and human-computer interaction [3]. It is also perceived as a key technology to enable the seamless and immersive interaction between the real world and the virtual world in the so-called "Metaverse" [4].…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we present the design, implementation and evaluation of an energy-efficient edge-assisted multi-camera system for multi-human 3D pose estimation, dubbed E 3 Pose. At the core of E 3 Pose is a camera selection scheme that adaptively selects a subset of cameras to perform the 3D pose estimation task depending on the cameras' energy states and their view qualities (i.e., occlusions). Obviously, performing the 3D pose estimation using only a subset of cameras allows some cameras to enter a power-saving mode, thereby extending the battery lifetime of the cameras and the overall system.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

E$^3$Pose: Energy-Efficient Edge-assisted Multi-camera System for Multi-human 3D Pose Estimation

Zhang¹,

Xu²

2023

Preprint

View full text Add to dashboard Cite

Multi-human 3D pose estimation plays a key role in establishing a seamless connection between the real world and the virtual world. Recent efforts adopted a two-stage framework that first builds 2D pose estimations in multiple camera views from different perspectives and then synthesizes them into 3D poses. However, the focus has largely been on developing new computer vision algorithms on the offline video datasets without much consideration on the energy constraints in real-world systems with flexibly-deployed and battery-powered cameras. In this paper, we propose an energyefficient edge-assisted multiple-camera system, dubbed E 3 Pose, for real-time multi-human 3D pose estimation, based on the key idea of adaptive camera selection. Instead of always employing all available cameras to perform 2D pose estimations as in the existing works, E 3 Pose selects only a subset of cameras depending on their camera view qualities in terms of occlusion and energy states in an adaptive manner, thereby reducing the energy consumption (which translates to extended battery lifetime) and improving the estimation accuracy. To achieve this goal, E 3 Pose incorporates an attention-based LSTM to predict the occlusion information of each camera view and guide camera selection before cameras are selected to process the images of a scene, and runs a camera selection algorithm based on the Lyapunov optimization framework to make long-term adaptive selection decisions. We build a prototype of E 3 Pose on a 5-camera testbed, demonstrate its feasibility and evaluate its performance. Our results show that a significant energy saving (up to 31.21%) can be achieved while maintaining a high 3D pose estimation accuracy comparable to state-of-the-art methods.

show abstract

Section: Discussion On the Choice Of Methodsmentioning

confidence: 99%

Section: Evaluation 51 Experiments Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

E$^3$Pose: Energy-Efficient Edge-assisted Multi-camera System for Multi-human 3D Pose Estimation

Zhang¹,

Xu²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Ref. [ 30 ] proposes a multi-modal approach that uses 2D labels on RGB images as weak supervision to perform 3D HPE. And the multi-modal architecture also incorporates the camera and LiDAR with an auxiliary segmentation branch.…”

Section: Related Workmentioning

confidence: 99%

PA-Tran: Learning to Estimate 3D Hand Pose with Partial Annotation

Bidulka

McKeown

et al. 2023

Sensors

View full text Add to dashboard Cite

This paper tackles a novel and challenging problem—3D hand pose estimation (HPE) from a single RGB image using partial annotation. Most HPE methods ignore the fact that the keypoints could be partially visible (e.g., under occlusions). In contrast, we propose a deep-learning framework, PA-Tran, that jointly estimates the keypoints status and 3D hand pose from a single RGB image with two dependent branches. The regression branch consists of a Transformer encoder which is trained to predict a set of target keypoints, given an input set of status, position, and visual features embedding from a convolutional neural network (CNN); the classification branch adopts a CNN for estimating the keypoints status. One key idea of PA-Tran is a selective mask training (SMT) objective that uses a binary encoding scheme to represent the status of the keypoints as observed or unobserved during training. In addition, by explicitly encoding the label status (observed/unobserved), the proposed PA-Tran can efficiently handle the condition when only partial annotation is available. Investigating the annotation percentage ranging from 50–100%, we show that training with partial annotation is more efficient (e.g., achieving the best 6.0 PA-MPJPE when using about 85% annotations). Moreover, we provide two new datasets. APDM-Hand, is for synthetic hands with APDM sensor accessories, which is designed for a specific hand task. PD-APDM-Hand, is a real hand dataset collected from Parkinson’s Disease (PD) patients with partial annotation. The proposed PA-Tran can achieve higher estimation accuracy when evaluated on both proposed datasets and a more general hand dataset.

show abstract

Hybrid attention adaptive sampling network for human pose estimation in videos

Song,

Zhang,

Liu

et al. 2024

Computer Animation & Virtual

View full text Add to dashboard Cite

Human pose estimation in videos often uses sampling strategies like sparse uniform sampling and keyframe selection. Sparse uniform sampling can miss spatial‐temporal relationships, while keyframe selection using CNNs struggles to fully capture these relationships and is costly. Neither strategy ensures the reliability of pose data from single‐frame estimators. To address these issues, this article proposes an efficient and effective hybrid attention adaptive sampling network. This network includes a dynamic attention module and a pose quality attention module, which comprehensively consider the dynamic information and the quality of pose data. Additionally, the network improves efficiency through compact uniform sampling and parallel mechanism of multi‐head self‐attention. Our network is compatible with various video‐based pose estimation frameworks and demonstrates greater robustness in high degree of occlusion, motion blur, and illumination changes, achieving state‐of‐the‐art performance on Sub‐JHMDB dataset.

show abstract

Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving

Cited by 28 publications

References 36 publications

E$^3$Pose: Energy-Efficient Edge-assisted Multi-camera System for Multi-human 3D Pose Estimation

E$^3$Pose: Energy-Efficient Edge-assisted Multi-camera System for Multi-human 3D Pose Estimation

PA-Tran: Learning to Estimate 3D Hand Pose with Partial Annotation

Hybrid attention adaptive sampling network for human pose estimation in videos

Contact Info

Product

Resources

About