LiftFormer: 3D Human Pose Estimation using attention models

Llopart, Adrian

doi:10.48550/arxiv.2009.00348

Cited by 3 publications

(5 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To make the algorithm invariant to the camera angle, the 3D joints were rotated to bring the vector between the hip joints into the frontal plane and to align the vector from the mid hip to the sternum vertically in the frontal plane. The 3D joint locations were tokenized as in [27]: by concatenating the together the joint locations for each frame into a perframe vector and passing them through an MLP to match the transformer embedding dimension, and using sinusoidal embedding for positional encoding. To include the subject's height, we provided a token of height embedded with an additional MLP and a learned positional embedding.…”

Section: Methods: Gait Analysis Pipeline and Trainingmentioning

confidence: 99%

Transforming Gait: Video-Based Spatiotemporal Gait Analysis

Cotton¹,

Emoonah²,

Anthony³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human pose estimation from monocular video is a rapidly advancing field that offers great promise to human movement science and rehabilitation. This potential is tempered by the smaller body of work ensuring the outputs are clinically meaningful and properly calibrated. Gait analysis, typically performed in a dedicated lab, produces precise measurements including kinematics and step timing. Using over 7000 monocular video from an instrumented gait analysis lab, we trained a neural network to map 3D joint trajectories and the height of individuals onto interpretable biomechanical outputs including gait cycle timing and sagittal plane joint kinematics and spatiotemporal trajectories. This task specific layer produces accurate estimates of the timing of foot contact and foot off events. After parsing the kinematic outputs into individual gait cycles, it also enables accurate cycle-by-cycle estimates of cadence, step time, double and single support time, walking speed and step length.

show abstract

Section: Methods: Gait Analysis Pipeline and Trainingmentioning

confidence: 99%

Transforming Gait: Video-Based Spatiotemporal Gait Analysis

Cotton¹,

Emoonah²,

Anthony³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Deep learning [22], multi-scale temporal features, spatio-temporal KCS pose differentiation, and occlusion data augmentation [29] have been used for the 2D to 3D development of human pose estimation [30,31]. Other methods use attention models [32] and multi-scale networks with phase inference optimization [33], introducing many parameters requiring manual tuning. The performance of the graphical model-based approach has been surpassed by convolutional neural networks (CNNs) [31,34].…”

Section: Related Workmentioning

confidence: 99%

Pose Detection and Recurrent Neural Networks for Monitoring Littering Violations

Husni,

Felia,

Abdurrahman

et al. 2023

Eng

View full text Add to dashboard Cite

Infrastructure development requires various considerations to maintain its continuity. Some public facilities cannot survive due to human indifference and irresponsible actions. Unfortunately, the government has to spend a lot of money, effort, and time to repair the damage. One of the destructive behaviors that can have an impact on infrastructure and environmental problems is littering. Therefore, this paper proposes a device as an alternative for catching littering rule violators. The proposed device can be used to monitor littering and provide warnings to help officers responsible for capturing the violators. In this innovation, the data obtained by the camera are sent to a mini-PC. The device will send warning information to a mobile phone when someone litters. Then, a speaker will turn on and issue a sound warning: “Do not litter”. The device uses pose detection and a recurrent neural network (RNN) to recognize a person’s activity. All activities can be monitored in a more distant place using IoT technology. In addition, this tool can also monitor environmental conditions and replace city guards to monitor the area. Thus, the municipality can save money and time.

show abstract

“…2 Related Works 3D Human Pose Estimation 3D human pose estimation is mainly categorized into top-down and bottom-up methods. Top-down methods use a cropped bounding box as input that contains a single person (Li and Chan 2014;Sun et al 2017;Pavlakos et al 2017;Sun et al 2018;Moon, Chang, and Lee 2019;Martinez et al 2017;Nie, Wei, and Zhu 2017;Gong, Zhang, and Feng 2021;Llopart 2020). Meanwhile, bottomup methods estimate all personal keypoints from the input image and then group them into each set of a person (Fabbri et al 2020;Lin and Lee 2020;Mehta et al 2020;Wang et al 2010).…”

Section: Introductionmentioning

confidence: 99%

“…The alternative is a two-stage approach with a lifting network (Martinez et al 2017;Nie, Wei, and Zhu 2017;Llopart 2020;Gong, Zhang, and Feng 2021). The two-stage method first estimates 2D keypoints' coordinates and then translates 2D coordinates into 3D coordinates using an additional lifting network.…”

Section: Introductionmentioning

confidence: 99%

“…The two types of LSTM used the results of 2D pose estimation and input image patches as input; they achieved better performance lifting 2D keypoints to 3D keypoints. (Llopart 2020) used attention-based transformer encoder blocks to predict 3D keypoints; the inputs for this method were a sequence of 2D keypoints and the network generated 3D keypoints. (Gong, Zhang, and Feng 2021) proposed an online augmentation method that could generate harder poses to estimate.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Integrated In-vehicle Monitoring System Using 3D Human Pose Estimation and Seat Belt Segmentation

Kim¹,

Kim²,

Kim³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, along with interest in autonomous vehicles, the importance of monitoring systems for both drivers and passengers inside vehicles has been increasing. This paper proposes a novel in-vehicle monitoring system the combines 3D pose estimation, seat-belt segmentation, and seat-belt status classification networks. Our system outputs various information necessary for monitoring by accurately considering the data characteristics of the in-vehicle environment. Specifically, the proposed 3D pose estimation directly estimates the absolute coordinates of keypoints for a driver and passengers, and the proposed seat-belt segmentation is implemented by applying a structure based on the feature pyramid. In addition, we propose a classification task to distinguish between normal and abnormal states of wearing a seat belt using results that combine 3D pose estimation with seat-belt segmentation. These tasks can be learned simultaneously and operate in real-time. Our method was evaluated on a private dataset we newly created and annotated. The experimental results show that our method has significantly high performance that can be applied directly to real in-vehicle monitoring systems.

show abstract

LiftFormer: 3D Human Pose Estimation using attention models

Cited by 3 publications

References 42 publications

Transforming Gait: Video-Based Spatiotemporal Gait Analysis

Transforming Gait: Video-Based Spatiotemporal Gait Analysis

Pose Detection and Recurrent Neural Networks for Monitoring Littering Violations

Integrated In-vehicle Monitoring System Using 3D Human Pose Estimation and Seat Belt Segmentation

Contact Info

Product

Resources

About