Learning body part‐based pose lexicons for semantic action recognition

Abstract

Semantic action recognition aims to classify actions based on the associated semantics, which can be applied in video captioning and human‐machine interaction. In this paper the problem is addressed by jointly learning multiple pose lexicons based on multiple body parts. Specifically, multiple visual pose models are learnt, and one visual pose model is associated with one body part, which characterises the likelihood of an observed video frame being generated from hidden visual poses. Moreover, multiple pose lexicon models are simultaneously learnt along with visual pose models. One pose lexicon model is associated with one body part that establishes a probabilistic mapping between the hidden visual poses and semantic poses parsed from textual instructions. To capture the temporal relations among body parts, a transition model is also learnt to measure the probability of the alignment transitioned from one position to another position. The body part‐based pose lexicon learning provides a novel method of cross‐modality semantic correlation, which can be applied in other spatial and temporal data. Action classification is finally formulated as the problem of finding the maximum posterior probability that a given multiple sequences of visual frames follow multiple sequences of semantic poses, subject to the most likely visual pose sequences and alignment sequences. Experiments were conducted on five action datasets to validate the effectiveness of the proposed method.

The Application of Action Recognition Based on MPP-YOLOv3 Algorithm in Posture Correction

“…This indicated that the model had practical value in pose correction. Compared to the action recognition model proposed by Zhou et al [7] and the model proposed by Y Liu et al [8], in the NTU60 RGB+D and NTU120 RGB+D datasets, the average CV index improved by 1.02% and the average CS index improved by 3.5%. In summary, the action recognition model based on MPP-YOLOv3 Tiny network proposed in the study has significant application value in posture correction.…”

“…The researchers identified hidden poses in video frames and mapped them to actions in semantic instructions. The experiment outcomes indicated that their solution was effective on multiple datasets [7]. Liu et al learned and put forward a Transformer network for skeleton based human posture recognition.…”

“…Figure 10(b) indicates that more than 19 individuals are successful, primarily due to significant changes in their movements. Next, the study compares the semantic action recognition model based on Pose Lexicon proposed by reference [7], and the action recognition model based on graph transformer network proposed by reference [8]. The experimental datasets used are the NTU60 RGB+D dataset and the NTU120 RGB+D dataset, respectively, with the latter being an extension of the former.…”