Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions

Abstract

This paper describes a novel approach to linguistic mutual inference, which enables robots not only to linguistically interpret the motion patterns in the form of sentences but also to generate the motions from the sentences. The inference can be established based on two modules, the motion language model and the natural language model. The motion language model stochastically represents an association structure between symbols of motion patterns and the words in sentences assigned to the motion. This is a statistical model with a three layered structure of motion symbols, latent states and words. The natural language model statistically represents a structure of sentences based on word bigrams. The motion language model and the natural language model correspond to semantics and syntax respectively. An approach to the integration of motion language model with the natural language model allows the linguistic mutual inference for the robots. The two kinds of inference can be made by solving search problems, search for a sequence of words corresponding to a motion and search for a symbol of motion pattern corresponding to a sentence. The proposed approach to interpretation of motion patterns as sentences and generation of motion patterns from the sentences through the integration of motion language model with the natural language model is validated by an experiment on the human behavioral data.

Classification of Multi-class Daily Human Motion using Discriminative Body Parts and Sentence Descriptions

“…The experiments were conducted using both CrSub and Non-CrSub settings. As shown in this table, the average classification rate of FV-HMM/MKL-SVM (18D) was higher than Takano and Nakamura (2015) for both settings. Additionally, the average classification rates with the CrSub setting were relatively low.…”

“…Table 7 shows the comparison of the average BLEU scores for above two models. As shown in this table, the average BLEU score of FV-HMM/MKL-SVM (18D) was higher than Takano and Nakamura (2015). Figure 15 shows the sentences associated with motions in FV-HMM/MKL-SVM (18D).…”

“…However, in these approaches it was difficult to perform training tasks with a large number of motions and sentences because the motion and sentence were combined by parameters shared by two neural networks. Takano and Nakamura (2015) have previously proposed a system of robot language processing that represents human motions as multiple sentences, by integrating a motion model, a motion language model and a natural language model. Additionally, this framework created whole body motions from sentence commands for a humanoid robot, and thus may lead to human-robot interaction through natural language or remote control using a linguistic interface.…”

“…This framework classifies an input motion symbol into the corresponding category of the closest motion symbol using the same algorithm as 1-nearest neighbor (1-NN). Compared with Takano and Nakamura (2015) and Goutsu et al (2013), our approach extends the motion model as described in the next chapter.…”

“…This model converts continuous motion patterns into discrete motion symbols and can be adapted to motion classification, even when there are many motion categories, because the combination of discriminative parts from the skeleton configurations expands the capacity to represent human motions. We also develop a motion recognition system proposed in Takano and Nakamura (2015) that generates multiple sentences associated with human motions by expanding the above motion model. This system statistically represents the association relationship between motion symbols and words, and then constructs network structures that represent the arrangement of words for sentence generation.…”