Embodied Language Learning with Paired Variational Autoencoders

Özdemir, Ozan; Kerzel, Matthias; Wermter, Stefan

doi:10.1109/icdl49984.2021.9515668

Cited by 6 publications

(16 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PVAE is able to translate from actions to descriptions with 100% accuracy for all 144 patterns, including 108 training and 36 test patterns (see Table 1 ). This matches the results reported in Özdemir et al ( 2021 ).…”

Section: Embodied Spatial Relation Learningsupporting

confidence: 93%

“…A bidirectional embodied model, such as the PRAE (paired recurrent autoencoders; Yamada et al, 2018 ), is attractive to approach grounding of language, since it is able to both execute simple robot actions given language descriptions and to generate language descriptions given executed and visually perceived actions. In our recent extension of the model in a robotic scenario (Özdemir et al, 2021 ), schematically shown in Figure 2 , two cubes of different colors are placed on a table at which the NICO robot (Kerzel et al, 2017 ) is seated to interact with them (see Figure 3 ). Given proprioceptive and visual input, the approach is capable of translating robot actions to textual descriptions.…”

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

“… The NICO robot (Kerzel et al, 2017 ) in the simulation environment (Özdemir et al, 2021 ). (Left) NICO is sliding the right cube.…”

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

“…A binding loss brings the two VAEs closer by reducing the distance between two latent variables. Additionally, we introduced a channel-separated CAE (convolutional autoencoder), for the PVAE approach, for extracting visual features from the egocentric scene images (Özdemir et al, 2021 ). The channel separation refers to training the same CAE once for each RGB channel and concatenating the features extracted from the middle layer of the CAE for each color channel to arrive at the combined visual features.…”

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

“…For simplicity, the cubes are placed on two fixed positions and the two cubes on the table are never of the same color. We have trained the model with the modified descriptions using the same hyperparameters as in Özdemir et al ( 2021 ) for 15,000 iterations with a learning rate of 10 −4 and batch size of 100 4 .…”

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

See 4 more Smart Citations

Spatial relation learning in complementary scenarios with deep neural networks

et al. 2022

Self Cite

View full text Add to dashboard Cite

A cognitive agent performing in the real world needs to learn relevant concepts about its environment (e.g., objects, color, and shapes) and react accordingly. In addition to learning the concepts, it needs to learn relations between the concepts, in particular spatial relations between objects. In this paper, we propose three approaches that allow a cognitive agent to learn spatial relations. First, using an embodied model, the agent learns to reach toward an object based on simple instructions involving left-right relations. Since the level of realism and its complexity does not permit large-scale and diverse experiences in this approach, we devise as a second approach a simple visual dataset for geometric feature learning and show that recent reasoning models can learn directional relations in different frames of reference. Yet, embodied and simple simulation approaches together still do not provide sufficient experiences. To close this gap, we thirdly propose utilizing knowledge bases for disembodied spatial relation reasoning. Since the three approaches (i.e., embodied learning, learning from simple visual data, and use of knowledge bases) are complementary, we conceptualize a cognitive architecture that combines these approaches in the context of spatial relation learning.

show abstract

Section: Embodied Spatial Relation Learningsupporting

confidence: 93%

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

“… The NICO robot (Kerzel et al, 2017 ) in the simulation environment (Özdemir et al, 2021 ). (Left) NICO is sliding the right cube.…”

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

Section: Embodied Spatial Relation Learningmentioning

confidence: 99%

See 3 more Smart Citations

Spatial relation learning in complementary scenarios with deep neural networks

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

Learning Flexible Translation Between Robot Actions and Language Descriptions

Özdemir¹,

Kerzel²,

Weber³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Handling various robot action-language translation tasks flexibly is an essential requirement for natural interaction between a robot and a human. Previous approaches require change in the configuration of the model architecture per task during inference, which undermines the premise of multi-task learning. In this work, we propose the paired gated autoencoders (PGAE) for flexible translation between robot actions and language descriptions in a tabletop object manipulation scenario. We train our model in an end-to-end fashion by pairing each action with appropriate descriptions that contain a signal informing about the translation direction. During inference, our model can flexibly translate from action to language and vice versa according to the given language signal. Moreover, with the option to use a pretrained language model as the language encoder, our model has the potential to recognise unseen natural language input. Another capability of our model is that it can recognise and imitate actions of another agent by utilising robot demonstrations. The experiment results highlight the flexible bidirectional translation capabilities of our approach alongside with the ability to generalise to the actions of the opposite-sitting agent.

show abstract