Word2vec to behavior: morphology facilitates the grounding of language in machines

Matthews, David; Kriegman, Sam; Cappelle, Collin; Bongard, Josh

doi:10.1109/iros40897.2019.8967639

Cited by 7 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Source code is available in the GitHub repository ( https://github.com/robodiff/robodiff ) ( 51 ). All other data are included in the manuscript and/or supporting information .…”

Section: Data Materials and Software Availabilitymentioning

confidence: 99%

Efficient automatic design of robots

Matthews,

Spielberg,

Rus

et al. 2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Robots are notoriously difficult to design because of complex interdependencies between their physical structure, sensory and motor layouts, and behavior. Despite this, almost every detail of every robot built to date has been manually determined by a human designer after several months or years of iterative ideation, prototyping, and testing. Inspired by evolutionary design in nature, the automated design of robots using evolutionary algorithms has been attempted for two decades, but it too remains inefficient: days of supercomputing are required to design robots in simulation that, when manufactured, exhibit desired behavior. Here we show de novo optimization of a robot’s structure to exhibit a desired behavior, within seconds on a single consumer-grade computer, and the manufactured robot’s retention of that behavior. Unlike other gradient-based robot design methods, this algorithm does not presuppose any particular anatomical form; starting instead from a randomly-generated apodous body plan, it consistently discovers legged locomotion, the most efficient known form of terrestrial movement. If combined with automated fabrication and scaled up to more challenging tasks, this advance promises near-instantaneous design, manufacture, and deployment of unique and useful machines for medical, environmental, vehicular, and space-based tasks.

show abstract

“…Source code is available in the GitHub repository ( https://github.com/robodiff/robodiff ) ( 51 ). All other data are included in the manuscript and/or supporting information .…”

Section: Data Materials and Software Availabilitymentioning

confidence: 99%

Efficient automatic design of robots

Matthews,

Spielberg,

Rus

et al. 2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Some studies generated actions from descriptions represented by pre-trained word embeddings [2][3] [6][11] [16]. Zhong et al [11], Matthews et al [3], and Lunch et al [2], in particular, generated actions from commands including unseen words. Zhong et al [11] used averaged pre-trained word embeddings for this task; however, this approach fails to generate actions from descriptions whose meaning depends on the order of the words.…”

Section: A Action Generation Using Pre-trained Word Embeddingsmentioning

confidence: 99%

“…Zhong et al [11] used averaged pre-trained word embeddings for this task; however, this approach fails to generate actions from descriptions whose meaning depends on the order of the words. Although Matthews et al [3] generated actions from single-word commands, the correspondence between a command and the corresponding action is fixed.…”

Section: A Action Generation Using Pre-trained Word Embeddingsmentioning

confidence: 99%

See 1 more Smart Citation

Embodying Pre-Trained Word Embeddings Through Robot Actions

Toyoda

Suzuki

Mori

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

We propose a promising neural network model with which to acquire a grounded representation of robot actions and the linguistic descriptions thereof. Properly responding to various linguistic expressions, including polysemous words, is an important ability for robots that interact with people via linguistic dialogue. Previous studies have shown that robots can use words that are not included in the action-description paired datasets by using pre-trained word embeddings. However, the word embeddings trained under the distributional hypothesis are not grounded, as they are derived purely from a text corpus. In this paper, we transform the pre-trained word embeddings to embodied ones by using the robot's sensory-motor experiences. We extend a bidirectional translation model for actions and descriptions by incorporating non-linear layers that retrofit the word embeddings. By training the retrofit layer and the bidirectional translation model alternately, our proposed model is able to transform the pre-trained word embeddings to adapt to a paired action-description dataset. Our results demonstrate that the embeddings of synonyms form a semantic cluster by reflecting the experiences (actions and environments) of a robot. These embeddings allow the robot to properly generate actions from unseen words that are not paired with actions in a dataset.

show abstract

“…Several joint learning methods for language and motion have been proposed to address these issues. Using contextindependent word embeddings derived from a large corpus, [2] and [3] dealt with unseen words that were not included in the training data. In other studies, stepwise learning methods were proposed to integrate multiple models from different domains [4] [5].…”

Section: Introductionmentioning

confidence: 99%

Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data

Toyoda,

Suzuki,

Hayashi

et al. 2022

Preprint

View full text Add to dashboard Cite

This study achieved bidirectional translation between descriptions and actions using small paired data. The ability to mutually generate descriptions and actions is essential for robots to collaborate with humans in their daily lives. The robot is required to associate real-world objects with linguistic expressions, and large-scale paired data are required for machine learning approaches. However, a paired dataset is expensive to construct and difficult to collect. This study proposes a two-stage training method for bidirectional translation. In the proposed method, we train recurrent autoencoders (RAEs) for descriptions and actions with a large amount of non-paired data. Then, we fine-tune the entire model to bind their intermediate representations using small paired data. Because the data used for pre-training do not require pairing, behavior-only data or a large language corpus can be used. We experimentally evaluated our method using a paired dataset consisting of motion-captured actions and descriptions. The results showed that our method performed well, even when the amount of paired data to train was small. The visualization of the intermediate representations of each RAE showed that similar actions were encoded in a clustered position and the corresponding feature vectors well aligned.

show abstract

Word2vec to behavior: morphology facilitates the grounding of language in machines

Cited by 7 publications

References 27 publications

Efficient automatic design of robots

Efficient automatic design of robots

Embodying Pre-Trained Word Embeddings Through Robot Actions

Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data

Contact Info

Product

Resources

About