Neural Variational Learning for Grounded Language Acquisition

Pillai, Nisha; Matuszek, Cynthia; Ferraro, Francis

doi:10.1109/ro-man50785.2021.9515374

Cited by 3 publications

(3 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, we are not the first to make these assumptions. Visual question answering models have already been used to explore neural networks' capacity to learn meaningful representations of referential words, such as nouns and predicates when trained on language tasks grounded in the visual world (Jiang et al, 2023;Mao, Gan, Kohli, Tenenbaum, & Wu, 2019;Pillai, Matuszek, & Ferraro, 2021;Wang, Mao, Gershman, & Wu, 2021;Zellers et al, 2021). As for function words, Hill, Hermann, Blunsom, and Clark (2018) briefly consider how visually grounded models learn negation, and Kuhnle and Copestake (2019) studied how these models interpret the quantifier most.…”

Section: The Current Studymentioning

confidence: 99%

Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model

Portelance,

Frank,

Jurafsky

2024

Cognitive Science

View full text Add to dashboard Cite

Interpreting a seemingly simple function word like “or,” “behind,” or “more” can require logical, numerical, and relational reasoning. How are such words learned by children? Prior acquisition theories have often relied on positing a foundation of innate knowledge. Yet recent neural‐network‐based visual question answering models apparently can learn to use function words as part of answering questions about complex visual scenes. In this paper, we study what these models learn about function words, in the hope of better understanding how the meanings of these words can be learned by both models and children. We show that recurrent models trained on visually grounded language learn gradient semantics for function words requiring spatial and numerical reasoning. Furthermore, we find that these models can learn the meanings of logical connectives and and or without any prior knowledge of logical reasoning as well as early evidence that they are sensitive to alternative expressions when interpreting language. Finally, we show that word learning difficulty is dependent on the frequency of models' input. Our findings offer proof‐of‐concept evidence that it is possible to learn the nuanced interpretations of function words in a visually grounded context by using non‐symbolic general statistical learning algorithms, without any prior knowledge of linguistic meaning.

show abstract

Section: The Current Studymentioning

confidence: 99%

Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model

Portelance,

Frank,

Jurafsky

2024

Cognitive Science

View full text Add to dashboard Cite

show abstract

“…In (Matuszek, 2018), machine-learned classifiers ground words and phrases provided by a human in an agent's perception of the world. Language can also be grounded more directly in perception, by machine learning the relevant perceptual categories from data, rather than pre-specifying them in a formal semantics (Pillai et al, 2019). In (Liu et al, 2016), an agent learns cloth folding through rich verbal communication, based on AND-OR graphs.…”

Section: Related Workmentioning

confidence: 99%

Dialogue Policies for Learning Board Games through Multimodal Communication

Zare,

Ayub,

Liu

et al. 2020

Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

This paper presents MDP policy learning for agents to learn strategic behavior-how to play board games-during multimodal dialogues. Policies are trained offline in simulation, with dialogues carried out in a formal language. The agent has a temporary belief state for the dialogue, and a persistent knowledge store represented as an extensive-form game tree. How well the agent learns a new game from a dialogue with a simulated partner is evaluated by how well it plays the game, given its dialoguefinal knowledge state. During policy training, we control for the simulated dialogue partner's level of informativeness in responding to questions. The agent learns best when its trained policy matches the current dialogue partner's informativeness. We also present a novel data collection for training natural language modules. Human subjects who engaged in dialogues with a baseline system rated the system's language skills as above average. Further, results confirm that human dialogue partners also vary in their informativeness.

show abstract

“…In (Matuszek, 2018), machine-learned classifiers ground words and phrases provided by a human in an agent's perception of the world. Language can also be grounded more directly in perception, by machine learning the relevant perceptual categories from data, rather than pre-specifying them in a formal semantics (Pillai et al, 2019). In , an agent learns cloth folding through rich verbal communication, based on AND-OR graphs.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue

2020

View full text Add to dashboard Cite

show abstract

Neural Variational Learning for Grounded Language Acquisition

Cited by 3 publications

References 42 publications

Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model

Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model

Dialogue Policies for Learning Board Games through Multimodal Communication

Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue

Contact Info

Product

Resources

About