Resolving Language and Vision Ambiguities Together: Joint
            Segmentation and Prepositional Attachment Resolution in Captioned
            Scenes

Christie, Gordon; Laddha, Ankit; Agrawal, Aishwarya; Antol, Stanislaw; Goyal, Yash; Kochersberger, Kevin; Batra, Dhruv

doi:10.18653/v1/d16-1156

“…Adding visual information provides grounding for words, potentially allowing the networks to learn references from words to objects, or at least visual features in the input (Hill et al., 2021; Vong & Lake, 2022). Multimodal learning has been shown to help resolve ambiguities when only linguistic information is present (Berzak, Barbu, Harari, Katz, & Ullman, 2015; Christie et al., 2016), induce constituent structures (Shi, Mao, Gimpel, & Livescu, 2019), and ground events described in language to video (Siddharth, Barbu, & Siskind, 2014; Yu, Siddharth, Barbu, & Siskind, 2015).…”

Section: Neural Network and Trainingmentioning

confidence: 99%

Finding Structure in One Child's Linguistic Experience

Wang

¹

,

Vong

²

,

Kim

³

et al. 2023

Cognitive Science

3

0

View full text Add to dashboard Cite

Neural network models have recently made striking progress in natural language processing, but they are typically trained on orders of magnitude more language input than children receive. What can these neural networks, which are primarily distributional learners, learn from a naturalistic subset of a single child's experience? We examine this question using a recent longitudinal dataset collected from a single child, consisting of egocentric visual data paired with text transcripts. We train both languageonly and vision-and-language neural networks and analyze the linguistic knowledge they acquire. In parallel with findings from Jeffrey Elman's seminal work, the neural networks form emergent clusters of words corresponding to syntactic (nouns, transitive and intransitive verbs) and semantic categories (e.g., animals and clothing), based solely on one child's linguistic input. The networks also acquire sensitivity to acceptability contrasts from linguistic phenomena, such as determiner-noun agreement and argument structure. We find that incorporating visual information produces an incremental gain in predicting words in context, especially for syntactic categories that are comparatively more easily grounded, such as nouns and verbs, but the underlying linguistic representations are not fundamentally altered. Our findings demonstrate which kinds of linguistic knowledge are learnable from a snapshot of a single child's real developmental experience.

show abstract

“…Visually-aided Language Learning Previous research attempt to introduce visual information to improve language learning on various Natural Language Processing (NLP) scenarios, including but not limited to machine translation (Grubinger et al, 2006;Elliott et al, 2016), information retrieval (Funaki and Nakayama, 2015;Gu et al, 2018), semantic parsing (Christie et al, 2016;Shi et al, 2019), natural language inference (Xie et al, 2019), bilingual lexicon learning (Kiela et al, 2015;Vulic et al, 2016), natural language generation evaluation (Zhu et al, 2021), spatial commonsense reasoning (Liu et al, 2022) and language representation learning (Lazaridou et al, 2015;Collell et al, 2017;Kiela et al, 2018;Bordes et al, 2019;Lu et al, 2019;Li et al, 2019;Luo et al, 2020;Li et al, 2020;Tan and Bansal, 2020;Radford et al, 2021). While most of these studies acquire visual information through retrieval from the web or large-scale image sets, a recent line of studies attempt to generate visual supervision from scratch.…”

Section: Related Workmentioning

confidence: 99%

Imagination-Augmented Natural Language Understanding

Lu¹,

Zhu²,

Wang³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural language understanding tasks from a novel learning perspective-imagination-augmented cross-modal understanding. iACE enables visual imagination with external knowledge transferred from the powerful generative and pretrained vision-and-language models. Extensive experiments on GLUE SWAG (Zellers et al., 2018) show that iACE achieves consistent improvement over visuallysupervised pre-trained models. More importantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances. 1

show abstract

“…Visually-aided Language Learning Previous research attempt to introduce visual information to improve language learning on various Natural Language Processing (NLP) scenarios, including but not limited to machine translation (Grubinger et al, 2006;Elliott et al, 2016), information retrieval (Funaki and Nakayama, 2015;Gu et al, 2018), semantic parsing (Christie et al, 2016;Shi et al, 2019), natural language inference (Xie et al, 2019), bilingual lexicon learning (Kiela et al, 2015;Vulic et al, 2016), natural language generation evaluation (Zhu et al, 2021), spatial commonsense reasoning (Liu et al, 2022) and language representation learning (Lazaridou et al, 2015;Collell et al, 2017;Kiela et al, 2018;Bordes et al, 2019;Lu et al, 2019;Li et al, 2019;Luo et al, 2020;Li et al, 2020;Tan and Bansal, 2020;Radford et al, 2021). While most of these studies acquire visual information through retrieval from the web or large-scale image sets, a recent line of studies attempt to generate visual supervision from scratch.…”

Section: Related Workmentioning

confidence: 99%

Imagination-Augmented Natural Language Understanding

Lu¹,

Zhu²,

Wang³

et al. 2022

Preprint

0

View full text Add to dashboard Cite

Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural language understanding tasks from a novel learning perspective-imagination-augmented cross-modal understanding. iACE enables visual imagination with external knowledge transferred from the powerful generative and pre-trained vision-and-language models. Extensive experiments on GLUE SWAG (Zellers et al., 2018) show that iACE achieves consistent improvement over visually-supervised pre-trained models. More importantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances. 1

show abstract

Resolving Language and Vision Ambiguities Together: Joint Segmentation and Prepositional Attachment Resolution in Captioned Scenes

Cited by 15 publications

References 28 publications

Finding Structure in One Child's Linguistic Experience

Finding Structure in One Child's Linguistic Experience

Imagination-Augmented Natural Language Understanding

Imagination-Augmented Natural Language Understanding

Contact Info

Product

Resources

About