Imagination-Augmented Natural Language Understanding

Abstract

Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural language understanding tasks from a novel learning perspective-imagination-augmented cross-modal understanding. iACE enables visual imagination with external knowledge transferred from the powerful generative and pretrained vision-and-language models. Extensive experiments on GLUE SWAG (Zellers et al., 2018) show that iACE achieves consistent improvement over visuallysupervised pre-trained models. More importantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances. 1

Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

“…In this line of work, imagination achieves promising performance in various NLP domains (Long et al, 2021;Zhu et al, 2021;Lu et al, 2022). Previous imaginationbased work in NLP either study non-generation problems (Zhu et al, 2021;Lu et al, 2022) or utilize non-visual information (Long et al, 2021;. Our work explores the potential of generating visual imagination to improve open-ended text generation tasks.…”

“…Visually-aided NLP Recent work show the power of visual guidance in natural language processing, spanning from the language representation learning (Lu et al, 2019;Li et al, 2019;Sun et al, 2019;Luo et al, 2020;Tan and Bansal, 2020;Lu et al, 2022), the downstream tasks (Grubinger et al, 2006;Elliott et al, 2016;Xie et al, 2019;Christie et al, 2016;Shi et al, 2019;Lu et al, 2022) and evaluation (Zhu et al, 2021). They either leverage visual information from an external vision-and-language corpus or obtain such visual knowledge from the large pretrained model.…”

“…They either leverage visual information from an external vision-and-language corpus or obtain such visual knowledge from the large pretrained model. In this line of work, imagination achieves promising performance in various NLP domains (Long et al, 2021;Zhu et al, 2021;Lu et al, 2022). Previous imaginationbased work in NLP either study non-generation problems (Zhu et al, 2021;Lu et al, 2022) or utilize non-visual information (Long et al, 2021;.…”

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

“…A natural question to ask is whether image-text data can also help learn better language representations. Vokenization (Tan and Bansal, 2020) and its follow-up work iACE (Lu et al, 2022c) propose to concatenate tokens and token-related images as vokens to enrich learned language representations. In VidLanKD (Tang et al, 2021b), the authors show that it is beneficial to use video-distilled knowledge transfer to improve language understanding tasks that involve world knowledge, physical reasoning, and temporal reasoning.…”