Visually Grounded Continual Learning of Compositional Phrases

Abstract

Humans acquire language continually with much more limited access to data samples at a time, as compared to contemporary NLP systems. To study this human-like language acquisition ability, we present VisCOLL, a visually grounded language learning task, which simulates the continual acquisition of compositional phrases from streaming visual scenes. In the task, models are trained on a paired image-caption stream which has shifting object distribution; while being constantly evaluated by a visually-grounded masked language prediction task on held-out test sets. VisCOLL compounds the challenges of continual learning (i.e., learning from continuously shifting data distribution) and compositional generalization (i.e., generalizing to novel compositions). To facilitate research on VisCOLL, we construct two datasets, COCO-shift and Flickrshift, and benchmark them using different continual learning methods. Results reveal that SoTA continual learning approaches provide little to no improvements on VisCOLL, since storing examples of all possible compositions is infeasible. We conduct further ablations and analysis to guide future work 1 .

Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA

“…Is there a dataset that can prevent all shortcuts? Our automatic method for creating contrast sets allows us to ask those questions, while we believe that future work in better training mechanisms, as suggested in and Jin et al (2020), could help in making more robust models.…”

Recent Advances of Continual Learning in Computer Vision: An Overview

“…As with different characteristics of different computer vision problems, a simple adaptation of methods proposed for image classification may not lead to satisfactory performance in other computer vision problems. For example, in video grounding, Jin et al [218] pointed out that simple adaptation of ideas from image classification fails with this compositional phrases learning scenario of the language input. In visual question answering (VQA), Perez et al [219] pointed out that their model cannot preserve previously learned knowledge well after trained continuously on objects with different colors.…”

Fine-tuned Language Models are Continual Learners

“…Cao et al (2021) propose a new Continual Learning framework for NMT models, while Ke et al (2021) proposes a novel capsule network based model called B-CL (Bert based Continual Learning) for sentiment classification tasks. Jin et al (2020) show how existing Continual Learning algorithms fail at learning compositional phrases. More recently Sun et al (2019) propose a lifelong learning method LAMOL that is capable of continually learning new tasks by replaying pseudo-samples of previous tasks that require no extra memory or model capacity.…”