An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

Suglia, Alessandro; Bisk, Yonatan; Konstas, Ioannis; Vergari, Antonio; Bastianelli, Emanuele; Vanzo, Andrea; Lemon, Oliver

doi:10.18653/v1/2021.eacl-main.183

Cited by 4 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If such knowledge is generic enough, the agent should be able to reuse it for solving other tasks as well. For instance, (Suglia, Bisk, Konstas, Vergari, Bastianelli, Vanzo, & Lemon, 2021) argue that visual guessing games can be developed as a generic transfer learning procedure for grounded language learning. This stresses the idea that more general architectures able to solve multiple tasks are required (Brown et al, 2020).…”

Section: What Type Of Representations Can Be Learned Via Language Games?mentioning

confidence: 99%

Visually Grounded Language Learning: A Review of Language Games, Datasets, Tasks, and Models

Suglia,

Konstas,

Lemon

2024

jair

View full text Add to dashboard Cite

In recent years, several machine learning models have been proposed. They are trained with a language modelling objective on large-scale text-only data. With such pretraining, they can achieve impressive results on many Natural Language Understanding and Generation tasks. However, many facets of meaning cannot be learned by “listening to the radio” only. In the literature, many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field. We rely on Wittgenstein’s idea of ‘language games’ to categorise such tasks into 3 different families: 1) discriminative games, 2) generative games, and 3) interactive games. Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events. Overall, these represent key requirements for developing grounded meanings in neural models.

show abstract

Section: What Type Of Representations Can Be Learned Via Language Games?mentioning

confidence: 99%

Visually Grounded Language Learning: A Review of Language Games, Datasets, Tasks, and Models

Suglia,

Konstas,

Lemon

2024

jair

View full text Add to dashboard Cite

show abstract

“…A recent research focus has been on systems that include visual as well as linguistic information in the interaction, and which in some cases learn visually-grounded word meanings [61,64,65], seeking to address the classic 'symbolgrounding' problem of AI [23]. Such setups are also known as 'multi-modal' interaction systems, as they combine the information modalities of language and vision.…”

Section: Vision and Languagementioning

confidence: 99%

“…The few exceptions, such as Cups [41] and MeetUp [25], have only small volumes of data, and almost no datasets go beyond pairs of agents. Moreover, while work on visually grounded language learning commonly uses shared real images [14,61], prior work on coordination in conversational grounding in visual tasks (such as [75]) has almost exclusively used simulated, artificial, and abstract data (e.g. using abstract shapes [39,43,77]).…”

Section: New Data Collections For Multi-agent Conversational Collabor...mentioning

confidence: 99%

Conversational AI for multi-agent communication in Natural Language

Lemon

2022

AIC

Self Cite

View full text Add to dashboard Cite

Research at the Interaction Lab focuses on human-agent communication using conversational Natural Language. The ultimate goal is to create systems where humans and AI agents (including embodied robots) can spontaneously form teams and coordinate shared tasks through the use of Natural Language conversation as a universal communication interface. This paper first introduces machine learning approaches to problems in conversational AI in general, where computational agents must coordinate with humans to solve tasks using conversational Natural Language. It also covers some of the practical systems developed in the Interaction Lab, ranging from speech interfaces on smart speakers to embodied robots interacting using visually grounded language. In several cases communication between multiple agents is addressed. The paper surveys the central research problems addressed here, the approaches developed, and our main results. Some key open research questions and directions are then discussed, leading towards a future vision of conversational, collaborative multi-agent systems.

show abstract

“…In this paper, we focus on the effect of different training sets using the same decoding strategy. In a concurrent work, Suglia et al (2021) propose a new training paradigm called Self-play via Iterated Experience Learning (SPIEL), in which the Questioner agent learns the task from games previously generated by other instances of the same Questioner architecture. The authors investigate to what extent the representations learned while playing guessing games can be transferred to other downstream tasks, such as VQA (Antol et al 2015).…”

Section: Figurementioning

confidence: 99%

Garbage In, Flowers Out: Noisy Training Data Help Generative Models at Test Time

Testoni¹,

Bernardi²

2022

ijcol

View full text Add to dashboard Cite

Despite important progress, conversational systems often generate dialogues that sound unnatural to humans. We conjecture that the reason lies in the different training and testing conditions: agents are trained in a controlled "lab" setting but tested in the "wild". During training, they learn to utter a sentence given the ground-truth dialogue history generated by human annotators. On the other hand, during testing, the agents must interact with each other, and hence deal with noisy data. We propose to fill this gap between the training and testing environments by training the model with mixed batches containing both samples of human and machine-generated dialogues. We assess the validity of the proposed method on GuessWhat?!, a visual referential game. We show that our method improves the linguistic quality of the generated dialogues, and it leads to higher accuracy of the guessing task; simple perturbations of the ground-truth dialogue history that mimic machine-generated data do not account for a similar improvement. Finally, we run a human evaluation experiment on a sample of machine-machine dialogues to complement the quantitative analysis. This experiment shows that also human annotators successfully exploit dialogues generated by a model trained with mixed batches to solve the task. Hence, the mixed-batch training does not cause a language drift. Moreover, we find that the new training regime allows human annotators to be significantly more confident when selecting the target object, showing that the generated dialogues are informative.

show abstract

An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

Cited by 4 publications

References 24 publications

Visually Grounded Language Learning: A Review of Language Games, Datasets, Tasks, and Models

Visually Grounded Language Learning: A Review of Language Games, Datasets, Tasks, and Models

Conversational AI for multi-agent communication in Natural Language

Garbage In, Flowers Out: Noisy Training Data Help Generative Models at Test Time

Contact Info

Product

Resources

About