Know What You Don’t Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories

Abstract

Zero-shot learning in Language & Vision is the task of correctly labelling (or naming) objects of novel categories. Another strand of work in L&V aims at pragmatically informative rather than "correct" object descriptions, e.g. in reference games. We combine these lines of research and model zero-shot reference games, where a speaker needs to successfully refer to a novel object in an image. Inspired by models of "rational speech acts", we extend a neural generator to become a pragmatic speaker reasoning about uncertain object categories. As a result of this reasoning, the generator produces fewer nouns and names of distractor categories as compared to a literal speaker. We show that this conversational strategy for dealing with novel objects often improves communicative success, in terms of resolution accuracy of an automatic listener.

Linguistic issues behind visual question answering

“…In the language and vision community, pragmatic aspects have been taken into account in the task of IC, where approaches building on Bayesian frameworks have been proposed to generate descriptions that contrastively refer to one but not another (similar) image (Achlioptas et al., 2019; Andreas & Klein, 2016; Cohn‐Gordon et al., 2018; Monroe et al., 2017). Similar approaches have been proposed for zero‐shot referring expression generation (Zarrieß & Schlangen, 2019).…”

Decoding Methods in Neural Language Generation: A Survey

“…Zarrieß and Schlangen [171] extend RSA-based reasoning to a zero-shot setting, where the speaker's task is to refer to target object of an "unknown" category that the literal speaker has not encountered during training. This resembles the set-up described in Anderson et al [161], where the decoding procedure extends the capabilities of the underlying language model to out-of-domain data, though Zarrieß and Schlangen [171]'s reasoning scheme does not widen the model's vocabulary but aims at leveraging the training vocabulary in efficient way for referring to unknown objects.…”

When in Doubt: Improving Classification Performance with Alternating Normalization

“…This assumption is reasonable for classification. It is also common practice when working with a finite set of intents in RSA (Monroe et al, 2017;Zarrieß and Schlangen, 2019). The column norm (Equation 6) describes a mathematical formulation aligned with how a pragmatic listener infers speaker expectations.…”

“…Previous work has applied RSA to systems that generate and understand language (Andreas and Klein, 2016;Mao et al, 2016;Vedantam et al, 2017;Cohn-Gordon et al, 2018;Zarrieß and Schlangen, 2019) in both referential games (Frank and Goodman, 2012;Goodman and Frank, 2016;Monroe et al, 2017) and sequential decisionmaking systems (Fried et al, 2018a,b). Our method departs from these applications by focusing on the ambiguity avoidance property of the listener agent as applied to generic classification tasks.…”