Pragmatic Issue-Sensitive Image Captioning

Abstract

Image captioning systems need to produce texts that are not only true but also relevant in that they are properly aligned with the current issues. For instance, in a newspaper article about a sports event, a caption that not only identifies the player in a picture but also comments on their ethnicity could create unwanted reader reactions. To address this, we propose Issue-Sensitive Image Captioning (ISIC). In ISIC, the captioner is given a target image and an issue, which is a set of images partitioned in a way that specifies what information is relevant. For the sports article, we could construct a partition that places images into equivalence classes based on player position. To model this task, we use an extension of the Rational Speech Acts model. Our extension is built on top of state-of-the-art pretrained neural image captioners and explicitly uses image partitions to control caption generation. In both automatic and human evaluations, we show that these models generate captions that are descriptive and issue-sensitive. Finally, we show how ISIC can complement and enrich the related task of Visual Question Answering.

Reconciling truthfulness and relevance as epistemic and decision-theoretic utility.

“…Unfortunately, these systems’ strength is also their weakness: committing to a task-specific decision problem restricts their language understanding to that domain. Thus, while linguistic principles have been employed in AI research (Andreas & Klein, 2016; Dale & Reiter, 1995; Fried, Andreas, & Klein, 2018; Fried et al, 2021; Fried, Hu, et al, 2018; Golland et al, 2010; Monroe & Potts, 2015; Nie et al, 2020; Shen et al, 2019; Sumers, Ho, et al, 2021; Wang et al, 2016), these applications have not themselves been used to drive general theories of speech acts.…”

Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches

“…Other work, particularly in the computational pragmatics literature, has formulated captioning as a contrastive task (Andreas and Klein, 2016;Vedantam et al, 2017;Cohn-Gordon et al, 2018), where a target image must be described to contrast it from other similar, distractor images. This setting can be viewed as a scaled-up reference game involving complex visual inputs, and many such pragmatically-motivated variations on standard image captioning have appeared in recent years: Nie et al (2020) define issue-sensitive image captioning, in which models implicitly caption several target images at a time, while train coherence-aware captioning models which may vary in the degree of subjectivity or the extent to which inferences about target images are made.…”

Reconciling truthfulness and relevance as epistemic and decision-theoretic utility

“…Unfortunately, these systems' strength is also their weakness: committing to an explicit model of a decision context necessarily restricts them to that particular domain. Thus, while linguistic principles have been directly employed (or found to emerge) in service of task-specific performance (Andreas & Klein, 2016;Dale & Reiter, 1995;Fried et al, 2021;Fried, Hu, et al, 2018;Golland et al, 2010;Jaques et al, 2019;Monroe & Potts, 2015;Nie et al, 2020;Shen et al, 2019;Vogel et al, 2013), these applications have not themselves been used to drive more general theories of speech acts. Indeed, belief-and action-oriented approaches are in some sense disjoint in their coverage: while belief-oriented theories explicitly disregard imperative language (Roberts, 2012), most modern natural language interfaces can only handle imperatives (Luketina et al, 2019;Tellex et al, 2020).…”