Obtaining referential word meanings from visual and distributional
            information: Experiments on object naming

Zarrieß, Sina; Schlangen, David

doi:10.18653/v1/p17-1023

Cited by 11 publications

(5 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In other words, our goal is to develop the linguistic theory that could (and we think should) be used to inform a more sophisticated model of visual processing; one that draws on the coherence of embodied conversation as well as purely visual features. Our work thus complements that of Larsson (2013), Dobnik et al (2013), andZarrieß &Schlangen (2017), who investigate the conceptualization problem of nonlinguistic objects when a single clause or even a single word is uttered in a fixed early access Hunter, Asher, and Lascarides nonlinguistic context, in which the goals and interests of the agents are also clear and fixed.…”

Section: From Discourse To the Interpretation Of Nonlinguistic Eventsmentioning

confidence: 79%

A formal semantics for situated conversation

Hunter

Asher

Lascarides

2018

S&P

View full text Add to dashboard Cite

While linguists and philosophers have sought to model the various ways in which the meaning of what we say can depend on the nonlinguistic context, this work has by and large focused on how the nonlinguistic context can be exploited to ground or anchor referential or otherwise context-sensitive expressions. In this paper, we focus on examples in which nonlinguistic events contribute entire discourse units that serve as arguments to coherence relations, without the mediation of context-sensitive expressions. We use both naturally occurring and constructed examples to highlight these interactions and to argue that extant coherence-based accounts of discourse should be extended to model them. We also argue that extending coherence-based accounts in this way is a nontrivial task. It forces us to reassess basic notions of the nonlinguistic context and rhetorical relations as well as models of discourse structure, evolution, and interpretation. Our paper addresses the conceptual and technical revisions that these types of interaction demand.

show abstract

Section: From Discourse To the Interpretation Of Nonlinguistic Eventsmentioning

confidence: 79%

A formal semantics for situated conversation

Hunter

Asher

Lascarides

2018

S&P

View full text Add to dashboard Cite

show abstract

“…Previous work also explores the reverse problem of generating natural language description of objects (Vinyals et al, 2014;Karpathy and Fei-Fei, 2015;Zarriaiß and Schlangen, 2017). We hope that our dataset could also be useful for exploring the reverse task of describing actions on web pages.…”

Section: Related Work and Discussionmentioning

confidence: 99%

Mapping natural language commands to web elements

Pasupat¹,

Jiang²,

Liu³

et al. 2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "topmost article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.

show abstract

“…Still, the general notion of positive context appears to be highly relevant for visual REG. This can be seen in e.g., object naming, i.e., deciding on adequate designation terms for depicted entities (Brown, 1958;Ordonez et al, 2016;Pontillo, 2017;Zarrieß and Schlangen, 2017;Eisape et al, 2020). While naming is indispensable for neural end-to-end approaches, there is no one-to-one relationship between depicted entities and lexical items.…”

Section: Perspective Cues In Visual Contextmentioning

confidence: 99%

Rethinking symbolic and visual context in Referring Expression Generation

Schüz¹,

Gatt²,

Zarrieß³

2023

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

Situational context is crucial for linguistic reference to visible objects, since the same description can refer unambiguously to an object in one context but be ambiguous or misleading in others. This also applies to Referring Expression Generation (REG), where the production of identifying descriptions is always dependent on a given context. Research in REG has long represented visual domains through symbolic information about objects and their properties, to determine identifying sets of target features during content determination. In recent years, research in visual REG has turned to neural modeling and recasted the REG task as an inherently multimodal problem, looking at more natural settings such as generating descriptions for objects in photographs. Characterizing the precise ways in which context influences generation is challenging in both paradigms, as context is notoriously lacking precise definitions and categorization. In multimodal settings, however, these problems are further exacerbated by the increased complexity and low-level representation of perceptual inputs. The main goal of this article is to provide a systematic review of the types and functions of visual context across various approaches to REG so far and to argue for integrating and extending different perspectives on visual context that currently co-exist in research on REG. By analyzing the ways in which symbolic REG integrates context in rule-based approaches, we derive a set of categories of contextual integration, including the distinction between positive and negative semantic forces exerted by context during reference generation. Using this as a framework, we show that so far existing work in visual REG has considered only some of the ways in which visual context can facilitate end-to-end reference generation. Connecting with preceding research in related areas, as possible directions for future research, we highlight some additional ways in which contextual integration can be incorporated into REG and other multimodal generation tasks.

show abstract

Obtaining referential word meanings from visual and distributional information: Experiments on object naming

Cited by 11 publications

References 33 publications

A formal semantics for situated conversation

A formal semantics for situated conversation

Mapping natural language commands to web elements

Rethinking symbolic and visual context in Referring Expression Generation

Contact Info

Product

Resources

About