Collaborative Models for Referring Expression Generation in Situated Dialogue

Fang, Rui; Doering, Malcolm; Chai, Joyce

doi:10.1609/aaai.v28i1.8934

Cited by 13 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The learned weights can be applied to referential grounding and/or word model learning algorithms to improve the referential grounding performance. They can also be utilized by referring expression generation algorithms (e.g., Fang et al 2013;Fang, Doering, and Chai 2014) to facilitate referential communication between robots and humans. Our current evaluation is based on several simplifications, including the simulation of perceptual errors and the strong assumption that the correct grounding information can be provided to the robot through dialogue with a human.…”

Section: Discussionmentioning

confidence: 99%

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

Liu

Chai

2015

AAAI

Self Cite

View full text Add to dashboard Cite

In human-robot dialogue, although a robot and its human partner are co-present in a shared environment, they have significantly mismatched perceptual capabilities (e.g., recognizing objects in the surroundings). When a shared perceptual basis is missing, it becomes difficult for the robot to identify referents in the physical world that are referred to by the human (i.e., a problem of referential grounding). To overcome this problem, we have developed an optimization based approach that allows the robot to detect and adapt to perceptual differences. Through online interaction with the human, the robot can learn a set of weights indicating how reliably/unreliably each dimension (e.g., object type, object color, etc.) of its perception of the environment maps to the human's linguistic descriptors and thus adjust its word models accordingly. Our empirical evaluation has shown that this weight-learning approach can successfully adjust the weights to reflect the robot's perceptual limitations. The learned weights, together with updated word models, can lead to a significant improvement for referential grounding in future dialogues.

show abstract

Section: Discussionmentioning

confidence: 99%

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

Liu

Chai

2015

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Going beyond one-shot reference, Zarrieß and Schlangen 2016 generate incrementally produced installments to gradually guide the addressee to the intended referent. In Fang et al ( 2014 , 2015 ), both installments and deictic gestures are used to account for perceptual mismatches between humans and artificial agents in situated dialog. Mental states and perceptual capabilities of interlocutors play an important role in natural communication, but are rarely considered in REG (but see e.g., Horacek 2005 for an exception).…”

Section: The Reg Taskmentioning

confidence: 99%

“…For linguistic interaction in shared visual environments, additional problems arise, such as perceptual mismatches between interlocutors (cf. Fang et al 2013Fang et al , 2014Fang et al , 2015 for related work in situated dialog with artificial agents).…”

Section: Visual Regmentioning

confidence: 99%

Rethinking symbolic and visual context in Referring Expression Generation

Schüz¹,

Gatt²,

Zarrieß³

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

Situational context is crucial for linguistic reference to visible objects, since the same description can refer unambiguously to an object in one context but be ambiguous or misleading in others. This also applies to Referring Expression Generation (REG), where the production of identifying descriptions is always dependent on a given context. Research in REG has long represented visual domains through symbolic information about objects and their properties, to determine identifying sets of target features during content determination. In recent years, research in visual REG has turned to neural modeling and recasted the REG task as an inherently multimodal problem, looking at more natural settings such as generating descriptions for objects in photographs. Characterizing the precise ways in which context influences generation is challenging in both paradigms, as context is notoriously lacking precise definitions and categorization. In multimodal settings, however, these problems are further exacerbated by the increased complexity and low-level representation of perceptual inputs. The main goal of this article is to provide a systematic review of the types and functions of visual context across various approaches to REG so far and to argue for integrating and extending different perspectives on visual context that currently co-exist in research on REG. By analyzing the ways in which symbolic REG integrates context in rule-based approaches, we derive a set of categories of contextual integration, including the distinction between positive and negative semantic forces exerted by context during reference generation. Using this as a framework, we show that so far existing work in visual REG has considered only some of the ways in which visual context can facilitate end-to-end reference generation. Connecting with preceding research in related areas, as possible directions for future research, we highlight some additional ways in which contextual integration can be incorporated into REG and other multimodal generation tasks.

show abstract

“…These results have shown that collaborative models are more effective in mitigating perceptual differences between humans and robots in referential communication. More details about the approach and empirical results are described by Fang, Doering, and Chai (2014).…”

Section: Grounded Language Generationmentioning

confidence: 99%

“…To understand this new challenge, we have revisited the problem of REG in the context of mismatched perceptual basis (Fang et al 2013). We extended a well‐known graph‐based approach (Krahmer, Van Erk, and Verleg 2003) that has shown effective in previous works and by incorporating uncertainties in perception into cost functions.…”

Section: Grounding Language To Perceptionmentioning

confidence: 99%

Collaborative Language Grounding Toward Situated Human‐Robot Dialogue

et al. 2016

Self Cite

View full text Add to dashboard Cite

To enable situated human-robot dialogue, techniques to support grounded language communication are essential. One particular challenge is to ground human language to robot internal representation of the physical world. Although copresent in a shared environment, humans and robots have mismatched capabilities in reasoning, perception, and action. Their representations of the shared environment and joint tasks are significantly misaligned. Humans and robots will need to make extra effort to bridge the gap and strive for a common ground of the shared world. Only then, is the robot able to engage in language communication and joint tasks. Thus computational models for language grounding will need to take collaboration into consideration. A robot not only needs to incorporate collaborative effort from human partners to better connect human language to its own representation, but also needs to make extra collaborative effort to communicate its representation in language that humans can understand. To address these issues, the Language and Interaction Research group (LAIR) at Michigan State University has investigated multiple aspects of collaborative language grounding. This article gives a brief introduction to this research effort and discusses several collaborative approaches to grounding language to perception and action.

show abstract

Collaborative Models for Referring Expression Generation in Situated Dialogue

Cited by 13 publications

References 13 publications

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

Rethinking symbolic and visual context in Referring Expression Generation

Collaborative Language Grounding Toward Situated Human‐Robot Dialogue

Contact Info

Product

Resources

About