Contrast perception as a visual heuristic in the formulation of referential expressions

Long, Madeleine; Moore, Isabelle; Mollica, Francis; Rubio-Fernández, Paula

doi:10.1016/j.cognition.2021.104879

Cited by 6 publications

(5 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Listeners are often quicker to identify a referent when they are provided with a (technically redundant) color word, despite the utterance being longer (Arts et al, 2011; Gwendolyn et al, 2021; Mangold & Pobel, 1988; Paraboni et al, 2007; Paraboni & van Deemter, 2014; Sonnenschein & Whitehurst, 1982; Rubio-Fernandez, 2021; Tourtouri et al, 2019). Consistent with the findings from language comprehension studies, speakers are more likely to use redundant color words as a function of their visual salience (Long et al, 2021; Rubio-Fernandez, 2021; Rubio-Fernandez et al, 2020).…”

Section: Validating Speaker–listener Alignmentsupporting

confidence: 81%

The social basis of referential communication: Speakers construct physical reference based on listeners’ expected visual search.

Jara‐Ettinger¹,

Rubio-Fernández²

2022

Psychological Review

Self Cite

View full text Add to dashboard Cite

A foundational assumption of human communication is that speakers should say as much as necessary, but no more. Yet, people routinely produce redundant adjectives and their propensity to do so varies crosslinguistically. Here, we propose a computational theory, whereby speakers create referential expressions designed to facilitate listeners' reference resolution, as they process words in real time. We present a computational model of our account, the Incremental Collaborative Efficiency (ICE) model, which generates referential expressions by considering listeners' real-time incremental processing and reference identification. We apply the ICE framework to physical reference, showing that listeners construct expressions designed to minimize listeners' expected visual search effort during online language processing. Our model captures a number of known effects in the literature, including cross-linguistic differences in speakers' propensity to over-specify. Moreover, the ICE model predicts graded acceptability judgments with quantitative accuracy, systematically outperforming an alternative, brevity-based model. Our findings suggest that physical reference production is best understood as driven by a collaborative goal to help the listener identify the intended referent, rather than by an egocentric effort to minimize utterance length.

show abstract

Section: Validating Speaker–listener Alignmentsupporting

confidence: 81%

The social basis of referential communication: Speakers construct physical reference based on listeners’ expected visual search.

Jara‐Ettinger¹,

Rubio-Fernández²

2022

Psychological Review

Self Cite

View full text Add to dashboard Cite

show abstract

“…While previous work in this direction manipulated artificial object arrays, either simple (e.g., Belke, 2001) or complex ones (e.g., Elsner et al, 2018), and sometimes even movie clips depicting motion events (Von Stutterheim, Andermann, Caroll, Flecken, & Schmiedtová, 2012), the current study measured eye movements using static, photo-realistic scenes. Our overall finding that increased realism in such scenes leads to substantially longer SOTs and arguably more complex scanning patterns, but not to fundamentally different search strategies, is relevant for models of reference production: it suggests that testing such models against empirical data gathered with simplified stimuli (e.g., Chen & Van Deemter, 2023;Gatt et al, 2017;Long et al, 2021) can be valid, as long as the nature of the scenes does not require a reduction of the set of distractors for contextual, semantic reasons. In general, our findings lend support for the notion that distractors shape the production of referring expressions, an idea that has been central to many computational models of reference production (since Dale & Reiter, 1995), and also has been argued to underpin models of human reference production (see, e.g., Van Gompel, van Deemter, Gatt, Snoeren, & Krahmer, 2019 for discussion).…”

Section: Discussionmentioning

confidence: 87%

“…This lack of realism may affect reference production. For example, on the object level, color atypicality reinforces the pop‐out effect: stored knowledge about typical object colors makes atypically colored objects salient in the visual scene (Becker et al., 2007; Sedivy, 2003), and atypical colors are, therefore, more often referred to than typical colors (Degen et al., 2020; Long, Moore, Mollica, & Rúbio‐Fernandez, 2021; Westerbeek, Koolen, & Maes, 2015). On the scene level, visual realism affects saliency as well, in the sense that low‐level saliency cues are less dominant in predicting fixation regions for realistic scenes such as photographs (Henderson, Brockmole, Castelhano, & Mack, 2007).…”

Section: Introductionmentioning

confidence: 99%

Realistic About Reference Production: Testing the Effects of Domain Size and Saturation

Koolen,

Krahmer

2024

Cognitive Science

View full text Add to dashboard Cite

Experiments on visually grounded, definite reference production often manipulate simple visual scenes in the form of grids filled with objects, for example, to test how speakers are affected by the number of objects that are visible. Regarding the latter, it was found that speech onset times increase along with domain size, at least when speakers refer to nonsalient target objects that do not pop out of the visual domain. This finding suggests that even in the case of many distractors, speakers perform object‐by‐object scans of the visual scene. The current study investigates whether this systematic processing strategy can be explained by the simplified nature of the scenes that were used, and if different strategies can be identified for photo‐realistic visual scenes. In doing so, we conducted a preregistered experiment that manipulated domain size and saturation; replicated the measures of speech onset times; and recorded eye movements to measure speakers’ viewing strategies more directly. Using controlled photo‐realistic scenes, we find (1) that speech onset times increase linearly as more distractors are present; (2) that larger domains elicit relatively fewer fixation switches back and forth between the target and its distractors, mainly before speech onset; and (3) that speakers fixate the target relatively less often in larger domains, mainly after speech onset. We conclude that careful object‐by‐object scans remain the dominant strategy in our photo‐realistic scenes, to a limited extent combined with low‐level saliency mechanisms. A relevant direction for future research would be to employ less controlled photo‐realistic stimuli that do allow for interpretation based on context.

show abstract

“…Recent cross-linguistic studies comparing the production and real-time comprehension of modified descriptions (e.g., “the red cup” vs. “the plastic cup”) have revealed that speakers use redundant (or noncontrastive) modification when it facilitates their listener’s visual search for a referent given their adjective position (i.e., whether it is prenominal or postnominal; Rubio-Fernandez, 2021b; Rubio-Fernandez & Jara-Ettinger, 2021; Rubio-Fernandez et al, 2021; for computational models and further empirical evidence, see Jara-Ettinger & Rubio-Fernandez, 2022). Extensive experience with referential communication may also result in the development of heuristics , as when perceptual contrast triggers the use of redundant modification (Ferreira, 2019; Long et al, 2021). Therefore, experimental studies and computational models suggest that our experience as listeners informs the choices we make as speakers in referential communication.…”

Section: Part 1: a New Theory Of The Relation Between Language And So...mentioning

confidence: 99%

Cultural evolutionary pragmatics: Investigating the codevelopment and coevolution of language and social cognition.

Rubio-Fernández¹

2024

Psychological Review

Self Cite

View full text Add to dashboard Cite

Language and social cognition come together in communication, but their relation has been intensely contested. Here, I argue that these two distinctively human abilities are connected in a positive feedback loop, whereby the development of one cognitive skill boosts the development of the other. More specifically, I hypothesize that language and social cognition codevelop in ontogeny and coevolve in diachrony through the acquisition, mature use, and cultural evolution of reference systems (e.g., demonstratives: “this” vs. “that”; articles: “a” vs. “the”; pronouns: “I” vs. “you”). I propose to study the connection between reference systems and communicative social cognition across three parallel timescales—language acquisition, language use, and language change, as a new research program for cultural evolutionary pragmatics. Within that framework, I discuss the coevolution of language and communicative social cognition as cognitive gadgets, and introduce a new methodological approach to study how universals and cross-linguistic differences in reference systems may result in different developmental pathways to human social cognition.

show abstract

Contrast perception as a visual heuristic in the formulation of referential expressions

Cited by 6 publications

References 64 publications

The social basis of referential communication: Speakers construct physical reference based on listeners’ expected visual search.

The social basis of referential communication: Speakers construct physical reference based on listeners’ expected visual search.

Realistic About Reference Production: Testing the Effects of Domain Size and Saturation

Cultural evolutionary pragmatics: Investigating the codevelopment and coevolution of language and social cognition.

Contact Info

Product

Resources

About