How optimal is word recognition under multimodal uncertainty?

Fourtassi, Abdellah; Frank, Michael C.

doi:10.1016/j.cognition.2019.104092

Cited by 6 publications

(6 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The integration model provides a formal description of the process of information integration, at least at the computational level of analysis (Marr, 1982). As such, our work complements theorizing about information integration in other domains of language comprehension (e.g., Fourtassi & Frank, 2020; McClelland et al, 2006; Smith et al, 2017).…”

Section: Discussionmentioning

confidence: 72%

Predicting pragmatic cue integration in adults’ and children’s inferences about novel word meanings.

Bohn¹,

Tessler²,

Merrick³

et al. 2022

Journal of Experimental Psychology: General

Self Cite

View full text Add to dashboard Cite

Language is learned in complex social settings where listeners must reconstruct speakers' intended meanings from context. To navigate this challenge, children can use pragmatic reasoning to learn the meaning of unfamiliar words. A critical challenge for pragmatic reasoning is that it requires integrating multiple information sources, which have typically been studied separately. Here we study this integration process. First, we experimentally isolate two sources of pragmatic information: expectations about informative communication and common ground. Next, we use a probabilistic model of conversational reasoning to formalize how these information sources should be combined and how this process might develop. We use this model to generate quantitative predictions, which we test against new experimental data from 3-to 5-year-old children (N = 243) and adults (N = 694). Results show close alignment between model predictions and data. Furthermore, the model provided a better explanation of the data compared with simpler alternative models assuming that participants selectively ignore one information source. This work integrates distinct sets of findings regarding information sources for early language learning and suggests that pragmatic reasoning models can provide a quantitative framework for understanding developmental changes in language learning.

show abstract

Section: Discussionmentioning

confidence: 72%

Predicting pragmatic cue integration in adults’ and children’s inferences about novel word meanings.

Bohn¹,

Tessler²,

Merrick³

et al. 2022

Journal of Experimental Psychology: General

Self Cite

View full text Add to dashboard Cite

show abstract

“…In fact, the role of relative reliability in integrating multimodal inputs has been extensively found. For example, Fourtassi and Frank (2020) reported that the brain optimally weights the auditory and visual cues. And when one modality has noise added to it, the human brain systematically prefers the unperturbed modality.…”

Section: Discussionmentioning

confidence: 99%

“…This aids the rapid generation of largely accurate perceptual experiences (Press et al, 2020). To test this hypothesis, we refer to previous studies to reduce this relative reliability by blurring stimuli (Binur et al, 2022; Fourtassi & Frank, 2020; Rohlf et al, 2020). If the above framework holds, we expected facial expression clarity to modulate the effect of the scene on facial expression representation.…”

Section: Introductionmentioning

confidence: 99%

Neural response and representation: Facial expressions in scenes

et al. 2022

View full text Add to dashboard Cite

show abstract

“…In future work, we seek to build more comprehensive models that integrate multimodal cues -besides verbal language -that likely play a role in signaling communicative intents including vocal and visual cues. Indeed, such cues are picked up on by adults and children and are integrated to optimize language understanding and learning (e.g., Fourtassi & Frank, 2020;Fourtassi et al, 2021). This effort will involve collecting multimodal data of spontaneous child-caregiver conversations as well as the development of machine learning methods for the automatic annotation of speech acts using linguistic, acoustic, and visual features.…”

Section: Discussionmentioning

confidence: 99%

Large-scale study of speech acts' development using automatic labelling

Nikolaus¹,

Maes²,

Auguste³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Studies of children's language use in the wild (e.g., in the context of child-caregiver social interaction) have been slowed by the time- and resource- consuming task of hand annotating utterances for communicative intents/speech acts. Existing studies have typically focused on investigating rather small samples of children, raising the question of how their findings generalize both to larger and more representative populations and to a richer set of interaction contexts. Here we propose a simple automatic model for speech act labeling in early childhood based on the INCA-A coding scheme (Ninio et al., 1994). After validating the model against ground truth labels, we automatically annotated the entire English-language data from the CHILDES corpus. The major theoretical result was that earlier findings generalize quite well at a large scale. Our model will be shared with the community so that researchers can use it with their data to investigate various questions related to language use development.

show abstract

How optimal is word recognition under multimodal uncertainty?

Cited by 6 publications

References 50 publications

Predicting pragmatic cue integration in adults’ and children’s inferences about novel word meanings.

Predicting pragmatic cue integration in adults’ and children’s inferences about novel word meanings.

Neural response and representation: Facial expressions in scenes

Large-scale study of speech acts' development using automatic labelling

Contact Info

Product

Resources

About