Multimodal aspects of sentence comprehension: Do facial and color cues interact with processing negated and affirmative sentences?

Abstract

Negation is usually considered as a linguistic operator reversing the truth value of a proposition. However, there are various ways to express negation in a multimodal manner. It still remains an unresolved issue whether nonverbal expressions of negation can influence linguistic negation comprehension. Based on extensive evidence demonstrating that language comprehenders are able to instantly integrate extralinguistic information such as a speaker's identity, we expected that nonverbal cues of negation and affirmation might similarly affect sentence comprehension. In three preregistered experiments, we examined how far nonverbal markers of negation and affirmation-specifically, the so-called "not face" (see Benitez-Quiroz et al., 2016) and red or green color (see Dudschig et al., 2023)-interact with comprehending negation and affirmation at the sentential level. Participants were presented with photos ("not face" vs. positive control; Experiments 1 and 2) or color patches (red vs. green; Experiment 3). They then read negated and affirmative sentences in a self-paced manner or judged the sensibility of negated and affirmative sentences (e.g., "No, I do not want to sing" vs. "Yes, I would like to buy a sofa"). Both frequentist statistics and Bayes factors resulting from linear mixed-effects analyses showed that processing times for negated and affirmative sentences were not significantly modulated by the nonverbal features under investigation. This indicates that their influence might not extend to sentential negation or affirmation comprehension.