Language-guided visual processing affects reasoning: The role of referential and spatial anchoring

Abstract

a b s t r a c tLanguage is more than a source of information for accessing higher-order conceptual knowledge. Indeed, language may determine how people perceive and interpret visual stimuli. Visual processing in linguistic contexts, for instance, mirrors language processing and happens incrementally, rather than through variously-oriented fixations over a particular scene. The consequences of this atypical visual processing are yet to be determined. Here, we investigated the integration of visual and linguistic input during a reasoning task. Participants listened to sentences containing conjunctions or disjunctions (Nancy examined an ant and/or a cloud) and looked at visual scenes containing two pictures that either matched or mismatched the nouns. Degree of match between nouns and pictures (referential anchoring) and between their expected and actual spatial positions (spatial anchoring) affected fixations as well as judgments. We conclude that language induces incremental processing of visual scenes, which in turn becomes susceptible to reasoning errors during the language-meaning verification process.

Gestalt Reasoning with Conjunctions and Disjunctions

“…However, experimental evidence also suggests that adults as well as children manifest an irrepressible need towards verifying the reference of all words in a sentence (for a review see [ 53 ], in press) such that, as soon as a noun refers to an object, they are more likely to inspect that particular object rather than any other semantically associated item in a visual display. We confirmed these findings for conjunction and for disjunction sentences in an eye-tracking study ([ 39 ]) where we reported that, in the condition of interest, where the two visual stimuli matched the two nouns mentioned, participants readily shifted their gaze towards the referent of the second noun in both conjunction and disjunction trials.…”

“…In contrast, model-theoretical approaches predict that reasoners should be able to report on the number of possible situations they are imagining. Recent evidence that Gestalt processing takes places at the sub-conscious level comes from an eye-movements study [ 39 ] showing that, when participants are presented with two stationary pictures (e.g., the picture of an ant next to the picture of a cloud), they shift their gaze faster from the picture representing the first word to the picture representing the second word while hearing conjunction descriptions (e.g., There is an ant and a cloud ) than when hearing disjunction descriptions (e.g., There is an ant or a cloud ). Indeed, difficulties in shifting attention between objects (here, between two Gestalts in disjunction trials) but not between object parts (here, between the two halves of a single Gestalt in conjunction trials) are well documented in the object-processing literature [ 40 – 41 ].…”

Moving stimuli guide retrieval and (in)validation of coordination simulations

“…For example, when hearing Nancy examined an ant or a cloud, participants shifted their gaze from the picture of an ant to the picture of a cloud significantly later than when hearing Nancy examined an ant and a cloud. The results by Dumitru et al (2013) align well with findings from visual processing studies that there is no cost involved in shifting attention between two object parts, but that there is a marked cost in shifting attention between two objects (Egly et al 1994;Lamy and Egeth 2002).…”

“…Treisman and Gelade (1980), for instance, reported that visual search for targets defined by one or more disjunctive features (e.g., blue or curved) occurs in parallel across a spatial display, whereas search for targets defined by a conjunction of features (e.g., red and vertical) requires a serial scan through items in the display. More recently, Dumitru et al (2013) used the visual world paradigm to highlight distinct visual sampling patterns associated with conjunction expressions as well as with disjunction expressions. The authors showed that saccades to the second of two stationary targets presented together in a visual display were launched later when the second target was mentioned in disjunction sentences than in conjunction sentences.…”

The ideomotor recycling theory for language

“…More important, language processing may reflect knowledge of the world that goes beyond people's awareness and beyond language structures (cf. Dumitru et al 2013). In particular, when understanding conjunction and disjunction expressions, people rapidly establish grounded connections between the two items mentioned (i.e., the concepts evoked by the nouns linked by conjunction or by disjunction) in the form of Gestalts.…”