Reciprocal semantic predictions drive categorization of scene contexts and objects even when they are separate

Abstract

Visual categorization improves when object-context associations in scenes are semantically consistent, thus predictable from schemas stored in long-term memory. However, it is unclear whether this is due to differences in early perceptual processing, in matching of memory representations or in later stages of response selection. We tested these three concurrent explanations across five experiments. At each trial, participants had to categorize a scene context and an object briefly presented within the same image (Experiment 1), or separately in simultaneous images (Experiments 2-5). We analyzed unilateral (Experiments 1, 3) and bilateral presentations (Experiments 2, 4, 5), and presentations on the screen's horizontal midline (Experiments 1-2) and in the upper and lower visual fields (Experiments 3, 4). In all the experiments, we found a semantic consistency advantage for both context categorization and object categorization. this shows that the memory for object-context semantic associations is activated regardless of whether these two scene components are integrated in the same percept. Our study suggests that the facilitation effect of semantic consistency on categorization occurs at the stage of matching the percept with previous knowledge, supporting the object selection account and extending this framework to an object-context reciprocal influence on matching processes (objectcontext selection account). Viewers are able to categorize visual images from a brief glance, in a few dozens of milliseconds 1. This ability is a central dimension of cognition 2 : We make sense of the world fundamentally by relating and differentiating its components in terms of known groups, or categories 3,4. This allows us to process large amounts of information, while minimizing cognitive costs, and perceive common structures of elements 4 , even within the complex scenes that characterize everyday visual world. These scenes are composed of multiple objects, organized within a background (the scene's context) following semantic and spatial rules, which define the plausibility of object occurrence in the scene and of the scene's layout, respectively 5. Object-context associations are at the core of the concept of "scene" and a key, omnipresent aspect in our visual experience. However, it is still unclear how they influence visual information processing, namely in which conditions and at what stage they operate. Many studies have focused on the plausibility of object occurrence and have shown better categorization performance in the case of consistent, expected associations than inconsistent, unexpected associations. This semantic consistency advantage has been mainly found for categorization of the objects (e.g., "parasol", "notebook") included in the scene 6-10 but also for the categorization of the scene's context 7,11,12 in terms of its gist 13 (e.g., "beach", "office"). Several studies using even-related potentials (ERPs) 14-18 have reliably reported a greater N400 evoked by inconsistent objects, and interpreted it as reflecti...

The forest, the trees, or both? Hierarchy and interactions between gist and object processing during perception of real-world scenes

“…Importantly, we also found less accurate classification of backgrounds when they comprised an incongruent object. We are aware of three previous studies in which a similar effect was found (Davenport and Potter, 2004;Davenport, 2007;Leroy et al, 2020). This observation is particularly important, as it seems to be at odds with the discussed theoretical models.…”

“…This view is supported by computational work showing that influence of an incongruent object can be accounted for using scene statistics only (Mack and Palmeri, 2010). On the other hand, finding a congruence effect even when a scene and object are presented separately, in different screen locations, goes against this account and supports a semantic component (Leroy et al, 2020).…”

“…What supports this prediction is that objects semantically incongruent with the gist, namely those with a very low probability of occuring in a given context (Biederman et al, 1982), are identified slower and less accurately (Boyce et al, 1989; Davenport and Potter, 2004;Davenport, 2007;Joubert et al, 2008;Fize et al, 2011;Mack et al, 2017; Leroy et al, 2020) and their processing requires additional attentional resources (Võ and Henderson 2009;Mudrik et al, 2011). However, there is emerging evidence that representations of objects can also affect gist recognition, as presence of an incongruent object was found to impair gist recognition (Davenport and Potter, 2004;Davenport, 2007;Leroy et al 2020). This observation is in conflict with the discussed global-to-models, which assume gist-and object-recognition to unfold in a rather serial manner, with the former influencing the latter, but not vice versa.…”

Generative adversarial networks unlock new methods for cognitive science

“…than scenecongruent objects (or vice versa). Perhaps GANs could arrive at realistically looking incongruent stimuli more efficiently compared to editing the stimuli manually in Adobe Photoshop CS 9.0 [87] or staging actual incongruent scenes and photographing them [88]. SinGAN [42] can blend (foreign) objects and backgrounds (see also [89,90]).…”

To Grasp the World at a Glance: The Role of Attention in Visual and Semantic Associative Processing

“…Nevertheless, several studies have suggested that the rapid extraction of a scene's gist provides enough information to create a preliminary semantic template of the world, which can guide attention to areas of interest within the scene (e.g., [32,33]). Furthermore, grasping a scene's main theme can facilitate object recognition by pre-activating or priming associated objects embedded within the scene to a degree that is sufficient for their partial recognition (e.g., [34][35][36][37][38][39][40][41][42]; but see different views on this issue in [43][44][45]). Support for this approach comes mainly from studies examining the effects of scene recognition (e.g., a farm) on the processing of contextually consistent (e.g., a cow) or inconsistent objects (e.g., a vacuum cleaner) embedded in the scene.…”