Of “what” and “where” in a natural search task: Active object handling supports object location memory beyond the object’s identity

Abstract

Looking for as well as actively manipulating objects that are relevant to ongoing behavioral goals are intricate parts of natural behavior. It is, however, not clear to what degree these two forms of interaction with our visual environment differ with regard to their memory representations. In a real-world paradigm, we investigated if physically engaging with objects as part of a search task influences identity and position memory differently for task-relevant versus irrelevant objects. Participants equipped with a mobile eye tracker either searched for cued objects without object interaction (Find condition) or actively collected the objects they found (Handle condition). In the following free-recall task, identity memory was assessed, demonstrating superior memory for relevant compared to irrelevant objects, but no difference between the Handle and Find conditions. Subsequently, location memory was inferred via times to first fixation in a final object search task. Active object manipulation and task-relevance interacted in that location memory for relevant objects was superior to irrelevant ones only in the Handle condition. Including previous object recall performance as a covariate in the linear mixed-model analysis of times to first fixation allowed us to explore the interaction between remembered/ forgotten object identities and the execution of location memory. Identity memory performance predicted location memory in the Find but not the Handle condition, suggesting that active object handling leads to strong spatial representations independent of object identity memory. We argue that object handling facilitates the prioritization of relevant location information, but this might come at the cost of deprioritizing irrelevant information.

Scene grammar shapes the way we interact with objects, strengthens memories, and speeds search

“…Further, with every subsequent search within the same environment reaction times decreased, showing that participants profited from repeatedly searching the same environment during this active task 44 , 45 , compared to searches in 2D 46 – 48 . However, the influence of scene grammar was mediated by the different building blocks of a scene: small/movable (local) and large/generally stationary (global) objects, which we have come to call “anchors”.…”

To search or to like: Mapping fixations to differentiate two forms of incidental scene memory

“…Interestingly, we found that fixation map similarity during search failed to predict whether a scene would be correctly recognized later (Figure 5d). Based on previous research suggesting that viewing tasks may affect the extraction (Võ & Wolfe, 2012) and/or the retention (Maxcey-Richard & Hollingworth, 2013) of visual information and that top-down goals may have prioritized task-relevant information over task-irrelevant information (Draschkow & Võ, 2016;, we speculate that the visual search task prioritized search-related operations during search, which could increase memory for those specific objects, but doing so reduces the extraction and retention of overall scene visual information that is irrelevant to search but critical for incidental encoding of the scene in totality. When the visual search task is completed (i.e., the search object is found), however, the priority of encoding-related operations is normalized and incidental scene encoding resumes during free-viewing (i.e., the viewing time remaining after the search object has been found).…”

“…These results suggest that viewing tasks may affect the extraction (Võ & Wolfe, 2012) and/or the retention (Maxcey-Richard & Hollingworth, 2013) of visual information within fixations. In doing so, top-down goals may have prioritized task-relevant information over task-irrelevant information (Draschkow & Võ, 2016;. In addition, viewing tasks (e.g., visual search) may have contributed to better integration of bottom-up visual information and contextual scene semantics, leading to stronger memory representations than intentional encoding (Draschkow et al, 2014;Josephs et al 2016;Võ & Wolfe, 2015).…”

“…By establishing such template fixation maps for scene encoding, we were able to quantify the deployment of visual attention on a trialby-trial basis, allowing us to implement trial-level generalized linear mixed-effect models (GLMMs) that can isolate effects of viewing tasks, fixation counts, and visual attention, respectively. This procedure mimics analyses from recent eye-tracking studies, which used linear mixed-effect model analyses to study task effects on fixation durations (Einhäuser & Nuthmann, 2016;Nuthmann, 2017) and object memory (Draschkow & Võ, 2016;Josephs et al, 2016;. We also examined the relationship between scene memory and scene aesthetic preference by collecting scene preference ratings from a group of participants that were recruited online, called G3, because scene aesthetics are known to be significantly associated with scene memorability (Isola, Xiao, Parikh, Torralba, & Oliva, 2014).…”

Memory shapes visual search strategies in large-scale environments

“…A large body of work in visual search has revealed many of the factors that influence the search efficiency, such as the stimulus features of the image, top down guidance, and scene semantics 1 . However, most of this work has been done using 2D displays on computer monitors, and only a relatively small number of studies have examined visual search in the natural world 2 – 4 . This is an important issue because the nature of the stimulus in standard experimental paradigms differs considerably from that in everyday experience 3 , 5 – 7 .…”