Is categorization in visual working memory a way to reduce mental effort? A pupillometry study

Abstract

Recent studies on visual working memory (VWM) have shown that visual information can be stored in VWM as continuous (e.g., a specific shade of red) as well as categorical representations (e.g., the general category red). It has been widely assumed, yet never directly tested, that continuous representations require more VWM mental effort than categorical representations; given limited VWM capacity, this would mean that fewer continuous, as compared to categorical, representations can be maintained simultaneously. We tested this assumption by measuring pupil size, as a proxy for mental effort, in a delayed estimation task. Participants memorized one to four ambiguous (boundaries between adjacent color categories) or prototypical colors to encourage continuous or categorical representations, respectively; after a delay, a probe indicated the location of the to-be-reported color. We found that, for set size 1, pupil size was larger while maintaining ambiguous as compared to prototypical colors, but without any difference in memory precision; this suggests that participants relied on an effortful continuous representation to maintain a single ambiguous color, thus resulting in pupil dilation while preserving precision. In contrast, for set size 2 and higher, pupil size was equally large while maintaining ambiguous and prototypical colors, but memory precision was now substantially reduced for ambiguous colors; this suggests that participants now also relied on categorical representations for ambiguous colors (which are by definition a poor fit to any category), thus reducing memory precision but not resulting in pupil dilation. Taken together, our results suggest that continuous representations are more effortful than categorical representations, and that very few continuous representations (perhaps only one) can be maintained simultaneously.Significance statementVisual working memory (VWM) can store visual information in two more-or-less distinct formats: continuous and categorical representations. It is widely assumed that VWM capacity for continuous representations is more limited than for categorical representations, yet this assumption has never been explicitly tested. Here we address this crucial question of whether continuous representations cost more resources, and as a consequence limits VWM capacity, as compared to categorical representations. To do so, we measured pupil size (as a proxy for mental effort) and memory performance in a delayed estimation task. Crucially, our results suggest that continuous representations are more effortful than categorical representations, and that only one continuous representation can be maintained in VWM at one time.

Categorical bias as a crucial parameter in visual working memory: the effect of memory load and retention interval

“…This pattern—while complex—is consistent with previous findings that continuous and categorical representations each have a limited capacity, and that this limit is different for each type of representation. Specifically, Hardman and colleagues (2017) suggested that only a single continuous representation and two categorical representations can be maintained at the same time (see also Zhou et al, 2021 for converging evidence). In the current study, it is likely that at very short retention intervals (100 and 500 ms), visual information was still stored largely in iconic memory (Sperling 1960; or perhaps in ‘fragile working memory’; Sligte et al, 2008), which has a high capacity and relies on early visual cortex (Teeuwen et al, 2021); at this point, there was little (but some) categorical bias.…”

Cognitive aging and categorical representations in visual working memory

“…Recent studies have shown that continuous representations are more fragile (Bae & Luck, 2019) and require more storage capacity as compared to categorical representations (Hardman et al, 2017;Zhou et al, 2022). This assumption is supported by the evidence that storing continuous representations appears to be more effortful than categorical representations, which limits the number of continuous representations that can be stored when memory load increases (Zhou et al, 2021). In a previous study, we asked participants to remember a varied number of colors (one to four); after a delay period of 100, 500, 1,000 or 2,000 ms, they had to report one of the memorized colors on a color circle (Zhou et al, 2022).…”

The colorful memory

“…This is possibly because when storing precise, detailed information in VWM becomes (Panichello et al, 2019), but also in primates (Freedman et al, 2001, Panichello et al, 2019 and corvid (Apostel et al, 2023). The attractor dynamics model (Almeida et al, 2015;Panichello et al, 2019) suggests that as memory load increases, internal noise of memory errors (or "random diffusion") increases, while VWM is more biased towards certain nearby locations (or "attractors"), such as color prototypes (Bae et al, 2015;Zhou et al, 2021) and cardinal orientations (Bae, 2021). These attractors draw VWM towards a limited number of stable representations that mitigate the effect of noise, but at the same time, introduces bias.…”

“…As more research has been done to understand the strategies of working memory storage, increasing evidence has suggested that VWM storage cannot be simply explained by only the slot or the resource model (e.g., Pratte, 2020;Zhou et al, 2021). Although the slot-versus-resource debate is not the focus of this thesis, we propose that there are a limited number of slots for both continuous and categorical representations, and these slots have a variable precision depending on the need of the situation.…”

“…Lately, a prominent view on VWM storage is that continuous and categorical representations correspond to items that are stored in two distinct states of VWM depending on the focus of attention (Bae & Luck, 2019;Zhou et al, 2021): active and accessory state (Cowan, 2005;Oberauer, 2002;Shipstead & Engle, 2013). When a VWM item is relevant to the ongoing task, it is actively stored inside the focus of attention; this state is sustained by activity in early visual cortex (Bae & Luck, 2019;Foster et al, 2017;Harrison & Tong, 2009;Pasternak & Greenlee, 2005); therefore, this item is stored as a detailed, precise representation; that is, a continuous representation.…”