The interplay of language and visual perception in working memory

Abstract

How do perception and language interact to form the representations that guide our thoughts and actions over the short-term? Here, we provide a first examination of this question by investigating the role of verbal labels in a continuous visual working memory (WM) task. Across four experiments, participants retained in memory the continuous color of a set of dots which were presented sequentially (Experiments 1-3) or simultaneously (Experiment 4). At test, they reproduced the colors of all dots using a color wheel. During stimulus presentation participants were required to either label the colors (color labeling) or to repeat "bababa" aloud (articulatory suppression), hence prompting or preventing verbal labeling, respectively. We tested four competing hypotheses of the labeling effect: (1) labeling generates a verbal representation that overshadows the visual representation; (2) labeling yields a verbal representation in addition to the visual one; (3) the labels function as a retrieval cue, adding distinctiveness to items in memory; and (4) labels activate visual categorical representations in long-term memory. Collectively, our experiments show that labeling does not overshadow the visual input; it augments it. Mixture modeling showed that labeling increased the quantity and quality of information in WM. Our findings are consistent with the hypothesis that labeling activates visual long-term categorical representations which help in reducing the noise in the internal representations of the visual stimuli in WM.

Aging and feature-binding in visual working memory: The role of verbal rehearsal.

“…Second, visual memory for difficult-to-label information may be limited because it affords less opportunity to activate long-term memory representations. Souza and Skóra (2017) found that labeling colors (overtly or sub-vocally) improved younger-adult memory performance by activating long-term memory representations, rather than simply by adding verbal memory traces. If older adults' memory for easy-to-label colors was also comparatively 'boosted' because such colors automatically activated lexical, semantic representations that helped compensate visual memory, this would explain why this special benefit for easy-to-label colors in older adults was not completely abolished under suppression.…”

“…Such translation of visual representations into verbal code has been found to improve visual memory performance in younger adults (Brown, Forbes, & McConnell, 2006;Souza & Skóra, 2017). Despite this, tasks are often assumed to measure either visual or verbal WM, perhaps incorrectly (e.g., Logie, 2018).…”

Visual Working Memory Cannot Trade Quantity for Quality

“…Thus, we do not argue that visual WM cannot be measured with long memory array intervals. For example, a recent study ( Souza and Skóra, 2017 ) have demonstrated that when given enough time (by using serial presentation) verbal encoding of colors can increase both the quality and quantity of the stored information.…”

Does rehearsal matter? Left anterior temporal alpha and theta band changes correlate with the beneficial effects of rehearsal on working memory

“…The benefit of a recoding a visually presented stimulus depends on whether semantic associations are automatically accessed without labeling 5,33 . In Experiment 1 automatic semantic associations occurred; thus, the addition of rehearsing with a generated label offered no more benefit than accessing those stored associations 12,14,26 .…”

“…Studies support that blocking rehearsal with articulatory suppression, repeating a word such as "the" over and over again, decreases performance compared with when someone rehearses 2,3 . Visual stimuli may also benefit from rehearsal when the stimulus is recoded into a verbal representation [4][5][6] . However, the benefit of labeling and rehearsal may depend on the working memory task [7][8][9] .…”