Relevance of working memory for reinforcement learning in older adults varies with timescale of learning

Abstract

In young adults, individual differences in working memory (WM) contribute to reinforcement learning (RL). Age-related RL changes, however, are mostly attributed to decreased reward predictionerror (RPE) signaling. Here, we investigated the contribution of WM to RL in young (18-35) and older (≥65) adults. Because WM supports maintenance across a limited timescale, we only expected a relation between RL and WM with short delays between stimulus repetitions. Our results demonstrated better learning with short than long delays. A week later, however, long-delay associations were remembered better. Computational modeling corroborated that during learning, WM was more engaged by young adults in the short-delay condition than in any other age-condition combination. Crucially, both model-derived and neuropsychological assessments of WM predicted short-delay learning in older adults, who further benefitted from using self-conceived learning strategies. Thus, depending on the timescale of learning, age-related RL changes may not only reflect decreased RPE signaling but also WM decline.

Reward learning and working memory: effects of massed versus spaced training and post-learning delay period

“…Supporting this view, we previously found that after 3 weeks, associations that had been trained over multiple sessions were robust to forgetting, unlike associations trained in a single session . Similar effects were recently reported in a related study (van de Vijver and Ligneul, 2019). Notably, however, our current results indicate that spacing can be successfully studied in just a single session.…”

“…Supporting this view, two previous studies have found effects of spacing after at least one week, allowing for sleep-related consolidation processes van de Vijver and Ligneul, 2019); here, training was either spaced across days or within a single session (van de Vijver and Ligneul, 2019). In the latter study, when a test was given immediately after learning, the authors found no change in performance from the end of learning (van de Vijver and Ligneul, 2019), aligning with the choice test results in our No-Delay group. By using a brief delay, however, our paradigm demonstrates that effects of spacing on the maintenance of learning can be successfully studied in just a single session.…”

“…When making a choice between an apple and a banana, for example, an experienced decision-maker relies on values shaped by many experiences spread across years. In general, repeated experiences of stimulus-and action-reward associations are often separated by minutes, hours, days, or even longer, and recent research has shown that spaced training leads to value associations that are resistant to forgetting, similar to habits (Kim et al, 2015;van de Vijver and Ligneul, 2019). However, there is a striking difference between the proposed slow, gradual learning mechanism that supports the learning of habitual stimulus-and actionvalue associations (Yin and Knowlton, 2006) and actual experimental designs that are commonly used to study this learning mechanism in humans.…”

“…Critically, recent research has begun to illuminate how performance in dominant massed reward learning paradigms is also supported by working memory processes (Collins and Frank, 2012;van de Vijver et al, 2015;van de Vijver and Ligneul, 2019). Working memory can maintain information -such as the identity of the best stimulus or the best response to a stimulus -in the face of interference, but has a limited capacity and a limited ability to store information over longer time periods (Ma et al, 2014;D'Esposito and Postle, 2015).…”

“…While understanding interactions between massed training and working memory has been the focus of a number of recent studies (Collins et al, 2017;Collins, 2018;Collins and Frank, 2018), the inverse of massed training -spaced training -has been relatively under-explored in humans van de Vijver and Ligneul, 2019). During learning, spacing between learning events has been associated with lower ongoing performance (Schmidt and Bjork, 1992;Taylor and Rohrer, 2010;Soderstrom and Bjork, 2015;van de Vijver and Ligneul, 2019). Critically, however, for many domains including verbal memory, motor skill learning, and educational performance, spacing between learning events is well-known to lead to reduced forgetting on later tests (reported in Ebbinghaus, 1913;Lee and Genovese, 1988;Donovan and Radosevich, 1999;Janiszewski et al, 2003;Cepeda et al, 2006).…”