Value signals guide abstraction during learning

Cortese, Aurelio; Yamamoto, Asuka; Hashemzadeh, Maryam; Sepúlveda, Pradyumna; Kawato, Mitsuo; Martino, Benedetto De

doi:10.7554/elife.68943

Cited by 15 publications

(17 citation statements)

References 91 publications

(148 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The decay mechanism allows their value to decay down to zero despite not being chosen (otherwise, the model updates only the values of chosen features). Note that, this feature-based RL model, although simple, is well suited to the additive reward structure of the task, and provides a better fit than more complex RL models, such a conjunction-based RL model [ 22 ] or an Expert RL model that combines a few RL “experts” each learning different combinations of the dimensions [ 23 ].…”

Section: Resultsmentioning

confidence: 99%

Humans combine value learning and hypothesis testing strategically in multi-dimensional probabilistic reward learning

et al. 2022

View full text Add to dashboard Cite

Realistic and complex decision tasks often allow for many possible solutions. How do we find the correct one? Introspection suggests a process of trying out solutions one after the other until success. However, such methodical serial testing may be too slow, especially in environments with noisy feedback. Alternatively, the underlying learning process may involve implicit reinforcement learning that learns about many possibilities in parallel. Here we designed a multi-dimensional probabilistic active-learning task tailored to study how people learn to solve such complex problems. Participants configured three-dimensional stimuli by selecting features for each dimension and received probabilistic reward feedback. We manipulated task complexity by changing how many feature dimensions were relevant to maximizing reward, as well as whether this information was provided to the participants. To investigate how participants learn the task, we examined models of serial hypothesis testing, feature-based reinforcement learning, and combinations of the two strategies. Model comparison revealed evidence for hypothesis testing that relies on reinforcement-learning when selecting what hypothesis to test. The extent to which participants engaged in hypothesis testing depended on the instructed task complexity: people tended to serially test hypotheses when instructed that there were fewer relevant dimensions, and relied more on gradual and parallel learning of feature values when the task was more complex. This demonstrates a strategic use of task information to balance the costs and benefits of the two methods of learning.

show abstract

Section: Resultsmentioning

confidence: 99%

Humans combine value learning and hypothesis testing strategically in multi-dimensional probabilistic reward learning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…As a small step in this direction, we plan to computationally simulate a simplified CRMN as a MoEXP architecture. To this end, we will model participant learning behaviors observed in Cortese et al (2021b). In that experiment, participants were able to learn an abstract representation for reinforcement learning from a small sample.…”

Section: Discussionmentioning

confidence: 99%

From internal models toward metacognitive AI

Kawato¹,

Cortese²

2021

Biol Cybern

Self Cite

View full text Add to dashboard Cite

In several papers published in Biological Cybernetics in the 1980s and 1990s, Kawato and colleagues proposed computational models explaining how internal models are acquired in the cerebellum. These models were later supported by neurophysiological experiments using monkeys and neuroimaging experiments involving humans. These early studies influenced neuroscience from basic, sensory-motor control to higher cognitive functions. One of the most perplexing enigmas related to internal models is to understand the neural mechanisms that enable animals to learn large-dimensional problems with so few trials. Consciousness and metacognition—the ability to monitor one’s own thoughts, may be part of the solution to this enigma. Based on literature reviews of the past 20 years, here we propose a computational neuroscience model of metacognition. The model comprises a modular hierarchical reinforcement-learning architecture of parallel and layered, generative-inverse model pairs. In the prefrontal cortex, a distributed executive network called the “cognitive reality monitoring network” (CRMN) orchestrates conscious involvement of generative-inverse model pairs in perception and action. Based on mismatches between computations by generative and inverse models, as well as reward prediction errors, CRMN computes a “responsibility signal” that gates selection and learning of pairs in perception, action, and reinforcement learning. A high responsibility signal is given to the pairs that best capture the external world, that are competent in movements (small mismatch), and that are capable of reinforcement learning (small reward-prediction error). CRMN selects pairs with higher responsibility signals as objects of metacognition, and consciousness is determined by the entropy of responsibility signals across all pairs. This model could lead to new-generation AI, which exhibits metacognition, consciousness, dimension reduction, selection of modules and corresponding representations, and learning from small samples. It may also lead to the development of a new scientific paradigm that enables the causal study of consciousness by combining CRMN and decoded neurofeedback.

show abstract

“…We recently investigated how humans learn to solve decision problems based on abstractions [23]. In the task, hidden rules defined what information was relevant or irrelevant.…”

Section: Box 1 How Artificial Intelligence (Ai) Agents Learn From Mul...mentioning

confidence: 99%

“…But given that stimuli are composed of many features/dimensions, a more serious concern in value-based choice is the possibility of compressing the wrong dimensions. If the abstraction is built at the wrong level, such as overly simple features, then the process becomes slow and inefficient [23,32]. We will come back to this problem later and discuss putative mechanisms that can monitor the reliability of the current abstraction given the agent's overarching goal(s).…”

Section: Box 1 How Artificial Intelligence (Ai) Agents Learn From Mul...mentioning

confidence: 99%