Inference and search on graph-structured spaces

Wu, Charley M.; Schulz, Eric; Gershman, Samuel J.

doi:10.1101/2020.03.06.981399

Cited by 7 publications

(12 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This provides a similarity metric based on transition dynamics, where an analytic method for computing the SR in closed form is to assume random transitions through the state space. This assumption of a random policy produces a nearly identical similarity metric as the RBF kernel [76], with exact equivalencies in certain cases [77].…”

Section: Related Workmentioning

confidence: 99%

“…Additionally, clustering methods (e.g., [79]) can also provide local approximations of GP inference by making predictions about novel options based on the mean of a local cluster. For instance, a related reward-learning task on graph structures [76] found that a k-nearest neighbors model provided a surprisingly effective heuristic for capturing aspects of human judgments and decisions. However, a crucial limitation of any clustering models is it would be incapable of learning and extrapolating upon any directional trends, which is a crucial feature of human function learning [59,60].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Similarities and differences in spatial and non-spatial cognitive maps

Schulz

Garvert

et al. 2020

PLoS Comput Biol

Self Cite

View full text Add to dashboard Cite

Learning and generalization in spatial domains is often thought to rely on a "cognitive map", representing relationships between spatial locations. Recent research suggests that this same neural machinery is also recruited for reasoning about more abstract, conceptual forms of knowledge. Yet, to what extent do spatial and conceptual reasoning share common computational principles, and what are the implications for behavior? Using a within-subject design we studied how participants used spatial or conceptual distances to generalize and search for correlated rewards in successive multi-armed bandit tasks. Participant behavior indicated sensitivity to both spatial and conceptual distance, and was best captured using a Bayesian model of generalization that formalized distance-dependent generalization and uncertainty-guided exploration as a Gaussian Process regression with a radial basis function kernel. The same Gaussian Process model best captured human search decisions and judgments in both domains, and could simulate realistic learning curves, where we found equivalent levels of generalization in spatial and conceptual tasks. At the same time, we also find characteristic differences between domains. Relative to the spatial domain, participants showed reduced levels of uncertainty-directed exploration and increased levels of random exploration in the conceptual domain. Participants also displayed a one-directional transfer effect, where experience in the spatial task boosted performance in the conceptual task, but not vice versa. While confidence judgments indicated that participants were sensitive to the uncertainty of their knowledge in both tasks, they did not or could not leverage their estimates of uncertainty to guide exploration in the conceptual task. These results support the notion that value-guided learning and generalization recruit cognitive-map dependent computational mechanisms in spatial and conceptual domains. Yet both behavioral and model-based analyses suggest domain specific differences in how these representations map onto actions.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Similarities and differences in spatial and non-spatial cognitive maps

Schulz

Garvert

et al. 2020

PLoS Comput Biol

Self Cite

View full text Add to dashboard Cite

show abstract

“…Previous studies have shown reliable signatures of generalization and directed exploration in adults, with relatively little random exploration (Wu et al, 2018; Wu, Schulz, Gershman, 2020). In a comparison of children aged 7–11 and adults, Schulz et al (2019) found no age‐related differences in random exploration.…”

Section: Goals and Scopementioning

confidence: 93%

Development of directed and random exploration in children

Meder

Schulz

et al. 2021

Developmental Science

Self Cite

View full text Add to dashboard Cite

Children are natural born explorers. While exploration and active learning are quintessential features of development and maturation, they also pose fundamental challenges to children and adults alike. In particular, efficiently searching for information and rewards requires balancing the dual goals of exploring unknown options to learn something new, and exploiting familiar options to obtain known rewards. At a restaurant, should you go with your usual favorite or should you try the chef's latest creation? As a child, should you play your favorite game again or try out something new? Exploring novel options can potentially reveal new and even better rewards, but could also lead to disappointment. Known as the explore-exploit dilemma, this

show abstract

“…An interesting direction for future studies is to combine surprise modulation with more abstract model building algorithms, e.g., for learning the structure of neighborhood relations of an environment in the form of a graph [73,74]. Such algorithms may explain the slight difference between the participants' adaptive behavior and SurNoR's predictions (Fig 6B).…”

Section: Surprise Modulates Learningmentioning

confidence: 99%

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Modirshanechi

Lehmann

et al. 2021

PLoS Comput Biol

View full text Add to dashboard Cite

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

show abstract

Inference and search on graph-structured spaces

Cited by 7 publications

References 84 publications

Similarities and differences in spatial and non-spatial cognitive maps

Similarities and differences in spatial and non-spatial cognitive maps

Development of directed and random exploration in children

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Contact Info

Product

Resources

About