Context-dependent decision-making: a simple Bayesian model

Abstract

Many phenomena in animal learning can be explained by a context-learning process whereby an animal learns about different patterns of relationship between environmental variables. Differentiating between such environmental regimes or 'contexts' allows an animal to rapidly adapt its behaviour when context changes occur. The current work views animals as making sequential inferences about current context identity in a world assumed to be relatively stable but also capable of rapid switches to previously observed or entirely new contexts. We describe a novel decision-making model in which contexts are assumed to follow a Chinese restaurant process with inertia and full Bayesian inference is approximated by a sequential-sampling scheme in which only a single hypothesis about current context is maintained. Actions are selected via Thompson sampling, allowing uncertainty in parameters to drive exploration in a straightforward manner. The model is tested on simple two-alternative choice problems with switching reinforcement schedules and the results compared with rat behavioural data from a number of T-maze studies. The model successfully replicates a number of important behavioural effects: spontaneous recovery, the effect of partial reinforcement on extinction and reversal, the overtraining reversal effect, and serial reversal-learning effects.

Incremental implicit learning of bundles of statistical patterns

“…One direction is to explore computational methods that approximate the inferences assumed by the ideal learner, but are computationally simpler and cognitively more plausible (Gershman et al, 2010; Griffiths, Vul, & Sanborn, 2012; Lloyd & Leslie, 2013; Sanborn et al, 2010). Another direction is to investigate whether well-established psychological accounts of how recent experience affects subsequent motor actions could account for the type of bundle-learning behavior observed here.…”

Context-dependent learning and causal structure

“…Bayesian structure learning has been proposed as an explanation for many behavioral and neural phenomena that are puzzling from the perspective of classical associative learning theory (Collins & Frank, 2013;Courville, Daw, & Touretzky, 2006;Gershman, Blei, & Niv, 2010;Gershman & Niv, 2012;Lloyd & Leslie, 2013;Soto, Gershman, & Niv, 2014). Of particular relevance is the model developed by Gershman et al (2010), which explained context-dependent renewal as the consequence of inferring a structure in which the acquisition and extinction trials were generated by distinct latent causes.…”

Learning bundles of stimuli renders stimulus order as a cue, not a confound

“…The HGP mixture model builds upon many previous ideas in both cognitive science and computer science (18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30). Detailed Fig.…”

“…ref. 18). Whereas other forms of prior expectations can be implemented with the HPG mixture model, a geometric prior allows us to greatly simplify the posterior inference process (more details in SI Text).…”