Computational mechanisms of curiosity and goal-directed exploration

Schwartenbeck, Philipp; Passecker, Johannes; Hauser, Tobias U.; FitzGerald, Thomas H. B.; Kronbichler, Martin; Friston, Karl J.

doi:10.1101/411272

Cited by 47 publications

(70 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the term to be maximized) as: We provide a full derivation of Eq 4 in Appendix 4 and discuss its relationship to several established formalisms. Here, we have decomposed epistemic value into state epistemic value, or salience, and parameter epistemic value, or novelty [53]. State epistemic value quantifies the degree to which the expected observations o τ reduce the uncertainty in an agent's beliefs about the hidden states s τ .…”

Section: Plos Computational Biologymentioning

confidence: 99%

Learning action-oriented models through active inference

2020

View full text Add to dashboard Cite

Converging theories suggest that organisms learn and exploit probabilistic models of their environment. However, it remains unclear how such models can be learned in practice. The open-ended complexity of natural environments means that it is generally infeasible for organisms to model their environment comprehensively. Alternatively, action-oriented models attempt to encode a parsimonious representation of adaptive agent-environment interactions. One approach to learning action-oriented models is to learn online in the presence of goal-directed behaviours. This constrains an agent to behaviourally relevant trajectories, reducing the diversity of the data a model need account for. Unfortunately, this approach can cause models to prematurely converge to sub-optimal solutions, through a process we refer to as a bad-bootstrap. Here, we exploit the normative framework of active inference to show that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. We illustrate our approach using a simple agent-based model of bacterial chemotaxis. We first demonstrate that learning via goal-directed behaviour indeed constrains models to behaviorally relevant aspects of the environment, but that this approach is prone to sub-optimal convergence. We then demonstrate that epistemic behaviours facilitate the construction of accurate and comprehensive models, but that these models are not tailored to any specific behavioural niche and are therefore less efficient in their use of data. Finally, we show that active inference agents learn models that are parsimonious, tailored to action, and which avoid bad bootstraps and sub-optimal convergence. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample efficient learning algorithms.

show abstract

Section: Plos Computational Biologymentioning

confidence: 99%

Learning action-oriented models through active inference

2020

View full text Add to dashboard Cite

show abstract

“…However, because interpreting EFE as a single psychological quantity is difficult, the intuitive understanding of how the information seeking behavior arises from EFE is less straightforward. In addition, how the epistemic value (i.e., curiosity; Friston, Lin, et al, 2017;Schwartenbeck et al, 2019) plays a role in minimizing (information) surprise is not obvious. PRS provides an intuitive account of how curiosity arises: informative action makes the hidden state less ambiguous, thus reducing RS.…”

Section: Discussionmentioning

confidence: 99%

“…The predictions of PRS and EFE can also be tested using an experimental design in an experience-based manner, which are applicable to animals other than humans. This can be achieved by extending the previous examples to explain the active inference based on EFE Friston et al (2015); FitzGerald, Schwartenbeck, Moutoussis, Dolan, and Friston (2015); Friston, FitzGerald, et al (2017); Schwartenbeck et al (2019). For example, Friston et al (2015) applied the active inference framework to a variant of T-maze task, in which visiting the tip of the center arm provides the subject with information on which of the other left or right arm is likely to offer a reward.…”

Section: Discussionmentioning

confidence: 99%

“…Because our examples and problem settings are simple (though more general in some sense) compared to previous studies (Friston et al, 2015;Schwöbel et al, 2018;Schwartenbeck et al, 2019) and the example calculations can be easily performed by readers themselves (perhaps with a scientific calculator). We believe that this paper will serve as an introductory guide to active inference in the FEP framework.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Retrospective surprise: A computational component for active inference

Katahira¹,

Kunisato²,

Okimura³

et al. 2019

Preprint

View full text Add to dashboard Cite

In the free energy principle (FEP), proposed by Friston, it is supposed that agents seek to minimize the “surprise”–the negative log (marginal) likelihood of observations (i.e., sensory stimuli)–given the agents' current belief. This is achieved by minimizing the free energy, which provides an upper bound on the surprise. The FEP has been applied to action selection in a framework called “active inference,” where agents are supposed to select an action so that they minimize the “expected free energy” (EFE). While the FEP and active inference have attracted the attention of researchers in a wide range of fields such as psychology and psychiatry, as well as neuroscience, it is not clear which psychological construct EFE is related to. To facilitate the discussion and interpretation of psychological processes underlying active inference, we introduce a computational component termed the “retrospective (or residual) surprise,” which is the surprise of an observation after updating the belief given the observation itself. We show that the predicted retrospective surprise (PRS) provides a lower bound on EFE: EFE is always larger than PRS. We illustrate the properties of EFE and PRS using examples of inference for a binary hidden cause given a binary observation. Essentially, EFE and PRS show similar behavior; however, in certain situations, they provide different predictions regarding action selection. This study also provides insights into the mechanism of active inference based on EFE.

show abstract

“…The counterfactual agent is able to interpret such signals as evidence that the hidden dynamics underwriting its sensory flows may have changed in some significant way (e.g., finding oneself confronted by oncoming traffic), and can draw on alternative possible models to evaluate which parameterisation affords the best explanation for the data at hand (cf. parameter exploration; Schwartenbeck et al 2019). If the contingent relations structuring relevant environmental properties have indeed altered (e.g., realising one is visiting a country where people drive on the opposite side of the road), the agent will need to update its model in order to capture these novel conditions (see Sales et al 2019).…”

Section: Model 3: Counterfactual Active Inferencementioning

confidence: 99%

From Allostatic Agents to Counterfactual Cognisers: Active Inference, Biological Regulation, and The Origins of Cognition

Corcoran

Pezzulo

Hohwy

2019

Preprint

View full text Add to dashboard Cite

What is the function of cognition? On one influential account, cognition evolved to co-ordinate behaviour with environmental change or complexity (Godfrey-Smith 1996). Liberal interpretations of this view ascribe cognition to an extraordinarily broad set of biological systems – even bacteria, which modulate their activity in response to salient external cues, would seem to qualify as cognitive agents. However, equating cognition with adaptive flexibility per se glosses over important distinctions in the way biological organisms deal with environmental complexity. Drawing on contemporary advances in theoretical biology and computational neuroscience, we cash these distinctions out in terms of the representation and resolution of different varieties of uncertainty. This analysis leads us to propose a formal criterion for delineating cognition from other, more pervasive forms of adaptive plasticity. On this view, biological cognition is rooted in a particular kind of functional organisation; namely, one that enables the agent to detach from the present and engage in counterfactual (active) inference.

show abstract

Computational mechanisms of curiosity and goal-directed exploration

Cited by 47 publications

References 28 publications

Learning action-oriented models through active inference

Learning action-oriented models through active inference

Retrospective surprise: A computational component for active inference

From Allostatic Agents to Counterfactual Cognisers: Active Inference, Biological Regulation, and The Origins of Cognition

Contact Info

Product

Resources

About