An empirical evaluation of active inference in multi-armed bandits

Marković, Dimitrije; Stojić, Hrvoje; Schwöbel, Sarah; Kiebel, Stefan J.

doi:10.1016/j.neunet.2021.08.018

Cited by 28 publications

(31 citation statements)

References 71 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If the initial belief π (0) is a Dirichlet distribution, then π (t) is also a Dirichlet distribution at any time t for a wide range of learning rules(Faraji et al, 2018;Liakoni et al, 2021;Maheu et al, 2019;Markovic et al, 2021;Meyniel et al, 2016;Modirshanechi et al, 2019;Ryali et al, 2018;Yu & Cohen, 2009) -since the Dirichlet distribution is the conjugate prior of the categorical distribution(Efron & Hastie, 2016).3 We note that the qualitative behavior of confidence in Fig.5and Fig.6is the same for both definitions C[π(t) ] and CatConf(t).…”

mentioning

confidence: 88%

“…Using this notation, we can write the joint distribution over the observation and the parameter P (t) θ t+1 , y t+1 |x t+1 as P Θ t+1 θ t+1 , y t+1 |x t+1 ; π (t) and the updated belief π (t+1) (θ) as P Θ t+1 θ|y t+1 , x t+1 ; π (t) . The variational loss or free energy can then be defined as (Friston, 2010;Friston et al, 2017;Liakoni et al, 2021;Markovic et al, 2021;Sajid et al, 2021)…”

Section: Belief-mismatch Surprise 2: Postdictive Surprisementioning

confidence: 99%

“…Formally, in variational inference, the belief π (t+1) (θ) is approximated by π(t+1) (θ) = q(θ; φ (t+1) ), where t+1) is the minimizer of the variational loss or free energy, i.e., φ (t+1) = arg min φ F (t+1) (φ) (Friston, 2010;Friston et al, 2017;Liakoni et al, 2021;MacKay, 2003;Markovic et al, 2021;Sajid et al, 2021).…”

Section: Belief-mismatch Surprise 2: Postdictive Surprisementioning

confidence: 99%

“…where the right side of the inequality is independent of φ, and P Θ t+1 .|y t+1 , x t+1 ; π(t) is the exact Bayesian update of the belief (according to the generative model in Definition 1) given the latest approximation of the belief π(t) (Liakoni et al, 2021;Markovic et al, 2021).…”

Section: Belief-mismatch Surprise 2: Postdictive Surprisementioning

confidence: 99%

See 3 more Smart Citations

Surprise: a unified theory and experimental predictions

Modirshanechi

Brea

Gerstner

2021

Preprint

View full text Add to dashboard Cite

Surprising events trigger measurable brain activity and influence human behavior by affecting learning, memory, and decision-making. Currently there is, however, no consensus on the definition of surprise. Here we identify 16 mathematical definitions of surprise in a unifying framework, show how these definitions relate to each other, and prove under what conditions they are indistinguishable. We classify these surprise measures into four main categories: (i) change-point detection surprise, (ii) information gain surprise, (iii) prediction surprise, and (iv) confidence-correction surprise. We design experimental paradigms where different categories make different predictions: we show that surprise-modulation of the speed of learning leads to sensible adaptive behavior only for change-point detection surprise whereas surprise-seeking leads to sensible exploration strategies only for information gain surprise. However, since neither change-point detection surprise nor information gain surprise perfectly reflect the definition of ‘surprise’ in natural language, a combination of prediction surprise and confidence-correction surprise is needed to capture intuitive aspects of surprise perception. We formalize this combination in a new definition of surprise with testable experimental predictions. We conclude that there cannot be a single surprise measure with all functions and properties previously attributed to surprise. Consequently, we postulate that multiple neural mechanisms exist to detect and signal different aspects of surprise.Author noteAM is grateful to Vasiliki Liakoni, Martin Barry, and Valentin Schmutz for many useful discussions in the course of the last few years, and to Andrew Barto for insightful discussions through and after EPFL Neuro Symposium 2021 on “Surprise, Curiosity and Reward: from Neuroscience to AI”. We thank K. Robbins and collaborators for their publicly available experimental data (Robbins et al., 2018). All code needed to reproduce the results reported here will be made publicly available after publication acceptance. This research was supported by Swiss National Science Foundation (no. 200020_184615). Correspondence concerning this article should be addressed to Alireza Modirshanechi, School of Computer and Communication Sciences and School of Life Sciences, EPFL, Lausanne, Switzerland. E-mail: alireza.modirshanechi@epfl.ch.

show abstract

mentioning

confidence: 88%

Section: Belief-mismatch Surprise 2: Postdictive Surprisementioning

confidence: 99%

Section: Belief-mismatch Surprise 2: Postdictive Surprisementioning

confidence: 99%

Section: Belief-mismatch Surprise 2: Postdictive Surprisementioning

confidence: 99%

See 2 more Smart Citations

Surprise: a unified theory and experimental predictions

Modirshanechi

Brea

Gerstner

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Although AIF has yet to be scaled-to tackle high dimensional problems-to the same extent as established approaches, such as deep reinforcement learning [17,18], numerical analyses generally show that active inference performs at least as well in simple environments [9,[19][20][21][22][23], and better in environments featuring volatility, ambiguity and context sensitivity [21,22]. In this paper, we consider how AIF's features could help address key technical challenges in robotics and discuss practical robotic applications.…”

Section: Active Inferencementioning

confidence: 99%

How Active Inference Could Help Revolutionise Robotics

Costa

Lanillos

Sajid

et al. 2022

Entropy

View full text Add to dashboard Cite

Recent advances in neuroscience have characterised brain function using mathematical formalisms and first principles that may be usefully applied elsewhere. In this paper, we explain how active inference—a well-known description of sentient behaviour from neuroscience—can be exploited in robotics. In short, active inference leverages the processes thought to underwrite human behaviour to build effective autonomous systems. These systems show state-of-the-art performance in several robotics settings; we highlight these and explain how this framework may be used to advance robotics.

show abstract

Searching for and evaluating outsourced chief investment officers

Park

Ryu

2023

Manage Decis Econ

View full text Add to dashboard Cite

This study analyzes the impact of the outsourced chief investment officer (OCIO) index and OCIO search consultant on asset owners' selection of OCIOs. We adopt an agent‐based model with a reinforcement learning method by defining the OCIO search problem as a multi‐armed bandit problem. Our model highlights the significance of the information regarding potential managers provided by the OCIO index and an OCIO search consultant. Our simulation results indicate that the more managers in the market, the harder it is for asset owners to identify the optimal OCIO. If asset owners place greater emphasis on their preferences when evaluating OCIOs, those challenges might be eased. Our results suggest that OCIO search consultants and the OCIO index can support asset owners' decision‐making.

show abstract

An empirical evaluation of active inference in multi-armed bandits

Cited by 28 publications

References 71 publications

Surprise: a unified theory and experimental predictions

Surprise: a unified theory and experimental predictions

How Active Inference Could Help Revolutionise Robotics

Searching for and evaluating outsourced chief investment officers

Contact Info

Product

Resources

About