Active Inference for Robotic Manipulation

Abstract

Robotic manipulation stands as a largely unsolved problem despite significant advances in robotics and machine learning in the last decades. One of the central challenges of manipulation is partial observability, as the agent usually does not know all physical properties of the environment and the objects it is manipulating in advance. A recently emerging theory that deals with partial observability in an explicit manner is Active Inference. It does so by driving the agent to act in a way that is not only goal-directed but also informative about the environment. In this work, we apply Active Inference to a hard-to-explore simulated robotic manipulation tasks, in which the agent has to balance a ball into a target zone. Since the reward of this task is sparse, in order to explore this environment, the agent has to learn to balance the ball without any extrinsic feedback, purely driven by its own curiosity. We show that the information-seeking behavior induced by Active Inference allows the agent to explore these challenging, sparse environments systematically. Finally, we conclude that using an information-seeking objective is beneficial in sparse environments and allows the agent to solve tasks in which methods that do not exhibit directed exploration fail.

Reward Maximization Through Discrete Active Inference

“…Briefly, building active inference agents entails (1) equipping the agent with a (generative) model of the environment, (2) fitting the model to observations through approximate Bayesian inference by minimizing variational free energy (i.e., optimizing an evidence lower bound Beal, 2003;Bishop, 2006;Blei et al, 2017;Jordan et al, 1998) and (3) selecting actions that minimize expected free energy, a quantity that that can be decomposed into risk (i.e., the divergence between predicted and preferred paths) and ambiguity, leading to context-specific combinations of exploratory and exploitative behavior (Millidge, 2021;Schwartenbeck et al, 2019). This framework has been used to simulate and explain intelligent behavior in neuroscience (Adams et al, 2013;Parr, 2019;Parr et al, 2021;Sajid et al, 2022), psychology and psychiatry Smith, Kirlic, Stewart, Touthang, Kuplicki, Khalsa, et al, 2021;Smith, Kirlic, Stewart, Touthang, Kuplicki, McDermott, et al, 2021;Smith, Kuplicki, Feinstein, et al, 2020;Smith, Kuplicki, Teed, et al, 2020;Smith, Mayeli, et al, 2021;Smith, Schwartenbeck, Stewart, et al, 2020;Smith, Taylor, et al, 2022), machine learning (Çatal et al, 2020;Fountas et al, 2020;Mazzaglia et al, 2021;Millidge, 2020;Tschantz et al, 2019;Tschantz, Millidge, et al, 2020), and robotics (Çatal et al, 2021;Lanillos et al, 2020;Oliver et al, 2021;Pezzato et al, 2020;Pio-Lopez et al, 2016;Sancaktar et al, 2020;Schneider et al, 2022).…”