Action and Perception as Divergence Minimization

Abstract

We introduce a unified objective for action and perception of intelligent agents. Extending representation learning and control, we minimize the joint divergence between the combined system of agent and environment and a target distribution. Intuitively, such agents use perception to align their beliefs with the world, and use actions to align the world with their beliefs. Minimizing the joint divergence to an expressive target maximizes the mutual information between the agent's representations and inputs, thus inferring representations that are informative of past inputs and exploring future inputs that are informative of the representations. This lets us explain intrinsic objectives, such as representation learning, information gain, empowerment, and skill discovery from minimal assumptions. Moreover, interpreting the target distribution as a latent variable model suggests powerful world models as a path toward highly adaptive agents that seek large niches in their environments, rendering task rewards optional. The framework provides a common

Active Inference and Epistemic Value in Graphical Models

“…The derivation of alternative functionals that preserve the desirable epistemic behavior of EFE optimization is an active research area [41,8]. There have been several interesting proposals such as the Free Energy of the Expected Future [22,7,9] or Generalized Free Energy [5], as well as amortization strategies [42,43]. However, the approach for a majority of the alternative functionals is to facilitate epistemics by the same mutual information term utilized by EFE while finessing the remainder of the functional.…”

“…The AIF literature describes multiple Free Energy (FE) objectives for policy planning, e.g., the Expected FE [4], Generalized FE [5] and Predicted (Bethe) FE [6] (among others, see e.g. [7,8,9]). Traditionally, the Expected Free Energy (EFE) is evaluated for a selection of policies, and a posterior distribution over policies is constructed from the corresponding EFEs.…”

Generalized Simultaneous Localization and Mapping (G-SLAM) as unification framework for natural and artificial intelligences: towards reverse engineering the hippocampal/entorhinal system and principles of high-level cognition

“…Such loop-closures have a further functional significance in allowing experiences to be bound together into a unified representational system where updates can be propagated in a mutually-constrained wholistic fashion, so providing a basis for the rapid and flexible construction and refinement of knowledge structures in the form of cognitive schemas that have both graph-like and map-like properties. With further experience, these schemata can then be transferred to the neocortex in the form of more stable adaptive action and thought tendencies, so forming a powerful hybrid architecture for instantiating robust causal world models (Hafner et al, 2020;Safron, 2021b).…”

On the Expressivity of Markov Reward

“…As a final note, we highlight alternative approaches to task specification. Building on the Free Energy Principle [13,12], Hafner et al [17] consider a variety of task types in terms of minimization of distance to a desired target distribution [3]. Alternatively, Littman et al [30] and Li et al [28] propose variations of linear temporal logic (LTL) as a mechanism for specifying a task to RL agents, with related literature extending LTL to the multi-task [58] and multi-agent [18] settings, or using reward machines for capturing task structure [19].…”