Learning and exploration in action-perception loops

Abstract

Discovering the structure underlying observed data is a recurring problem in machine learning with important applications in neuroscience. It is also a primary function of the brain. When data can be actively collected in the context of a closed action-perception loop, behavior becomes a critical determinant of learning efficiency. Psychologists studying exploration and curiosity in humans and animals have long argued that learning itself is a primary motivator of behavior. However, the theoretical basis of learning-driven behavior is not well understood. Previous computational studies of behavior have largely focused on the control problem of maximizing acquisition of rewards and have treated learning the structure of data as a secondary objective. Here, we study exploration in the absence of external reward feedback. Instead, we take the quality of an agent's learned internal model to be the primary objective. In a simple probabilistic framework, we derive a Bayesian estimate for the amount of information about the environment an agent can expect to receive by taking an action, a measure we term the predicted information gain (PIG). We develop exploration strategies that approximately maximize PIG. One strategy based on value-iteration consistently learns faster than previously developed reward-free exploration strategies across a diverse range of environments. Psychologists believe the evolutionary advantage of learning-driven exploration lies in the generalized utility of an accurate internal model. Consistent with this hypothesis, we demonstrate that agents which learn more efficiently during exploration are later better able to accomplish a range of goal-directed tasks. We will conclude by discussing how our work elucidates the explorative behaviors of animals and humans, its relationship to other computational models of behavior, and its potential application to experimental design, such as in closed-loop neurophysiology studies.

Sensory Metrics of Neuromechanical Trust

“…* We assume informational organisms will seek to recalibrate by seeking clean, unambiguous signals in natural environments. That is, they will find such signals attractive and palatable in some informational sense like "interestingness" (Schmidhuber, 2009) or "curiosity" (Little & Sommer, 2013). We map this appetite for clean sensory data to the concept of attractiveness or "palatability" (Lutter & Nestler, 2009;Na, Morris, Johnson, Beltz, & Johnson, 2006).…”

Parallel Signal Processing of a Wireless Pressure‐Sensing Platform Combined with Machine‐Learning‐Based Cognition, Inspired by the Human Somatosensory System

“…Through cognitive learning, we are able to improve the perception (e.g., quantification, categorization, spatial differentiation) of tactile information and better interact with our surroundings . Such an ability is also expected to be an important feature in humanoid applications, as it will enable them to adapt to changes to their surroundings and tasks . CNN has connectivity patterns between neurons that resemble the organization of the sensory cortex.…”

“…[38][39][40] Such an ability is also expected to be an important feature in humanoid applications, as it will enable them to adapt to changes to their surroundings and tasks. [41,42] CNN has connectivity patterns between neurons that resemble the organization of the sensory cortex. Analogously to synaptic strengths and neural receptive fields of the sensory cortex, CNN consists of i) linear operations filtering by a set of weights, [36] ii) a pointwise nonlinear operation, called activation, iii) pooling, a nonlinear Adv.…”

Nearly maximally predictive features and their dimensions

“…DOI: 10.1103/PhysRevE.95.051301 Often, we wish to find a minimal maximally predictive model consistent with available data. Perhaps we are designing interactive agents that reap greater rewards by developing a predictive model of their environment [1][2][3][4][5][6] or, perhaps, we wish to build a predictive model of experimental data because we believe that the resultant model gives insight into the underlying mechanisms of the system [7,8]. Either way, we are almost always faced with constraints that force us to efficiently compress our data [9].…”