Dynamic Predictive Coding with Hypernetworks

Abstract

The original predictive coding model of Rao & Ballard (1999) focused on spatial prediction to explain spatial receptive fields and contextual effects in the visual cortex. Here, we introduce a new dynamic predictive coding model that achieves spatiotemporal prediction of complex natural image sequences using time-varying transition matrices. We overcome the limitations of static linear transition models (as in, e.g., Kalman filters) using a hypernetwork to adjust the transition matrix dynamically for every time step, allowing the model to predict using a time-varying mixture of possible transition dynamics. We developed a single level model with recurrent modulation of transition weights by a hypernetwork and a two-level hierarchical model with top-down modulation based on a hypernetwork. At each time step, the model predicts the next input and estimates a sparse neural code by minimizing prediction error. When exposed to natural movies, the model learned localized, oriented spatial filters as well as both separable and inseparable (direction-selective) space-time receptive fields at the first level, similar to those found in the primary visual cortex (V1). Longer timescale responses and stability at the second level also emerged naturally from minimizing prediction errors for the first level dynamics. Our results suggest that the multiscale temporal response properties of cortical neurons could be the result of the cortex learning a hierarchical generative model of the visual world with higher order areas predicting the transition dynamics of lower order areas.

Antipsychotic drugs selectively decorrelate long-range interactions in deep cortical layers

“…Self-generated movements result in broad activation of most regions of the brain (Musall et al, 2019; Stringer et al, 2019), including some primary sensory areas like visual cortex (Keller et al, 2012; Saleem et al, 2013) where they have been shown to depend on visual context (Pakan et al, 2016). The framework of predictive processing postulates that in cortex, these movement related signals are internal model based predictions of the sensory consequences of movement that are compared to externally generated bottom-up input to compute prediction errors (Jiang and Rao, 2021; Keller and Mrsic-Flogel, 2018). Evidence for this interpretation has mainly come from the discovery of movement related prediction error responses in a variety of different cortical regions and species (Attinger et al, 2017; Audette et al, 2021; Ayaz et al, 2019; Eliades and Wang, 2008; Heindorf et al, 2018; Keller and Hahnloser, 2009; Keller et al, 2012; Stanley and Miall, 2007; Zmarz and Keller, 2016), and the observation that top-down inputs from motor areas of cortex appear to carry movement related predictions of sensory input to sensory areas of cortex (Audette et al, 2021; Leinweber et al, 2017).…”

Active Predictive Coding Networks: A Neural Solution to the Problem of Learning Reference Frames and Part-Whole Hierarchies

“…More broadly, our framework offers a new approach to hierarchical reinforcement learning and planning in continuous state and action spaces. Finally, given the close connection between APCNs and predictive coding models of brain function, the proposed framework paves the way for a new interpretation of the hierarchical architecture of the cortex and a new role for cortical feedback connections in modulating the dynamics of lower-level networks [8] similar to the role played by hypernets in APCNs.…”

“…• Predictive Coding: APCNs build on predictive coding models of cortical function [17,3,8], which emphasize the role of hierarchical prediction and prediction errors in driving learning and inference.…”

Recursive Neural Programs: Variational Learning of Image Grammars and Part-Whole Hierarchies

“…Here we use the decoded patches {x k−1 1 , ..., xk−1 τ k−1 } as accumulated evidence to update z k t (similar to other predictive coding models [11,3]).…”

“…Our model builds on past work on Active Predictive Coding Networks [3] in using state and action networks but is fully generative, recursive, and probabilistic, allowing a structured variational approach to inference and sampling of neural programs. The key differences between our approach and existing approaches are: 1) Our approach can be extended to arbitrary tree depth, creating a "grammar" for images that can be recursively applied 2) our approach provides a sensible way to perform gradient descent in hierarchical "program space," and 3) our model can be made adaptive by letting information flow from children to parents in the tree, e.g., via prediction errors [11,3].…”