Bits from Brains for Biologically Inspired Computing

Abstract

Inspiration for artificial biologically inspired computing is often drawn from neural systems. This article shows how to analyze neural systems using information theory with the aim of obtaining constraints that help to identify the algorithms run by neural systems and the information they represent. Algorithms and representations identified this way may then guide the design of biologically inspired computing systems. The material covered includes the necessary introduction to information theory and to the estimation of information-theoretic quantities from neural recordings. We then show how to analyze the information encoded in a system about its environment, and also discuss recent methodological developments on the question of how much information each agent carries about the environment either uniquely or redundantly or synergistically together with others. Last, we introduce the framework of local information dynamics, where information processing is partitioned into component processes of information storage, transfer, and modification -locally in space and time. We close by discussing example applications of these measures to neural data and other complex systems.

Partial and Entropic Information Decompositions of a Neuronal Modulatory Interaction

“…Information processing is more than simply transferring information from one time or place to another. As others have argued it also includes creating new information via synergetic interactions between separate inputs; see [26,38]. Our argument here is that in addition to "enhancing computational capabilities via synergy" information processing also includes distinguishing between currently relevant and currently irrelevant inputs.…”

“…The particular (local) evidence provided by the value of the input on that trial moved the conditional distribution in the wrong direction for that output value-i.e., it was misleading about that particular output value, because it suggested it was less likely to happen, but then it did happen anyway. The fact that negative local values correspond to misleading evidence from the perspective of prediction explains why they have been termed misleading information or "misinformation" [26].…”

“…Negative local information terms, which have been called "misinformation" [26], arise for symbols where h(x, y) > h(x) + h(y); that is, those symbols provide a synergistic contribution to the joint entropy expectation sum. The existence of such locally synergistic entropy terms suggest that synergistic entropy is a reasonable thing to quantify within the PED framework.…”

“…Since at a local or pointwise level [24][25][26][27][28] (i.e., the terms inside the expectation), information is equal to change in surprisal, I ccs seeks to measure shared information as the change in surprisal that is common to the input variables (hence CCS, Common Change in Surprisal). For two inputs, I ccs is defined as:…”

Quantifying Information Modification in Developing Neural Networks via Partial Information Decomposition

“…Thus, the meshed architecture seems to have evolved to "fuse" information from different input sources (including a neuron's recent spiking history and its current state) in a nontrivial way, e.g., other than simply multiplexing it in the output. In other words, the distributed computation in neural systems may heavily rely on information modification [1].…”