Contextually guided unsupervised learning using local multivariate binary processors

Abstract

We consider the role of contextual guidance in learning and processing within multi-stream neural networks. Earlier work (Kay & Phillips, 1994Phillips et al., 1995 showed how the goals of feature discovery and associative learning could be fused within a single objective, and made precise using information theory, in such a way that local binary processors could extract a single feature that is coherent across streams. In this paper we consider multi-unit local processors with multivariate binary outputs that enable a greater numberofcoherent features to be extracted. Using the Ising model, we de ne a class of information-theoretic objective functions and also local approximations, and derive the learning rules in both cases. These rules have similarities to, and di erences from, the celebrated BCM rule. Local and global versions of Infomax appear as by-products of the general approach, as well as multivariate versions of Coherent Infomax. Focussing on the more biologically plausible local rules, we describe some computational experiments designed to investigate speci c properties of the processors. The main conclusions are:1. The local methodology introduced in the paper has the required functionality. 2. Di erent units within the multi-unit processors learned to respond to di erent aspects of their receptive elds. 3. The units within each processor generally produced a distributed code in which the outputs were correlated, and which w as robust to damage in the special case where the numberofunitsavailable was only just su cient to transmit the relevant information, a form of competitive learning was produced. 4. The contextual connections enabled the information correlated across streams to be extracted, and, by improving feature detection with weak or noisy inputs, they played a useful role in short-term processing and in improving generalization. 5. The methodology allows the statistical associations between distributed self-organizing population codes to be learned.

On the functions, mechanisms, and malfunctions of intracortical contextual modulation

“…That is done by the theory of coherent infomax, which proposes a general account of cortical function in which contextual modulation plays a central role that is crucial to both learning and processing (Phillips, Kay and Smyth, 1995;Kay, Floreano and Phillips, 1998;Kay and Phillips, 2011). Contextual modulation as conceived within that theory is therefore examined further here.…”

Dynamic Coordination in Brain and Mind

“…This intuition is misleading. The information that is transmitted specifically about a modulatory or coordinating input given the coding, or receptive field, input can tend to zero, even when that modulating input has a large effect on the transmission of receptive field information (Kay et al 1998). Therefore, conditional mutual information measures can be used to distinguish coordinating from coding interactions (Smyth et al 1996).…”

“…Kay et al 1998); Coordination Dynamics (e.g. Kelso 1995); and predictive coding under the free-energy principle (e.g.…”

“…These goals require knowledge of what predicts what, and thus some form of learning analogous to latent structure analysis (e.g. Becker and Hinton 1992;Kay et al 1998). At least three other goals can also be distinguished, i.e.…”

“…What elementary computational operations are performed by cortical microcircuits, and, in particular, are there mechanisms that make a special contribution to dynamic coordination? Possible mechanisms include those underlying " modulatory" as contrasted with "driving" interactions (Sherman and Guillery 1998), "contextual fields" as contrasted with "receptive fields" (Kay et al 1998;Phillips and Singer 1997), local circuit mechanisms for gain control (Tiesinga and Sejnowski 2004), and local circuit mechanisms for coordinating phase relations between rhythmic activities (Whittington and Traub 2003). Rhythms were thought to be highly relevant because they have a special role in dynamic grouping, attention, and other relevant cognitive processes (Buzsáki 2006).…”