Optimal storage properties of neural network models

Gardner, E.; Derrida, Bernard

doi:10.1088/0305-4470/21/1/031

Cited by 506 publications

(433 citation statements)

References 26 publications

Supporting

Mentioning

412

Contrasting

Unclassified

Order By: Relevance

“…For certain degree of asymmetry, the capacity increases as the network size increases. Our results are consistent with the previous studies (41)(42)(43). Some details of the dynamical behavior of a system with weak asymmetric are studied in ref.…”

Section: Flux and Asymmetric Synaptic Connections In General Neuralsupporting

confidence: 93%

Nonequilibrium landscape theory of neural networks

Yan

Zhao

et al. 2013

Proc. Natl. Acad. Sci. U.S.A.

144

View full text Add to dashboard Cite

The brain map project aims to map out the neuron connections of the human brain. Even with all of the wirings mapped out, the global and physical understandings of the function and behavior are still challenging. Hopfield quantified the learning and memory process of symmetrically connected neural networks globally through equilibrium energy. The energy basins of attractions represent memories, and the memory retrieval dynamics is determined by the energy gradient. However, the realistic neural networks are asymmetrically connected, and oscillations cannot emerge from symmetric neural networks. Here, we developed a nonequilibrium landscape-flux theory for realistic asymmetrically connected neural networks. We uncovered the underlying potential landscape and the associated Lyapunov function for quantifying the global stability and function. We found the dynamics and oscillations in human brains responsible for cognitive processes and physiological rhythm regulations are determined not only by the landscape gradient but also by the flux. We found that the flux is closely related to the degrees of the asymmetric connections in neural networks and is the origin of the neural oscillations. The neural oscillation landscape shows a closed-ring attractor topology. The landscape gradient attracts the network down to the ring. The flux is responsible for coherent oscillations on the ring. We suggest the flux may provide the driving force for associations among memories. We applied our theory to rapid-eye movement sleep cycle. We identified the key regulation factors for function through global sensitivity analysis of landscape topography against wirings, which are in good agreements with experiments.neural circuits | free energy | entropy production | nonequilibrium thermodynamics A grand goal of biology is to understand the function of the human brain. The brain is a complex dynamical system (1-6). The individual neurons can develop action potentials and connect with each other through synapses to form the neural circuits. The neural circuits of the brain perpetually generate complex patterns of activity that have been shown to be related with special biological functions, such as learning, long-term associative memory, working memory, olfaction, decision making and thinking (7-9), etc. Many models have been proposed for understanding how neural circuits generate different patterns of activity. HodgkinHuxley model gives a quantitative description of a single neuronal behavior based on the voltage-clamp measurements of the voltage (4). However, various vital functions are carried out by the circuit rather than individual neurons. It is at present still challenging to explore the underlying global natures of the large neural networks built from individual neurons.Hopfield developed a model (5, 6) that makes it possible to explore the global natures of the large neural networks without losing the information of essential biological functions. For symmetric neural circuits, an energy landscape can be constructed that decreas...

show abstract

Section: Flux and Asymmetric Synaptic Connections In General Neuralsupporting

confidence: 93%

Nonequilibrium landscape theory of neural networks

Yan

Zhao

et al. 2013

Proc. Natl. Acad. Sci. U.S.A.

144

View full text Add to dashboard Cite

show abstract

“…In particular, the perceptron (35,36) and networks composed of McCulloch and Pitts neurons (37) received much attention (32,(38)(39)(40)(41)(42)(43)(44)(45)(46). This is due in part to the existence of a theoretical framework for solving such problems, which was initially developed in the context of statistical physics.…”

mentioning

confidence: 99%

Efficient associative memory storage in cortical circuits of inhibitory and excitatory neurons

Chapeton

Fares

LaSota

et al. 2012

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Many features of synaptic connectivity are ubiquitous among cortical systems. Cortical networks are dominated by excitatory neurons and synapses, are sparsely connected, and function with stereotypically distributed connection weights. We show that these basic structural and functional features of synaptic connectivity arise readily from the requirement of efficient associative memory storage. Our theory makes two fundamental predictions. First, we predict that, despite a large number of neuron classes, functional connections between potentially connected cells must be realized with <50% probability if the presynaptic cell is excitatory and >50% probability if the presynaptic cell is inhibitory. Second, we establish a unique relation between probability of connection and coefficient of variation in connection weights. These predictions are consistent with a dataset of 74 published experiments reporting connection probabilities and distributions of postsynaptic potential amplitudes in various cortical systems. What is more, our theory explains the shapes of the distributions obtained in these experiments.learning and memory | cortical connectivity | synaptic weight | perceptron | critical capacity F undamental functions of the brain, such as learning and memory storage, are mediated by many mechanisms of excitatory (1-4) and inhibitory (5-9) synaptic plasticity. Working together with the genetically encoded developmental mechanisms of circuit formation, synaptic plasticity shapes neural circuits by creating, modifying, and eliminating individual synaptic connections in an experience-dependent manner. It is, therefore, reasonable to hypothesize that many stereotypic features of adult synaptic connectivity, whether established through evolution or the developmental learning process, have arisen to facilitate memory storage.In this study, we focus on three such features of cortical connectivity. Cortical connectivity is predominantly excitatory; it is mediated by two major classes of neurons-excitatory glutamatergic and inhibitory GABAergic cells. Chemical synapses made by the axons of inhibitory cells in the adult brain are believed to be all inhibitory, whereas those synapses made by the axons of excitatory neurons are believed to be all excitatory (10). The resulting connectivity is largely excitatory, with only about 15-20% of inhibitory neurons and inhibitory synapses (11). The second stereotypic feature of cortical connectivity is sparseness. Networks in the cortex are thought to be organized into relatively small units ranging from hundreds to tens of thousands of neurons in size. Such units may include mini columns (12, 13), structural columns (14, 15), and a variety of functional columns (16,17). Analysis of neuron morphology (14,(18)(19)(20)(21) has shown that cells within such units have the potential of being connected by structural synaptic plasticity (22-24). However, despite this potential, synaptic connectivity within the units is sparse. For example, nearby excitatory neurons in the neocortex are sy...

show abstract

“…Surrogate modelling (Queipo et al, 2005) typically begins with a large amount of quantitative data, collected from application cases or generated from computer simulations, of the domain. Given these organised databases, various multiple regression methods, such as statistics (Moses, 1986), machine learning (Witten and Frank, 2005), neural network (Gardner and Derrida, 1988), etc., are employed to automatically generate empirical 'layered models' with varying degrees of abstraction level of the quantitative domain knowledge. Abstract qualitative surrogate models, which are built directly from the more detailed quantitative domain knowledge, can be used to support early-stage discourses among stakeholders.…”

Section: Measurability Of Preferences In Collaborative Engineering Dementioning

confidence: 99%

Collective rationality of group decisions in collaborative engineering

Lu¹

2009

IJCE

View full text Add to dashboard Cite

Abstract:Collaborative engineering is a dynamic socio-technical activity where a team of stakeholders works collaboratively to make group decisions based on collective rationality. This paper examines various impossibility conditions and possibility requirements for the existence of collective rationality from both theoretical and practical standpoints. Arrow's Impossibility Theorem is examined in light of the special characteristics of collaborative engineering problems. Since from a theoretical standpoint, no social welfare function can satisfy Arrow's rationality conditions of group decisions, this paper suggests some practical methods to guide collaborative engineering teams through teamwork and task-work iterations to approach collective rationality systematically when making group decisions.

show abstract

Optimal storage properties of neural network models

Cited by 506 publications

References 26 publications

Nonequilibrium landscape theory of neural networks

Nonequilibrium landscape theory of neural networks

Efficient associative memory storage in cortical circuits of inhibitory and excitatory neurons

Collective rationality of group decisions in collaborative engineering

Contact Info

Product

Resources

About