On the principles of Parsimony and Self-consistency for the emergence of intelligence

Ma, Yi; Tsao, Doris Y.; Shum, Heung-Yeung

doi:10.1631/fitee.2200297

Cited by 30 publications

(10 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This may be a general strategy for object representations in the primate brain 50 . Further, this particular kind of sparse representation has been explored in machine learning [51][52][53] and is thought to be essential for flexible and intelligent behavior 54 .…”

Section: Discussionmentioning

confidence: 99%

Abstract representations emerge naturally in neural networks trained to perform multiple tasks

Johnston

Fusi

2023

Nat Commun

View full text Add to dashboard Cite

Humans and other animals demonstrate a remarkable ability to generalize knowledge across distinct contexts and objects during natural behavior. We posit that this ability to generalize arises from a specific representational geometry, that we call abstract and that is referred to as disentangled in machine learning. These abstract representations have been observed in recent neurophysiological studies. However, it is unknown how they emerge. Here, using feedforward neural networks, we demonstrate that the learning of multiple tasks causes abstract representations to emerge, using both supervised and reinforcement learning. We show that these abstract representations enable few-sample learning and reliable generalization on novel tasks. We conclude that abstract representations of sensory and cognitive variables may emerge from the multiple behaviors that animals exhibit in the natural world, and, as a consequence, could be pervasive in high-level brain regions. We also make several specific predictions about which variables will be represented abstractly.

show abstract

Section: Discussionmentioning

confidence: 99%

Abstract representations emerge naturally in neural networks trained to perform multiple tasks

Johnston

Fusi

2023

Nat Commun

View full text Add to dashboard Cite

show abstract

“…In summary, in this study we provide detailed, comprehensive, longitudinal data on the formation of a cognitive map in the hippocampus during learning of a moderately complex task, which progresses in stages over the course of several training sessions. The excellent match between CSCGs, Hebbian-RNNs, and hippocampal data suggest that state machines with sparse, orthogonalized representations are likely to provide a powerful framework for neural computation 104 , memory 105,127 , and intelligence 128 . These results also underscore the need to understand how different learning rules such as local Hebbian learning and gradient descent-based learning might synergistically function to promote a more effective and efficient learning process in both natural and artificial learning systems 71,129,130 .…”

Section: Discussionmentioning

confidence: 99%

Learning produces a hippocampal cognitive map in the form of an orthogonalized state machine

Sun,

Winnubst,

Natrajan

et al. 2023

Preprint

View full text Add to dashboard Cite

Cognitive maps confer animals with flexible intelligence by representing spatial, temporal, and abstract relationships that can be used to shape thought, planning, and behavior. Cognitive maps have been observed in the hippocampus, but their algorithmic form and the processes by which they are learned remain obscure. Here, we employed large-scale, longitudinal two-photon calcium imaging to record activity from thousands of neurons in the CA1 region of the hippocampus while mice learned to efficiently collect rewards from two subtly different versions of linear tracks in virtual reality. The results provide a detailed view of the formation of a cognitive map in the hippocampus. Throughout learning, both the animal behavior and hippocampal neural activity progressed through multiple intermediate stages, gradually revealing improved task understanding and behavioral efficiency. The learning process led to progressive decorrelations in initially similar hippocampal neural activity within and across tracks, ultimately resulting in orthogonalized representations resembling a state machine capturing the inherent structure of the task. We show that a Hidden Markov Model (HMM) and a biologically plausible recurrent neural network trained using Hebbian learning can both capture core aspects of the learning dynamics and the orthogonalized representational structure in neural activity. In contrast, we show that gradient-based learning of sequence models such as Long Short-Term Memory networks (LSTMs) and Transformers do not naturally produce such representations. We further demonstrate that mice exhibited adaptive behavior in novel task settings, with neural activity reflecting flexible deployment of the state machine. These findings shed light on the mathematical form of cognitive maps, the learning rules that sculpt them, and the algorithms that promote adaptive behavior in animals. The work thus charts a course toward a deeper understanding of biological intelligence and offers insights toward developing more robust learning algorithms in artificial intelligence.

show abstract

“…In a broad sense, compositionality can be seen as a particular way exploiting or imposing structure in the inner representations of a network. It has also been argued that data representations should be concentrated in low-dimensional linear spaces [29,8], or even be "disentangled" with respect to factors variation in the data [23,7,1]. Our perspective on compositional representations is closely related to the definition of disentanglement given in [23].…”

Section: Related Workmentioning

confidence: 98%

Linear Spaces of Meanings: the Compositional Language of VLMs

Trager¹,

Perera²,

Zancato³

et al. 2023

Preprint

View full text Add to dashboard Cite

We investigate compositional structures in vector data embeddings from pre-trained vision-language models (VLMs). Traditionally, compositionality has been associated with algebraic operations on embeddings of words from a pre-existing vocabulary. In contrast, we seek to approximate label representations from a text encoder as combinations of a smaller set of vectors in the embedding space. These vectors can be seen as "ideal words" which can be used to generate new concepts in an efficient way. We present a theoretical framework for understanding linear compositionality, drawing connections with mathematical representation theory and previous definitions of disentanglement. We provide theoretical and empirical evidence that ideal words provide good compositional approximations of composite concepts and can be more effective than token-based decompositions of the same concepts.

show abstract

On the principles of Parsimony and Self-consistency for the emergence of intelligence

Cited by 30 publications

References 84 publications

Abstract representations emerge naturally in neural networks trained to perform multiple tasks

Abstract representations emerge naturally in neural networks trained to perform multiple tasks

Learning produces a hippocampal cognitive map in the form of an orthogonalized state machine

Linear Spaces of Meanings: the Compositional Language of VLMs

Contact Info

Product

Resources

About