Overparameterized neural networks implement associative memory

Abstract

Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. We provide empirical evidence that 1) overparameterized autoencoders store training samples as attractors and thus iterating the learned map leads to sample recovery, and that 2) the same mechanism allows for encoding sequences of examples and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.

Causal network models of SARS-CoV-2 expression and aging to identify candidates for drug repurposing

“…While autoencoders and other generative models have been used for computing signatures of perturbations also in other works 39,40 , these works have used autoencoders in the standard way to obtain a lower-dimensional embedding of the data. Motivated by our recent work which, quite counter-intuitively, described various benefits of using autoencoders to learn a latent representation of the data that is higher-dimensional than the original space 41 , we found that overparameterized autoencoders not only led to the better Protein coding genes SARS-CoV-2 L1000 Fig. 3 Mining FDA-approved drugs by correlating disease and drug signatures using an overparameterized autoencoder embedding.…”

“…First, in order to ensure that the CMap database, which measures expression using 1000 representative genes, would be useful in the context of SARS-CoV-2, we validated that the intersection of these genes with the SARS-CoV-2 differentially expressed genes was significant. Second, to establish drug signatures based on the CMap database, we employed a particular autoencoder framework 41 . Rather unintuitively, we showed that using an overparameterized autoencoder, i.e., by using an autoencoder not to perform dimension reduction as usual but to instead embed the data into a higher-dimensional space, aligned the drug signatures across different cell types.…”

Harnessing Matrix Completion to Unify and Extend Viral Serology Studies

“…For example, we used the same code to analyze the six influenza studies (with rank between 6-9) and the HIV-1 Catnap dataset containing hundreds of studies (with rank 23). Further refinements to our approach could incorporate side information such as virus sequence (Radhakrishnan et al, 2021) or use tensor factorization to decompose higher-order data (Liu & Moitra, 2020).…”

Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

“…One line of enquiry is interpreting depth recursively. Indeed, in certain settings increasing the depth manifests similarly to iterating a map given by a shallow network (Radhakrishnan, Belkin and Uhler 2020). Furthermore, fixed points of such iterations have been proposed as an alternative to deep networks, with some success (Bai, Kolter and Koltun 2019).…”