Sparse Hopfield network reconstruction with ℓ 1 regularization

Abstract

We propose an efficient strategy to infer sparse Hopfield network based on magnetizations and pairwise correlations measured through Glauber samplings. This strategy incorporates the ℓ1 regularization into the Bethe approximation by a quadratic approximation to the log-likelihood, and is able to further reduce the inference error of the Bethe approximation without the regularization. The optimal regularization parameter is observed to be of the order of M −ν where M is the number of independent samples. The value of the scaling exponent depends on the performance measure. ν ≃ 0.5001 for root mean squared error measure while ν ≃ 0.2743 for misclassification rate measure. The efficiency of this strategy is demonstrated for the sparse Hopfield model, but the method is generally applicable to other diluted mean field models. In particular, it is simple in implementation without heavy computational cost.

Clustering of neural code words revealed by a first-order phase transition

“…where t and η denote the learning step and learning rate, respectively. The maximum likelihood learning shown here has a simple interpretation of minimizing the Kullback-Leibler divergence between the empirical probability and the model probability [27,28]. In the learning equation (Eq.…”

Differential Covariance: A New Class of Methods to Estimate Sparse Connectivity from Neural Recordings

“…Winterhalder et al (2005) reviewed the nonparametric methods and Granger causality–based methods. Much progress has been made recently using the kinetic Ising model and Hopfield network (Huang, 2013; Dunn & Roudi, 2013; Battistin, Hertz, Tyrcha, & Roudi, 2015; Capone, Filosa, Gigante, Ricci-Tersenghi, & Del Giudice, 2015; Roudi & Hertz, 2011) with sparsity regularization (Pernice & Rotter, 2013). The GLM method (Okatan et al, 2005; Truccolo et al, 2005; Pillow et al, 2008) and the maximum entropy method (Schneidman, Berry, Segev, & Bialek, 2006) are two popular classes of methods and the main modern approaches for modeling multiunit recordings (Roudi, Dunn, & Hertz, 2015).…”

Variational Pseudolikelihood for Regularized Ising Inference

“…I examined a simple question: what features does the Ising model learn when trained on images of the letters A-J? Using the variational pseudolikelihood approach, it is simple to infer coupling matrices with a specific structure, such as that of a Hopfield neural network [5,21,26,29]. The couplings in a Hopfield network are described by a Hebbian rule such that…”

“…As a result, the inverse Ising problem has to be solved approximately. A number of approximate methods for inferring the parameters of the Ising model have been introduced including naive mean field theory [15], the Thouless-Anderson-Palmer (TAP) approximation [16], the isolated spin pair approximation [14], the Sessak-Monasson expansion [17], and others [5,[18][19][20][21][22][23][24][25][26][27]. The second obstacle -overfitting -is more fundamental and generally affects all high dimensional problems in statistical inference.…”

“…A common approach to mitigating the effects of overfitting is to penalize parameters with large values [28]. For example, the Ising model can be 'regularized' by adding an L 2 penalty λ i<j J 2 ij (as in [28]) or an L 1 penalty λ i<j |J ij | (as in [18,21]) to the objective function that describes the fit to the data. Alternatively, the empirical moments can be modified using a 'pseudocount' according to the rules σ i → (1 − α) σ i and [28].…”