The Landscape of the Spiked Tensor Model

Abstract

We consider the problem of estimating a large rank‐one tensor u⊗k ∈ (ℝn)⊗k, k ≥ 3, in Gaussian noise. Earlier work characterized a critical signal‐to‐noise ratio λ Bayes = O(1) above which an ideal estimator achieves strictly positive correlation with the unknown vector of interest. Remarkably, no polynomial‐time algorithm is known that achieved this goal unless λ ≥ Cn(k − 2)/4, and even powerful semidefinite programming relaxations appear to fail for 1 ≪ λ ≪ n(k − 2)/4. In order to elucidate this behavior, we consider the maximum likelihood estimator, which requires maximizing a degree‐k homogeneous polynomial over the unit sphere in n dimensions. We compute the expected number of critical points and local maxima of this objective function and show that it is exponential in the dimensions n, and give exact formulas for the exponential growth rate. We show that (for λ larger than a constant) critical points are either very close to the unknown vector u or are confined in a band of width Θ(λ−1/(k − 1)) around the maximum circle that is orthogonal to u. For local maxima, this band shrinks to be of size Θ(λ−1/(k − 2)). These “uninformative” local maxima are likely to cause the failure of optimization algorithms. © 2019 Wiley Periodicals, Inc.

Distribution of rare saddles in the p -spin energy landscape

“…High-dimensional systems are typically associated to complex, highly non-convex energy landscapes, in which the number of stationary points (local minima, maxima or saddles) increases steeply with the dimensionality. Classifying these points in terms of their energy, of their stability and of their location in the underlying configuration space is a topic that is of interest in a large variety of fields, including disordered systems [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15], ecology and biology [16][17][18][19], neural networks [20,21], inference [22][23][24][25][26], game theory [27], string theory and cosmology [28,29]. In many of these contexts, a crucial motivation for determining the distribution of stationary points is to understand how the energy functional is explored dynamically, through algorithms that proceed via local moves in configuration space, biased towards lower-energy configurations.…”

Nature Versus Nurture: Dynamical Evolution in Disordered Ising Ferromagnets

“…It is natural in these models to analyze the magnetization of either the Gibbs measure, or as a function of time with respect to some Langevin/gradient descent algorithm, as the spike strength λ varies. In the spherical setting, the structure of local minima and the behavior of zero-temperature gradient descent algorithms have seen much recent attention [2,14,13,1].…”

Local Minima in Disordered Mean-Field Ferromagnets

“…In particular, as the spike strength λ increases, the landscape trivializes so that there are no critical points besides the ground states above a threshold magnetization M triv (λ); however, it is expected (and confirmed at the annealed level) that there are many critical points with |M (S)| < M triv (λ). As in our setting, there is a close relation between this complexity picture and the trapping of gradient-based algorithms as well as their success in signal recovery for spiked tensors [14][15][16][17].…”