Algorithms and Barriers in the Symmetric Binary Perceptron Model

Abstract

The binary (or Ising) perceptron is a toy model of a single-layer neural network and can be viewed as a random constraint satisfaction problem with a high degree of connectivity. The model and its symmetric variant, the symmetric binary perceptron (SBP), have been studied widely in statistical physics, mathematics, and machine learning.The SBP exhibits a dramatic statistical-to-computational gap: the densities at which known efficient algorithms find solutions are far below the threshold for the existence of solutions. Furthermore, the SBP exhibits a striking structural property: at all positive constraint densities almost all of its solutions are 'totally frozen' singletons separated by large Hamming distance [PX21,ALS21b]. This suggests that finding a solution to the SBP may be computationally intractable. At the same time, however, the SBP does admit polynomial-time search algorithms at low enough densities. A conjectural explanation for this conundrum was put forth in [BDVLZ20]: efficient algorithms succeed in the face of freezing by finding exponentially rare clusters of large size. However, it was discovered recently that such rare large clusters exist at all subcritical densities, even at those well above the limits of known efficient algorithms [ALS21a]. Thus the driver of the statisticalto-computational gap exhibited by this model remains a mystery.In this paper, we conduct a different landscape analysis to explain the algorithmic tractability of this problem. We show that at high enough densities the SBP exhibits the multi Overlap Gap Property (m−OGP), an intricate geometrical property known to be a rigorous barrier for large classes of algorithms. Our analysis shows that the m−OGP threshold (a) is well below the satisfiability threshold; and (b) matches the best known algorithmic threshold up to logarithmic factors as m → ∞. We then prove that the m−OGP rules out the class of stable algorithms for the SBP above this threshold. We conjecture that the m → ∞ limit of the m-OGP threshold marks the algorithmic threshold for the problem. Furthermore, we investigate the stability of known efficient algorithms for perceptron models and show that the Kim-Roche algorithm [KR98], devised for the asymmetric binary perceptron, is stable in the sense we consider.

Algorithms and Barriers in the Symmetric Binary Perceptron Model

“…The proof of Theorem 1.4 is based on a Ramsey-theoretic argument developed in [GK21a] and also used in [GKPX22] coupled with the m-OGP result, Theorem 3.2; it rules out stable algorithms succeeding with a constant probability.…”

“…As an immediate corollary, this yields an efficient algorithm for the SBP that finds a solution σ ∈ S α (κ) w.h.p. if α = O(κ 2 ), see [GKPX22,Corollary 3.6]. In fact, this is the best known algorithmic guarantee both for the SBP and for discrepancy in the random proportional regime, see [GKPX22, Section 3.3].…”

“…And does the existence of efficient algorithms depend on α? Efficient algorithms at small densities α were given in [KR98] and [ALS21a] for the ABP and SBP respectively while on the negative side, [GKPX22] studied the limits of efficient algorithms (see details below). The works [BBBZ07, BIL + 15, BDVLZ20] put forth possible explanations for the success of efficient algorithms: while almost all solutions are totally frozen (conjectured in [MMZ05,HK14]), efficient algorithms access rare solutions lying in large clusters.…”

“…below α c (κ) which, per (2), is asymptotically 1 log 2 (1/κ) . Origins of this SCG were investigated in [GKPX22], where it was shown that the SBP exhibits an intricate geometrical property called the multi Overlap Gap Property (m-OGP) when α = Ω(κ 2 log 2 1 κ ) and consequently stable algorithms fail to find a satisfying solution for α = Ω(κ 2 log 2 1 κ ). It is worth noting, though, that stable algorithms need not include online algorithms, which achieve the computational threshold for the SBP.…”

Critical Window of The Symmetric Perceptron

“…There are interesting connections between average-case complexity and the geometry of the "solution space" for optimization problems. We refer the interested reader to [1,2,4,10,16] for further discussion.…”