Disordered systems insights on computational hardness

Abstract

In this review article we discuss connections between the physics of disordered systems, phase transitions in inference problems, and computational hardness. We introduce two models representing the behavior of glassy systems, the spiked tensor model and the generalized linear model. We discuss the random (non-planted) versions of these problems as prototypical optimization problems, as well as the planted versions (with a hidden solution) as prototypical problems in statistical inference and learning. Based on ideas from physics, many of these problems have transitions where they are believed to jump from easy (solvable in polynomial time) to hard (requiring exponential time). We discuss several emerging ideas in theoretical computer science and statistics that provide rigorous evidence for hardness by proving that large classes of algorithms fail in the conjectured hard regime. This includes the overlap gap property, a particular mathematization of clustering or dynamical symmetry-breaking, which can be used to show that many algorithms that are local or robust to changes in their input fail. We also discuss the sum-of-squares hierarchy, which places bounds on proofs or algorithms that use low-degree polynomials such as standard spectral methods and semidefinite relaxations, including the Sherrington–Kirkpatrick model. Throughout the manuscript we present connections to the physics of disordered systems and associated replica symmetry breaking properties.

A polynomial‐time approximation scheme for the maximal overlap of two independent Erdős–Rényi graphs

“…From what we can tell, [41,57] seem to be rare natural examples for which approximation algorithms were discovered for random instances whereas the worse-case problems are known to be hard to solve; along this line, our result can be seen as a contribution for another natural example in this category, and in fact our algorithm shares similarity in spirit with the greedy algorithm proposed in [57]. In addition, the works of [41,57] take advantage of the so-called full replica symmetry breaking (FRSB) property, and in fact it is tempting to conjecture that FRSB indicates low computational complexity [24]. That being said, our analysis is purely combinatorial and does not seem to rely on the full replica symmetry breaking property explicitly.…”

Neural-prior stochastic block model

“…This is an algorithmically hard phase where exact recovery is statistically possible, but the AMP-BP algorithm is sub-optimal. At the same time the AMP-BP algorithm is conjectured optimal among efficient algorithms (Gamarnik et al 2022) and thus the hardness of this phase is believed to be intrinsic. For λ > λ algo we only find the exact recovery fixed point.…”

Algorithms and Barriers in the Symmetric Binary Perceptron Model

“…Such gaps are a universal feature of many average-case algorithmic problems arising from random combinatorial structures and high-dimensional statistical inference. A partial list of problems with an SCG include random CSPs [MMZ05, ART06, ACO08, GS17b, BH21], optimization over random graphs [GS14, COE15, Wei20], spin glasses [GJ21, HS21], planted clique [DM15, BHK + 19], and tensor decomposition [Wei22], see also the surveys by Gamarnik [Gam21] and Gamarnik, Moore, and Zdeborová [GMZ22]. Unfortunately, standard computational complexity theory is often useless due to the average-case nature of such problems 1 .…”