Achieving Exact Cluster Recovery Threshold via Semidefinite Programming

Hajek, Bruce; Wu, Yihong; Xu, Jiaming

doi:10.1109/tit.2016.2546280

Cited by 184 publications

(203 citation statements)

References 24 publications

Supporting

Mentioning

197

Contrasting

Order By: Relevance

“…In view of [20,Theorem 5], A − E [A] ≤ c ′ √ log n with high probability for a positive constant c ′ depending only on a. Thus, the desired (14) holds in view of (13) and (15), completing the proof.…”

Section: It Follows Thatmentioning

confidence: 60%

“…In the converse direction, necessary conditions for the success of particular SDPs are obtained in [32], [13]. In contrast to the previous work mentioned above where the constants are often loose, the recent line of work initiated by [2], [1], and followed by [20], [9], [4], [31] and the current paper, focus on establishing necessary and sufficient conditions in the special case of a fixed number of clusters with sharp constants, attained via SDP relaxations.…”

Section: B Further Literature On Sdp For Cluster Recoverymentioning

confidence: 99%

“…In the setting of two equal-sized clusters, recently it has been shown in [20] that the semidefinite programming (SDP) relaxation of the maximum likelihood (ML) estimator achieves the optimal recovery threshold with high probability, in the asymptotic regime of p = a log n/n and q = b log n/n for fixed constants a, b. The result for two equal-sized clusters was originally conjectured in [2] and another resolution was recently given in [9] independently.…”

Section: Introductionmentioning

confidence: 99%

“…In particular, we show that a necessary condition for the maximum likelihood (ML) estimator to succeed coincides with a sufficient condition for the correctness of the SDP procedure, thereby establishing both the optimal recovery threshold and the optimality of the SDP relaxation. The proof techniques in this paper are similar to those in [20]; however, the construction and validation of dual certificates for the success of SDP are more challenging in the multiple-cluster case.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Achieving exact cluster recovery threshold via semidefinite programming under the stochastic block model

Hajek

2015

2015 49th Asilomar Conference on Signals, Systems and Computers

Self Cite

View full text Add to dashboard Cite

Resolving a conjecture of Abbe, Bandeira and Hall, the authors have recently shown that the semidefinite programming (SDP) relaxation of the maximum likelihood estimator achieves the sharp threshold for exactly recovering the community structure under the binary stochastic block model of two equal-sized clusters. Extending the proof techniques, in this paper it is shown that SDP relaxations also achieve the sharp recovery threshold under the stochastic block model with a fixed number of equal-sized clusters.

show abstract

Section: It Follows Thatmentioning

confidence: 60%

Section: B Further Literature On Sdp For Cluster Recoverymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Achieving exact cluster recovery threshold via semidefinite programming under the stochastic block model

Hajek

2015

2015 49th Asilomar Conference on Signals, Systems and Computers

Self Cite

View full text Add to dashboard Cite

show abstract

“…The problem of partitioning a graph into two well-connected subcommunities can be viewed as synchronization over the group f˙1g Š Z=2: each vertex has a latent group element g u 2 f˙1g, its community identity, and each edge is a noisy measurement of the relative status g u g 1 v [5]. A number of more structured approaches have been shown to improve over PCA, including modified spectral methods [37,38,44,50,59] and semidefinite programming [1,2,32,33,49]. However, this approach breaks down in sparse graphs: localized noise eigenvectors associated to high-degree vertices dominate the spectrum.…”

Section: Introductionmentioning

confidence: 99%

Message‐Passing Algorithms for Synchronization Problems over Compact Groups

Perry

Wein

Bandeira

et al. 2018

Comm Pure Appl Math

View full text Add to dashboard Cite

Various alignment problems arising in cryo-electron microscopy, community detection, time synchronization, computer vision, and other fields fall into a common framework of synchronization problems over compact groups such as Z=L, U.1/, or SO.3/. The goal in such problems is to estimate an unknown vector of group elements given noisy relative observations. We present an efficient iterative algorithm to solve a large class of these problems, allowing for any compact group, with measurements on multiple "frequency channels" (Fourier modes, or more generally, irreducible representations of the group). Our algorithm is a highly efficient iterative method following the blueprint of approximate message passing (AMP), which has recently arisen as a central technique for inference problems such as structured low-rank estimation and compressed sensing. We augment the standard ideas of AMP with ideas from representation theory so that the algorithm can work with distributions over general compact groups. Using standard but nonrigorous methods from statistical physics, we analyze the behavior of our algorithm on a Gaussian noise model, identifying phases where we believe the problem is easy, (computationally) hard, and (statistically) impossible. In particular, such evidence predicts that our algorithm is information-theoretically optimal in many cases, and that the remaining cases exhibit statistical-to-computational gaps.

show abstract

Proof of the Achievability Conjectures for the General Stochastic Block Model

Abbé

Sandon

2017

Comm Pure Appl Math

View full text Add to dashboard Cite

In a paper that initiated the modern study of the stochastic block model (SBM), Decelle, Krzakala, Moore, and Zdeborová, backed by Mossel, Neeman, and Sly, conjectured that detecting clusters in the symmetric SBM in polynomial time is always possible above the Kesten‐Stigum (KS) threshold, while it is possible to detect clusters information theoretically (i.e., not necessarily in polynomial time) below the KS threshold when the number of clusters k is at least 4. Massoulié, Mossel et al., and Bordenave, Lelarge, and Massoulié proved that the KS threshold is in fact efficiently achievable for k = 2, while Mossel et al. proved that it cannot be crossed at k = 2. The above conjecture remained open for k ≥ 3. This paper proves the two parts of the conjecture, further extending the results to general SBMs. For the efficient part, an approximate acyclic belief propagation (ABP) algorithm is developed and proved to detect communities for any k down to the KS threshold in quasi‐linear time. Achieving this requires showing optimality of ABP in the presence of cycles, a challenge for message‐passing algorithms. The paper further connects ABP to a power iteration method on a nonbacktracking operator of generalized order, formalizing the interplay between message passing and spectral methods. For the information‐theoretic part, a nonefficient algorithm sampling a typical clustering is shown to break down the KS threshold at k = 4. The emerging gap is shown to be large in some cases, making the SBM a good case study for information‐computation gaps. © 2017 Wiley Periodicals, Inc.

show abstract

Achieving Exact Cluster Recovery Threshold via Semidefinite Programming

Cited by 184 publications

References 24 publications

Achieving exact cluster recovery threshold via semidefinite programming under the stochastic block model

Achieving exact cluster recovery threshold via semidefinite programming under the stochastic block model

Message‐Passing Algorithms for Synchronization Problems over Compact Groups

Proof of the Achievability Conjectures for the General Stochastic Block Model

Contact Info

Product

Resources

About