Consistency of spectral clustering in stochastic block models

Abstract

We analyze the performance of spectral clustering for community extraction in stochastic block models. We show that, under mild conditions, spectral clustering applied to the adjacency matrix of the network can consistently recover hidden communities even when the order of the maximum expected degree is as small as log n, with n the number of nodes. This result applies to some popular polynomial time spectral clustering algorithms and is further extended to degree corrected stochastic block models using a spherical k-median spectral clustering method. A key component of our analysis is a combinatorial bound on the spectrum of binary random matrices, which is sharper than the conventional matrix Bernstein inequality and may be of independent interest.

Bounding Extremal Degrees of Edge-Independent Random Graphs Using Relative Entropy

“…This also lends support to the conjecture made in [3] that Theorem 1 therein (regarding the behavior of adjacency eigenvalues of edge-independent random graphs) holds when ∆(A) ln n. A partial solution in this direction can be found in [8].…”

“…The Estrada index and the normalized Laplacian Estrada index of G n (p ij ) for large n are examined in [7]. The problem of bounding the difference between eigenvalues of A and those of the adjacency matrix of G n (p ij ), together with its Laplacian spectra version, has been studied intensively recently; see, e.g., [3,8,9]. It is revealed in [9] that large deviation from the expected spectrum is caused by vertices with extremal degrees, where abnormally high-degree and low-degree vertices are obstructions to concentration of the adjacency and the Laplacian matrices, respectively.…”

Concentration and regularization of random graphs

“…One may wonder if Theorem 2.1 can be proved by developing an -net argument similar to the method of J. Kahn and E. Szemeredi [17] and its versions [2,15,27,12]. Although we can not rule out such possibility, we do not see how this method could handle a general regularization.…”

Minimax rates of community detection in stochastic block models

“…Detecting and identifying communities is fundamentally important to understand the underlying structure of the network [12]. Many models and methodologies have been proposed for community detection from different perspectives, including RatioCut[13], Ncut [26], and spectral method [19,25,16] from computer science, Newman-Girvan Modularity [12] from physics, semi-definite programming [7,14] from engineering, and maximum likelihood estimation [3,6] from statistics.Deep theoretical developments have been actively pursued as well. Recently, celebrated works of Mossel et al [20,21] and Massoulie [18] considered balanced two-community sparse networks, and discovered the threshold phenomenon for both weak and strong consistency of community detection.…”

“…Detecting and identifying communities is fundamentally important to understand the underlying structure of the network [12]. Many models and methodologies have been proposed for community detection from different perspectives, including RatioCut [13], Ncut [26], and spectral method [19,25,16] from computer science, Newman-Girvan Modularity [12] from physics, semi-definite programming [7,14] from engineering, and maximum likelihood estimation [3,6] from statistics.…”