Spectral redemption in clustering sparse networks

Abstract

Spectral algorithms are classic approaches to clustering and community detection in networks. However, for sparse networks the standard versions of these algorithms are suboptimal, in some cases completely failing to detect communities even when other algorithms such as belief propagation can do so. Here, we present a class of spectral algorithms based on a nonbacktracking walk on the directed edges of the graph. The spectrum of this operator is much betterbehaved than that of the adjacency matrix or other commonly used matrices, maintaining a strong separation between the bulk eigenvalues and the eigenvalues relevant to community structure even in the sparse case. We show that our algorithm is optimal for graphs generated by the stochastic block model, detecting communities all of the way down to the theoretical limit. We also show the spectrum of the nonbacktracking operator for some real-world networks, illustrating its advantages over traditional spectral clustering.D etecting communities or modules is a central task in the study of social, biological, and technological networks. Two of the most popular approaches are statistical inference, where we fix a generative model such as the stochastic block model to the network (1, 2); and spectral methods, where we classify vertices according to the eigenvectors of a matrix associated with the network such as its adjacency matrix or Laplacian (3).Both statistical inference and spectral methods have been shown to work well in networks that are sufficiently dense, or when the graph is regular (4-8). However, for sparse networks with widely varying degrees, the community detection problem is harder. Indeed, it was recently shown (9-11) that there is a phase transition below which communities present in the underlying block model are impossible for any algorithm to detect. Whereas standard spectral algorithms succeed down to this transition when the network is sufficiently dense, with an average degree growing as a function of network size (8), in the case where the average degree is constant these methods fail significantly above the transition (12). Thus, there is a large regime in which statistical inference succeeds in detecting communities, but where current spectral algorithms fail.It was conjectured in ref. 11 that this gap is artificial and that there exists a spectral algorithm that succeeds all of the way to the detectability transition even in the sparse case. Here, we propose an algorithm based on a linear operator considerably different from the adjacency matrix or its variants: namely, a matrix that represents a walk on the directed edges of the network, with backtracking prohibited. We give strong evidence that this algorithm indeed closes the gap.The fact that this operator has better spectral properties than, for instance, the standard random walk operator, has been used in the past in the context of random matrices and random graphs (13-15). In the theory of zeta functions of graphs, it is known as the edge adjacency operator, or the Hashimoto matrix...

Phase transitions in semidefinite relaxations

“…A sequence of recent papers (ref. 7 and references therein) demonstrate that classical spectral methods-such as PCA-fail to detect the hidden partition in graphs with bounded average degree. In contrast, Fig.…”

“…In the community detection problem, several authors recently developed ingenious spectral algorithms that achieve the information theoretically optimal threshold ða − bÞ= ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2ða + bÞ p = 1 (see, e.g., refs. 7,23,24,43,44).…”

“…On the other hand, the unknown object to be estimated has often a combinatorial structure: In clustering we aim at estimating a partition of the data points (5). Network analysis tasks usually require identification of a discrete subset of nodes in a graph (6,7). Parsimonious data explanations are sought by imposing combinatorial sparsity constraints (8).…”

Percolation in real interdependent networks

“…Thus, the generic entry of the matrix M is different from zero only if the ending node of the edge i → j corresponds to the starting vertex of the edge k → , but the starting and ending nodes i and are different. This matrix is known as the non-backtracking matrix of the graph [21,22]. A trivial solution of the preceding equation is given by r = 0, which in turn leads to s = 0.…”

Improving the performance of algorithms to find communities in networks

“…We use a home-build implementation of these methods, as described in Refs. [3,4]. Actual eigenvalue computation is via the scipy sparse linear algebra package [28], which is an interface to the ARPACK library [29], which itself uses the implicitly restarted Arnoldi method for computing k eigenvalues [30].…”