Impact of regularization on spectral clustering

Abstract

The performance of spectral clustering can be considerably improved via regularization, as demonstrated empirically in Amini et al. [2]. Here, we provide an attempt at quantifying this improvement through theoretical analysis. Under the stochastic block model (SBM), and its extensions, previous results on spectral clustering relied on the minimum degree of the graph being sufficiently large for its good performance. By examining the scenario where the regularization parameter τ is large we show that the minimum degree assumption can potentially be removed. As a special case, for an SBM with two blocks, the results require the maximum degree to be large (grow faster than log n) as opposed to the minimum degree. More importantly, we show the usefulness of regularization in situations where not all nodes belong to well-defined clusters. Our results rely on a 'bias-variance'-like trade-off that arises from understanding the concentration of the sample Laplacian and the eigen gap as a function of the regularization parameter. As a byproduct of our bounds, we propose a data-driven technique DKest (standing for estimated DavisKahan bounds) for choosing the regularization parameter. This technique is shown to work well through simulations and on a real data set.

Role of normalization in spectral clustering for stochastic blockmodels

“…These approaches ensure that, with high probability, the eigenvalues corresponding to the discriminating eigenvectors are of larger order than those corresponding to the uninteresting eigenvectors. Further analysis of regularization can be found in [14] and [20].…”

Concentration and regularization of random graphs

“…The dependence in Theorem 4.1 is not optimal, and we have not made efforts to improve it. Although it is natural to choose τ ∼ d as in Theorem 1.2, choosing τ d could also be useful [23]. Choosing τ d may be interesting as well, for then L(E A τ ) ≈ L(E A) and we obtain the concentration of L(A τ ) around the Laplacian of the expectation of the original (rather than regularized) matrix E A.…”

“…As an application of the new concentration results, we show that the regularized spectral clustering [3,23], one of the simplest most popular algorithms for community detection, can recover communities in the sparse regime. In general, spectral clustering works by computing the leading eigenvectors of either the adjacency matrix or the Laplacian or their regularized versions, and running the k-means clustering algorithm on these eigenvectors to recover the node labels.…”

“…Concentration of Laplacians of random graphs has been studied in [35,11,39,23,18]. Just like the adjacency matrix, the Laplacian is known to concentrate in the dense regime where d = Ω(log n), and it fails to concentrate in the sparse regime.…”

“…Here we will focus on the following simple way of regularization proposed in [3] and analyzed in [23,18]. Choose τ > 0 and add the same number τ /n to all entries of the adjacency matrix A, thereby replacing it with The following consequence of Theorem 1.1 shows that such regularization indeed forces Laplacian to concentrate.…”