Hypothesis Testing for Automated Community Detection in Networks

Abstract

Community detection in networks is a key exploratory tool with applications in a diverse set of areas, ranging from finding communities in social and biological networks to identifying link farms in the World Wide Web. The problem of finding communities or clusters in a network has received much attention from statistics, physics and computer science. However, most clustering algorithms assume knowledge of the number of clusters k. In this paper we propose to automatically determine k in a graph generated from a Stochastic Blockmodel. Our main contribution is twofold; first, we theoretically establish the limiting distribution of the principal eigenvalue of the suitably centered and scaled adjacency matrix, and use that distribution for our hypothesis test. Secondly, we use this test to design a recursive bipartitioning algorithm. Using quantifiable classification tasks on real world networks with ground truth, we show that our algorithm outperforms existing probabilistic models for learning overlapping clusters, and on unlabeled networks, we show that we uncover nested community structure.

A Block Model for Node Popularity in Networks with Community Structure

“…We now report the performance of PABM modularity and DCBM modularity in analysing three networks: the political blogs network, the British MPs Twitter network and the DBLP network. The political blogs network (Adamic and Glance, 2005) has been well studied in the networks literature in general and particularly in connection with the DCBM-starting from the original DCBM paper by Karrer and Newman (2011) to Zhao et al (2012), Amini et al (2013), Jin (2015), Bickel and Sarkar (2016) and Le et al (2016), to name a few. Following the usual practice, we extract the largest connected component and treat it as a simple graph with 1222 nodes and 16 714 edges, and K = 2 communities.…”

“…Given that the main advantage of the DCBM over the classical SBM is that the former allows flexible modelling of expected degrees, this strong structural restriction forcing expected degrees to take values in a finite set appears somewhat counterintuitive. As pointed out by Bickel and Sarkar (2016), this structural assumption of Zhao et al (2012) effectively characterizes the DCBM as an SBM with a larger number of communities. A similar comment was made earlier by Amini et al (2013) This structural restriction has also been remarked on by Jin (2015).…”

Evaluating Overfit and Underfit in Models of Network Community Structure

“…Ref. [72] proposes a sequential hypothesis testing approach to choose k. At each step, it tests whether to bipartition a graph or not. To this end, the authors derive and utilize the asymptotic null distribution for Erdős-Rényi random graphs.…”

Context‐aided analysis of community evolution in networks

“…Block models focus on the stochastic equivalence, keeping the same linking probabilities for nodes clustered in the same block. There are a number of different approaches to fitting a stochastic block model, from Markov chain Monte Carlo (MCMC) [18,30] to clustering after random projections [43] to recursive bipartitioning [5]. A weighted version of SBM was proposed by Aicher et al [1], together with a variational Bayes inference algorithm and a MATLAB implementation.…”