Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined Clusters

Abstract

A large body of work has been devoted to defining and identifying clusters or communities in social and information networks, i.e., in graphs in which the nodes represent underlying social entities and the edges represent some sort of interaction between pairs of nodes. Most such research begins with the premise that a community or a cluster should be thought of as a set of nodes that has more and/or better connections between its members than to the remainder of the network. In this paper, we explore from a novel perspective several questions related to identifying meaningful communities in large social and information networks, and we come to several striking conclusions.Rather than defining a procedure to extract sets of nodes from a graph and then attempt to interpret these sets as a "real" communities, we employ approximation algorithms for the graph partitioning problem to characterize as a function of size the statistical and structural properties of partitions of graphs that could plausibly be interpreted as communities. In particular, we define the network community profile plot, which characterizes the "best" possible community-according to the conductance measure-over a wide range of size scales. We study over 100 large real-world networks, ranging from traditional and on-line social networks, to technological and information networks and web graphs, and ranging in size from thousands up to tens of millions of nodes.Our results suggest a significantly more refined picture of community structure in large networks than has been appreciated previously. Our observations agree with previous work on small networks, but we show that large networks have a very different structure. In particular, we observe tight communities that are barely connected to the rest of the network at very small size scales (up to ≈ 100 nodes); and communities of size scale beyond ≈ 100 nodes gradually "blend into" the expanderlike core of the network and thus become less "community-like," with a roughly inverse relationship between community size and optimal community quality. This observation agrees well with the so-called Dunbar number which gives a limit to the size of a well-functioning community.However, this behavior is not explained, even at a qualitative level, by any of the commonly-used network generation models. Moreover, it is exactly the opposite of what one would expect based on intuition from expander graphs, low-dimensional or manifold-like graphs, and from small social networks that have served as testbeds of community detection algorithms. The relatively gradual increase of the network community profile plot as a function of increasing community size depends in a subtle manner on the way in which local clustering information is propagated from smaller to larger size scales in the network. We have found that a generative graph model, in which new edges are added via an iterative "forest fire" burning process, is able to produce graphs exhibiting a network community profile plot similar to what we observe i...

Knowledge and networks: An experimental test of how network knowledge affects coordination

“…The number of distinct network structures possible with just 16 nodes is 6.4 × 10 22 . To complicate matters, human networks-both organic and designed-often exhibit peculiar properties such as a scale-free degree distribution (see, for example, Ebel et al, 2002), and at large scales may have partitioning characteristics that do not match any of the common models for random network generation (Leskovec et al, 2009). Thus there are no clear criteria for generating the most representative or relevant sample of networks from the vast variety of possible networks.…”

Scalable graph exploration and visualization: Sensemaking challenges and opportunities

“…Global approaches, best characterized by Shneiderman's mantra "overview, zoom & filter, details-on-demand" pattern in visual information seeking [42], have conventionally received much attention and have worked well for numerous kinds of data in many domains [43], [44], [45], [46], [47], [48], [49], [50], [42]. However, in this big data era, top-down approaches that focus on providing overviews of global information landscapes face significant challenges when applied to graphs with millions or billions of nodes and edges [49], [50]: graph overviews for large graphs are time-consuming to generate [8], [7]; the seminal work on graph clustering by Leskovec & Faloutsos [9] suggests there are simply no perfect overviews (i.e., no single best way to partition graphs into smaller communities), a view echoed by sensemaking literature in that people may have very different mental representations of information depending on their individual goals and prior experiences [51].…”

“…Filtering [1], [2], [3] Sampling [4], [5], [6] Partitioning [7], [8], [9], [10] Clustering [11], [12], [13], [3], [14], [15] Local View Free Discovery Exploration [16], [17], [14], [18], [3], [15], [19], [20], [21] Network Motifs [22], [23], [24], [25], [26] Targeted Discovery Pattern Matching [27], [28], [29], [30], [31] Navigation [32], [33], [34], [35], [36], [19], [37] Fig. 1.…”

“…A common approach to creating an overview graph is to use partitioning methods on the graph and visualize the partitions [7], [8], [9]. Large graph partitioning is a computationally expensive step and in cases of scale-free and near scalefree networks the partitions may be exceedingly poor [9].…”

“…Large graph partitioning is a computationally expensive step and in cases of scale-free and near scalefree networks the partitions may be exceedingly poor [9]. A recent approach called PULP was designed to partition small-world networks and has demonstrated improvements over conventional spectral methods as well as METIS [10].…”