Tracking evolving communities in large linked networks

Abstract

We are interested in tracking changes in large-scale data by periodically creating an agglomerative clustering and examining the evolution of clusters (communities) over time. We examine a large real-world data set: the NEC CiteSeer database, a linked network of >250,000 papers. Tracking changes over time requires a clustering algorithm that produces clusters stable under small perturbations of the input data. However, small perturbations of the CiteSeer data lead to significant changes to most of the clusters. One reason for this is that the order in which papers within communities are combined is somewhat arbitrary. However, certain subsets of papers, called natural communities, correspond to real structure in the CiteSeer database and thus appear in any clustering. By identifying the subset of clusters that remain stable under multiple clustering runs, we get the set of natural communities that we can track over time. We demonstrate that such natural communities allow us to identify emerging communities and track temporal changes in the underlying structure of our network data. E mergent properties of large linked networks have recently become the focus of intense study. This research is driven by the increasing complexity and importance of large networks, such as the World Wide Web, the electricity grid, and large social networks that capture relationships between individuals. Realworld networks generally exhibit properties that lie somewhere in-between those of highly structured networks and purely random ones (1-4). So far, most research has focused on using static properties, such as the connectivity of the nodes in the network and the average distance between two nodes, to explain the complex structure. However, these networks generally evolve over time and so temporal characteristics are a key source of interest. Our goal in this paper is to provide techniques for the study of the evolution of large linked networks.In our approach, we use agglomerative clusterings of the linked network. By clustering the network at different points in time, we study its temporal evolution. This approach places a new burden on the underlying clustering method. Clustering methods can be surprisingly sensitive to minor changes of the input data. For obtaining a static view of the higher-level structure of the data, such instabilities may be acceptable because the resulting hierarchy often already reveals interesting structure. However, in tracking changes over time, we need to be able to find corresponding communities in clusterings taken from the data at different points in time. If the clusterings are very sensitive to small perturbations of the input data, distinguishing between ''real'' changes versus ''accidental'' changes in the higher-level structure becomes difficult, if not impossible. In the clusterings of our linked network data, we found there are a large number of relatively random clusters that do not correspond to real community structures. These random clusters obscure the real temporal changes. Fortunately...

Identifying Mycobacterium tuberculosis complex strain families using spoligotypes

“…The number of families present in the spoligotype data and the probability distribution for each of them were estimated using the Monte Carlo cross-validation (MCCV) technique, which was developed to extract as much information from the data as possible, without any prior knowledge (Smyth, 1996). We used the stability, or average best match (Hopcroft et al, 2004), and the log-likelihood to choose a final mixture model. The results were compared to the families that have been identified using the prototypes extracted from the SpolDB3 database (Filliol et al, 2002).…”

“…We chose a final mixture model based on the total stability (Hopcroft et al, 2004) and the total loglikelihood. We call the stability of a set, or putative family, of spoligotypes the average best match between this set and the sets identified using other models.…”

“…We call the stability of a set, or putative family, of spoligotypes the average best match between this set and the sets identified using other models. For any two sets and , the match (between 0 and 1) is defined as follows (Hopcroft et al, 2004):…”

Detecting Community Structure in Dynamic Social Networks Using the Concept of Leadership

“…Its basic idea is to apply a static clustering method to each snapshot independently and capture the evolution of the communities by comparing the clustering of the consecutive snapshots. Based on this approach, Hopcroft et al [30] were of the first authors who applied a static clustering on each snapshots and they tracked communities over time by the clusters similarities over consecutive timesteps. They showed that even small perturbations in the graph could lead significant changes in the structure of the detected communities.…”

Modularity-Driven Clustering of Dynamic Graphs

“…Apart from that, there have been attempts to track communities over time and interpret their evolution, using static snapshots of the network, e.g. [9,10], besides an array of case studies. In [11] a parameter-based dynamic graph clustering method is proposed which allows user exploration.…”