Evaluating accuracy of community detection using the relative normalized mutual information

Abstract

The Normalized Mutual Information (NMI) has been widely used to evaluate the accuracy of community detection algorithms. However in this article we show that the NMI is seriously affected by systematic errors due to finite size of networks, and may give a wrong estimate of performance of algorithms in some cases. We give a simple theory to the finite-size effect of NMI and test our theory numerically. Then we propose a new metric for the accuracy of community detection, namely the relative Normalized Mutual Information (rNMI), which considers statistical significance of the NMI by comparing it with the expected NMI of random partitions. Our numerical experiments show that the rNMI overcomes the finite-size effect of the NMI.Detection of community structures, which asks to group nodes in a network into groups, is a key problem in network science, computer science, sociology and biology. Many algorithms have been proposed for this problem, see ref.[1] for a review. However on a given network, different algorithms usually give different results. Thus evaluating performance of these algorithms and finding the best ones are of great importance.Usually the evaluations are performed on benchmark networks each of which has a reference partition. These benchmarks include networks generated by generative models, like Stochastic Block Model [2] and LFR model [3], with a planted partition as the reference partition; and some real-world networks, like the famous Karate Club network [4] and the Political Blog network [5], with a partition annotated by domain experts as the reference partition. The accuracy of a community detection algorithm is usually represented using similarity between the reference partition and partition found by the algorithm -the larger similarity, the better performance the algorithm has on the benchmark.Without losing generality, in what follows we call the reference partition A and the detected partition B, and our task is to study the measure of similarity between partition A and partition B.When the number of groups are identical, q A = q B = q, the similarity can be easily defined by the overlap, which is the number of identical group labels in A and B maximized over all possible permutations:

Community detection in networks: A user guide

“…However, the measure is sensitive to the number of clusters q Y of the detected partition, and may attain larger values the larger q Y , even though more refined partitions are not necessarily closer to the planted one. This may give wrong perceptions about the relative performance of algorithms (Zhang, 2015). A more promising measure, proposed by Meilȃ (Meilȃ, 2007) is the variation of information (VI)…”

Correction for Closeness: Adjusting Normalized Mutual Information Measure for Clustering Comparison

“…Experiments on the results of some well-known community detection algorithms showed to be consistent with the results found by Orman et al (2012), and that, among the three measures, the modified NMI is able to assess the similarity between a reference and predicted clustering in terms of both membership and topological properties. Zhang (2015) performed an analytic and experimental study showing that NMI has a systematic bias when evaluating methods obtaining different numbers of groups, because it prefers algorithms obtaining a large number of partitions. The author shows that this is due to the finite size effect of entropy, which is different when considering an infinite or finite number of nodes.…”

Community detection by laser network dynamics

“…NMI is often used for evaluating community detection algorithms by comparing the partition embedded in the benchmark graph with the partition obtained using algorithms [36]. However, NMI is known to be biased for a large number of divisions because of the finite size effect [38]. In particular, the partition in which the all nodes belong to the different groups represents a high NMI value compared to the case where all nodes belong to the same group, although both of those partitions are trivial cases for community detection.…”

“…In particular, the partition in which the all nodes belong to the different groups represents a high NMI value compared to the case where all nodes belong to the same group, although both of those partitions are trivial cases for community detection. Relative NMI (rNMI) has been proposed to consider this finite size effect [38]. The rNMI is defined as…”