Estimation and Prediction for Stochastic Blockstructures

Nowicki, Krzysztof; Snijders, Tom A. B.

doi:10.1198/016214501753208735

Cited by 943 publications

(922 citation statements)

References 18 publications

Supporting

Mentioning

913

Contrasting

Unclassified

Order By: Relevance

“…Block modelling [63][64][65][66][67] is in effect a form of statistical inference for networks. In the same way that we can gain some understanding from conventional numerical data by fitting, say, a straight line through data points, so we can gain understanding of the structure of networks by fitting them to a statistical network model.…”

Section: Block Modelsmentioning

confidence: 99%

Communities, modules and large-scale structure in networks

2011

View full text Add to dashboard Cite

Networks, also called graphs by mathematicians, provide a useful abstraction of the structure of many complex systems, ranging from social systems and computer networks to biological networks and the state spaces of physical systems. In the past decade there have been significant advances in experiments to determine the topological structure of networked systems, but there remain substantial challenges in extracting scientific understanding from the large quantities of data produced by the experiments. A variety of basic measures and metrics are available that can tell us about small-scale structure in networks, such as correlations, connections and recurrent patterns, but it is considerably more difficult to quantify structure on medium and large scales, to understand the 'big picture'. Important progress has been made, however, within the past few years, a selection of which is reviewed here.A network is, in its simplest form, a collection of dots joined together in pairs by lines (Fig. 1). In the jargon of the field, a dot is called a 'node' or 'vertex' (plural 'vertices') and a line is called an 'edge'. Networks are used in many branches of science as a way to represent the patterns of connections between the components of complex systems [1][2][3][4][5][6] . Examples include the Internet 7,8 , in which the nodes are computers and the edges are data connections such as optical-fibre cables, food webs in biology 9,10 , in which the nodes are species in an ecosystem and the edges represent predator-prey interactions, and social networks 11,12 , in which the nodes are people and the edges represent any of a variety of different types of social interaction including friendship, collaboration, business relationships or others.In the past decade there has been a surge of interest in both empirical studies of networks 13 and development of mathematical and computational tools for extracting insight from network data 1-6 . One common approach to the study of networks is to focus on the properties of individual nodes or small groups of nodes, asking questions such as, 'Which is the most important node in this network?' or 'Which are the strongest connections?' Such approaches, however, tell us little about large-scale network structure. It is this large-scale structure that is the topic of this paper.The best-studied form of large-scale structure in networks is modular or community structure 14,15 . A community, in this context, is a dense subnetwork within a larger network, such as a close-knit group of friends in a social network or a group of interlinked web pages on the World Wide Web (Fig. 1). Although communities are not the only interesting form of large-scale structure-there are others that we will come to-they serve as a good illustration of the nature and scope of present research in this area and will be our primary focus.Communities are of interest for a number of reasons. They have intrinsic interest because they may correspond to functional units within a networked system, an example of the kind of link...

show abstract

Section: Block Modelsmentioning

confidence: 99%

Communities, modules and large-scale structure in networks

2011

View full text Add to dashboard Cite

show abstract

“…Classical community membership models, like the stochastic blockmodel (5,6,27), assume that each node belongs to just one community. Such models cannot capture that a particular node's links might be explained by its membership in several overlapping groups, a property that is essential when analyzing realworld networks.…”

Section: The Model and Algorithmmentioning

confidence: 99%

“…network analysis | Bayesian statistics | massive data C ommunity detection algorithms (1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17) analyze networks to find groups of densely connected nodes. These algorithms have become vital to data-driven methods for understanding and exploring network data such as social networks (4), citation networks (18), communication networks (19), and networks induced by scientific observation [e.g., gene regulation networks (20)].…”

mentioning

confidence: 99%

“…The first is that many existing community detection algorithms assume that each node belongs to a single community (1,(3)(4)(5)(6)(7)(14)(15)(16). In real-world networks, each node will likely belong to multiple communities and its connections will reflect these multiple memberships (2,(8)(9)(10)(11)(12)(13)17).…”

mentioning

confidence: 99%

“…Many community detection algorithms iteratively analyze each pair of nodes, regardless of whether the nodes in the pair are connected in the network (5,6,10). Consequently, these algorithms run in time squared in the number of nodes, which makes analyzing massive networks computationally intractable.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Efficient discovery of overlapping communities in massive networks

Gopalan

Blei

2013

Proc. Natl. Acad. Sci. U.S.A.

217

162

View full text Add to dashboard Cite

Detecting overlapping communities is essential to analyzing and exploring natural networks such as social networks, biological networks, and citation networks. However, most existing approaches do not scale to the size of networks that we regularly observe in the real world. In this paper, we develop a scalable approach to community detection that discovers overlapping communities in massive realworld networks. Our approach is based on a Bayesian model of networks that allows nodes to participate in multiple communities, and a corresponding algorithm that naturally interleaves subsampling from the network and updating an estimate of its communities. We demonstrate how we can discover the hidden community structure of several real-world networks, including 3.7 million US patents, 575,000 physics articles from the arXiv preprint server, and 875,000 connected Web pages from the Internet. Furthermore, we demonstrate on large simulated networks that our algorithm accurately discovers the true community structure. This paper opens the door to using sophisticated statistical models to analyze massive networks. Community detection is important for both exploring a network and predicting connections that are not yet observed. For example, by finding the communities in a large citation graph of scientific articles, we can make hypotheses about the fields and subfields that they contain. By finding communities in a large social network, we can more easily make predictions to individual members about who they might be friends with but are not yet connected to.In this paper, we develop an algorithm that discovers communities in modern real-world networks. The challenge is that real-world networks are massive-they can contain hundreds of thousands or even millions of nodes. We will examine a network of scientific articles that contains 575,000 articles, a network of connected Web pages that contains 875,000 pages, and a network of US patents that contains 3,700,000 patents. Most approaches to community detection cannot handle data at this scale.There are two fundamental difficulties to detecting communities in such networks. The first is that many existing community detection algorithms assume that each node belongs to a single community (1,(3)(4)(5)(6)(7)(14)(15)(16). In real-world networks, each node will likely belong to multiple communities and its connections will reflect these multiple memberships (2,(8)(9)(10)(11)(12)(13)17). For example, in a large social network, a member may be connected to coworkers, friends from school, and neighbors. We need algorithms that discover overlapping communities to capture the heterogeneity of each node's connections.The second difficulty is that existing algorithms are too slow. Many community detection algorithms iteratively analyze each pair of nodes, regardless of whether the nodes in the pair are connected in the network (5, 6, 10). Consequently, these algorithms run in time squared in the number of nodes, which makes analyzing massive networks computationally intractable. Other a...

show abstract