Exploring Limits to Prediction in Complex Social Systems

Abstract

How predictable is success in complex social systems? In spite of a recent profusion of prediction studies that exploit online social and information network data, this question remains unanswered, in part because it has not been adequately specified. In this paper we attempt to clarify the question by presenting a simple stylized model of success that attributes prediction error to one of two generic sources: insufficiency of available data and/or models on the one hand; and inherent unpredictability of complex social systems on the other. We then use this model to motivate an illustrative empirical study of information cascade size prediction on Twitter. Despite an unprecedented volume of information about users, content, and past performance, our best performing models can explain less than half of the variance in cascade sizes. In turn, this result suggests that even with unlimited data predictive performance would be bounded well below deterministic accuracy. Finally, we explore this potential bound theoretically using simulations of a diffusion process on a random scale free network similar to Twitter. We show that although higher predictive power is possible in theory, such performance requires a homogeneous system and perfect ex-ante knowledge of it: even a small degree of uncertainty in estimating product quality or slight variation in quality across products leads to substantially more restrictive bounds on predictability. We conclude that realistic bounds on predictive accuracy are not dissimilar from those we have obtained empirically, and that such bounds for other complex social systems for which data is more difficult to obtain are likely even lower.

Prediction and explanation in social systems

“…The reason for this incoherence is that prediction results depend on many of the same "researcher degrees of freedom" that lead to false positives in traditional hypothesis testing (3). For example, consider the question of predicting the size of online diffusion "cascades" to understand how information spreads through social networks, a topic of considerable recent interest (6,7,10,11). Although seemingly unambiguous, this question can be answered only after it has first been translated into a specific computational task, which in turn requires the researcher to make a series of subjective choices, including the selection of the task, data set, model, and performance metric.…”

“…In addition to holding the data set fixed, for simplicity, we also restricted our analysis to a single choice of model, reported in (11), that predicts cascade size as a linear function of the average past performance of the "seed" individual (i.e., the one who initiated the cascade). Even with the data source and model held fixed, Fig.…”

“…In skill world, for example, if one could hypothetically measure skill, then in principle it would be Hofman http://science.sciencemag.org/ possible to predict success with almost perfect precision. In luck world, in contrast, even a "perfect" predictor would yield mediocre performance, no better than predicting that all items will experience the same (i.e., average) level of success (11). It follows, therefore, that the more that outcomes are determined by extrinsic random factors, the lower the theoretical best performance that can be attained by any model.…”

Luck is Hard to Beat

“…Indeed, Martin et al [15] showed that realistic bounds on predicting outcomes in social systems imposes drastic limits on what the best performing models can deliver. And yet, accurate prediction is a holy grail avidly sought in nancial markets [8], sports [7], arts and entertainment award events [13], and politics [23].…”

“…According to Blastland and Dilnot [1], the teams favored by bettors win half the time in soccer, 60% of the time in baseball, and 70% of the time in football and in basketball. Despite the large amount of money involved, there are no algorithms capable of producing accurate predictions and there is some evidence they will never be found [15].…”