Upscaling human activity data: A statistical ecology approach

Abstract

Big data require new techniques to handle the information they come with. Here we consider four datasets (email communication, Twitter posts, Wikipedia articles and Gutenberg books) and propose a novel statistical framework to predict global statistics from random samples. More precisely, we infer the number of senders, hashtags and words of the whole dataset and how their abundances (i.e. the popularity of a hashtag) change through scales from a small sample of sent emails per sender, posts per hashtag and word occurrences. Our approach is grounded on statistical ecology as we map inference of human activities into the unseen species problem in biodiversity. Our findings may have applications to resource management in emails, collective attention monitoring in Twitter and language learning process in word databases.

Deviation from neutral species abundance distributions unveils geographical differences in the structure of diatom communities

“…In community ecology, species abundance distribution (SAD)the distribution of individuals within a species in a given communityhas been a cornerstone of research (11,14), which can potentially shed light on the study of plankton microbial communities and their structure. Studying SADs not only allows for a characterization of ecological communities but also provides critical insights that enable species number estimations at larger scales beyond direct measurement, accomplished by inferring SAD distribution forms (15)(16)(17)(18)(19). Notably, the functional form of SADs has shown consistency across diverse ecosystems.…”