Finding the traces of behavioral and cognitive processes in big data and naturally occurring datasets

Abstract

Today, people generate and store more data than ever before as they interact with both real and virtual environments. These digital traces of behavior and cognition offer cognitive scientists and psychologists an unprecedented opportunity to test theories outside the laboratory. Despite general excitement about big data and naturally occurring datasets among researchers, three “gaps” stand in the way of their wider adoption in theory-driven research: the imagination gap, the skills gap, and the culture gap. We outline an approach to bridging these three gaps while respecting our responsibilities to the public as participants in and consumers of the resulting research. To that end, we introduce Data on the Mind (http://www.dataonthemind.org), a community-focused initiative aimed at meeting the unprecedented challenges and opportunities of theory-driven research with big data and naturally occurring datasets. We argue that big data and naturally occurring datasets are most powerfully used to supplement—not supplant—traditional experimental paradigms in order to understand human behavior and cognition, and we highlight emerging ethical issues related to the collection, sharing, and use of these powerful datasets.

Scaling up psychology via Scientific Regret Minimization

“…While it enables researchers to dissociate individual variables of interest, it can also lead to over-fixation on a specific paradigm and the small amount of variations it offers in contrast to more broadly sampling the space of experiments relevant to the behavior of interest. As a result, several researchers have started to call for a shift towards mining massive online datasets via crowdsourced experiments (Griffiths, 2015;Jones, 2016;Goldstone & Lupyan, 2016;McAbee, Landis, & Burke, 2017;Paxton & Griffiths, 2017;Schulz et al, 2019) because the scale offered by the internet enables scientists to quickly evaluate thousands of hypotheses on millions of participants.…”

Uncertainty Expressions in Accounting: Critical Issues and Recommendations

“…Second, research methods on uncertainty expressions are still primarily based on surveys or laboratory experiments. The development of databases used in accounting research, and particularly the increasing availability of 'big data', offer a promising research opportunity to use naturally occurring datasets (Paxton & Griffiths, 2017;Teoh, 2018), such as Google Trends, to trace people's perceptions of different uncertainty expressions over time. Also, ideally, this could offer a way to map uncertainty-expression scales across different culture and language contexts.…”

Social learning strategies for matters of taste

“…Although other established datasets used in the recommender literature (e.g., MovieLens, LastFM, or Netflix) may contain millions of evaluations, to the best of our knowledge, even the most popular items in those datasets have been evaluated by only a small fraction of all users. Using the rich Jester dataset allowed us to study the principles underlying the success of different social learning strategies at a very large scale [46,75].…”

“…Jester dataset allowed us to study the principles underlying the success of different social learning strategies at a very large scale [36,61].…”

“…We randomly sampled uniformly without replacement 500 training sets of size m = 25 from the training space D. The training space was thus sufficiently bigger that the training sets D = 3m. Furthermore, because there are 75 25 possible samples, the probability of sampling duplicate training sets was very small. We trained the models with each of these 500 samples.…”