Toward Cumulative Cognitive Science: A Comparison of Meta-Analysis, Mega-Analysis, and Hybrid Approaches

Koile, Ezequiel; Cristià, Alejandrina

doi:10.1162/opmi_a_00048

Cited by 11 publications

(10 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their deeper understanding is a long-term task for the discipline that can be greatly facilitated with the methods of cumulative open science. As researchers generate more data in in situ spatial cognition studies (for various research questions), assumptions about the existence of systematic biases could be formally tested in mega-analyses (Koile and Cristia 2021 )—an alternative to meta-analysis that uses unaggregated raw data from past studies. However, as for now too little datasets in the field are open.…”

Section: Discussionmentioning

confidence: 99%

Route effects in city-based survey knowledge estimates

2023

View full text Add to dashboard Cite

When studying wayfinding in urban environments, researchers are often interested in obtaining measures of participants’ survey knowledge, i.e., their estimate of distant locations relative to other places. Previous work showed that distance estimations are consistently biased when no direct route is available to the queried target or when participants follow a detour. Here we investigated whether a corresponding bias is manifested in two other popular measures of survey knowledge: a pointing task and a sketchmapping task. The aim of this study was to investigate whether there is a systematic bias in pointing/sketchmapping performance associated with the preferred route choice in an applied urban setting. The results were mixed. We found moderate evidence for the presence of a systematic bias, but only for a subset of urban locations. When two plausible routes to the target were available, survey knowledge estimates were significantly biased in the direction of the route chosen by the participant. When only one plausible route was available, we did not find a statistically significant pattern. The results may have methodological implications for spatial cognition studies in applied urban settings that might be obtaining systematically biased survey knowledge estimates at some urban locations. Researchers should be aware that the choice of urban locations from which pointing and sketchmapping are performed might systematically distort the results, in particular when two plausible but diverging routes to the target are visible from the location.

show abstract

Section: Discussionmentioning

confidence: 99%

Route effects in city-based survey knowledge estimates

2023

View full text Add to dashboard Cite

show abstract

“…Whereas developmental psychologists were constrained to collecting data in laboratory settings, our toolkit has expanded to include online testing (Chuey et al, 2021;Scott & Schulz, 2017), large-scale replication (Frank et al, 2017), and meta-and mega-analysis of existing data (Koile & Cristia, 2021;Tsuji et al, 2017). Developmental psychology has also partnered with the field of computational cognitive science to formalize theories about knowledge and learning, and design tests of those theories (Lake et al, 2017;Shu et al, 2021;Smith et al, 2019;Tenenbaum et al, 2011).…”

Section: Discussionmentioning

confidence: 99%

iCatcher+: Robust and automated annotation of infant's and young children's gaze direction from videos collected in laboratory, field, and online studies

Erel¹,

Shannon²,

Scott

et al. 2022

Preprint

View full text Add to dashboard Cite

Technological advances in psychological research have enabled large-scale studies of human behavior and streamlined pipelines for automatic processing of data. However, studies of infants and children have not fully reaped these benefits, because the behaviors of interest, such as gaze duration and direction, even when collected online, still have to be extracted from video through a laborious process of manual annotation. Recent advances in computer vision raise the possibility of automated annotation of video data. In this paper, we built on a system for automatic gaze annotation in human infants, iCatcher (Erel et al., 2022), by engineering improvements, and then training and testing the system (hereafter, iCatcher+) on two datasets with substantial video and participant variability (214 videos collected in lab and mobile testing centers, and 265 videos collected via webcams in homes; infants and children aged 4 months to 3.5 years). We found that when trained on each of these video datasets, iCatcher+ performed with near human-level accuracy on held out videos on distinguishing “LEFT” and “RIGHT” looking behavior, and “ON” versus “OFF” looking behavior, across both datasets. This high performance was achieved at the level of individual frames, experimental trials, and study videos, held across participant demographics (e.g., age, race/ethnicity) and video characteristics (e.g., resolution, luminance), and generalized to a third, entirely held-out dataset. We close by discussing next steps required to fully automate the lifecycle of online infant and child behavioral studies, representing a key step towards enabling rapid, high-powered developmental research.

show abstract

“…In this case, the researcher will need additional modeling steps, for instance using the body of previous literature in a rawer form. Tools at this stage include Individual Participant Data (IPD) meta-analyses (Riley et al, 2020;Verhage et al, 2020); mega-analyses (Sung et al, 2014); and hybrid meta-and mega-analyses or pseudo-IPD meta-analyses (Koile and Cristia, 2021;Papadimitropoulou et al, 2019). All points discussed here apply to these different formats of quantitatively aggregating evidence, but for simplicity we limit our considerations to summarizing group-level data.…”

Section: How Data Are Currently Used In Theory Evaluationmentioning

confidence: 99%

A meta-analytic approach to evaluating the explanatory adequacy of theories

Cristià

Tsuji

Bergmann

2022

View full text Add to dashboard Cite

How can data be used to check theories' explanatory adequacy? The two traditional and most widespread approaches use single studies and non-systematic narrative reviews to evaluate theories' explanatory adequacy; more recently, large-scale replications entered the picture. We argue here that none of these approaches fits in with cumulative science tenets. We propose instead Community-Augmented Meta-Analyses (CAMAs), which, like meta-analyses and systematic reviews, are built using all available data; like meta-analyses but not systematic reviews, can rely on sound statistical practices to model methodological effects; and like no other approach, are broad-scoped, cumulative and open. We explain how CAMAs entail a conceptual shift from meta-analyses and systematic reviews, a shift that is useful when evaluating theories' explanatory adequacy. We then provide step-by-step recommendations for how to implement this approach - and what it means when one cannot. This leads us to conclude that CAMAs highlight areas of uncertainty better than alternative approaches that bring data to bear on theory evaluation, and can trigger a much needed shift towards a cumulative mindset with respect to both theory and data, leading us to do and view experiments and narrative reviews differently.

show abstract

Toward Cumulative Cognitive Science: A Comparison of Meta-Analysis, Mega-Analysis, and Hybrid Approaches

Cited by 11 publications

References 29 publications

Route effects in city-based survey knowledge estimates

Route effects in city-based survey knowledge estimates

iCatcher+: Robust and automated annotation of infant's and young children's gaze direction from videos collected in laboratory, field, and online studies

A meta-analytic approach to evaluating the explanatory adequacy of theories

Contact Info

Product

Resources

About