The Meta-Science of Adult Statistical Word Segmentation: Part 1

Abstract

We report the first set of results in a multi-year project to assess the robustness – and the factors promoting robustness – of the adult statistical word segmentation literature. This includes eight total experiments replicating six different experiments. The purpose of these replications is to assess the reproducibility of reported experiments, examine the replicability of their results, and provide more accurate effect size estimates. Reproducibility was mixed, with several papers either lacking crucial details or containing errors in the description of method, making it difficult to ascertain what was done. Replicability was also mixed: although in every instance we confirmed above-chance statistical word segmentation, many theoretically important moderations of that learning failed to replicate. Moreover, learning success was generally much lower than in the original studies. In the General Discussion, we consider whether these differences are due to differences in subject populations, low power in the original studies, or some combination of these and other factors. We also consider whether these findings are likely to generalize to the broader statistical word segmentation literature.

A free and publicly available stimuli set for statistical learning experiments in language and music

“…To test whether the effects generalize beyond the stimulus items, and to account for the fact the accuracy DV is not a continuous measure, we also analysed the data with a binomial mixed effects model (based on a recent replication of Saffran et al, 1996by Hartshorne et al, 2019. Subject and item were random effects.…”

“…We also did not find evidence for segmentation with a binomial mixed-effects model, and the paired t-test was not significant in one of our samples. Notably, Hartshorne et al (2019) also did not find a significant effect with a binomial mixed-effects model in their replication of statistical learning in the language domain (with a different stimuli set). This suggests that the segmentation effect, though present with the current stimulus set, may not generalize to different items and people.…”

“…We also ran binomial mixed-effects models to address the non-scalar aspect of the dependent variable, as well as take into account participant and item variability (following Hartshorne et al, 2019). Although this analysis confirmed segmentation in the music domain, it did not confirm segmentation in the language domain.…”

“…This is not only inefficient, as researchers often recreate stimuli that already exist in other labs, but it also makes it difficult to compare statistical learning results across multiple experiments. For example, while the original studies of statistical learning in the language domain used computer generated monotone speech (Saffran, Newport & Aslin, 1996), other replications and extensions have used a range of methods to create stimuli, such as recording and manipulating naturally produced syllables (Graf Estes et al, 2007) or using IBM's Watson (Hartshorne et al, 2019).…”

“…The decisions made by researchers when creating artificial language stimuli can be conceptualized as balancing the requirements of internal versus external validity. Researchers who prioritize internal validity will lean towards computer generation with strict standardization of parameters (e.g., Hartshorne et al, 2019). This ensures a high degree of control over the physical properties of stimuli.…”