An Agenda for Purely Confirmatory Research

Wagenmakers, Eric‐Jan; Wetzels, Ruud; Borsboom, Denny; Maas, Han L. J. van der; Kievit, Rogier A.

doi:10.1177/1745691612463078

Cited by 916 publications

(950 citation statements)

References 51 publications

Supporting

Mentioning

893

Contrasting

Unclassified

Order By: Relevance

“…Here we demonstrate how the SBF procedure would have been applied to the empirical example. (For other applications of the SBF design to real data, see Matzke et al, 2015, Wagenmakers et al, 2012 Method and participants. We employed a Sequential Bayes Factor design, where Bayes factors (BFs) are computed repeatedly during data collection, until the BF exceeds an a priori defined grade of evidence.…”

Section: Sequential Bayes Factors: An Alternative Hypothesis Testing mentioning

confidence: 99%

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Schönbrodt

Wagenmakers

Zehetleitner

et al. 2017

Psychological Methods

Self Cite

422

466

View full text Add to dashboard Cite

Unplanned optional stopping rules have been criticized for inflating Type I error rates under the null hypothesis significance testing (NHST) paradigm. Despite these criticisms this research practice is not uncommon, probably as it appeals to researcher's intuition to collect more data in order to push an indecisive result into a decisive region. In this contribution we investigate the properties of a procedure for Bayesian hypothesis testing that allows optional stopping with unlimited multiple testing, even after each participant. In this procedure, which we call Sequential Bayes Factors (SBF), Bayes factors are computed until an a priori defined level of evidence is reached. This allows flexible sampling plans and is not dependent upon correct e↵ect size guesses in an a priori power analysis. We investigated the long-term rate of misleading evidence, the average expected sample sizes, and the biasedness of e↵ect size estimates when an SBF design is applied to a test of mean di↵erences between two groups. Compared to optimal NHST, the SBF design typically needs 50% to 70% smaller samples to reach a conclusion about the presence of an e↵ect, while having the same or lower long-term rate of wrong inference.

show abstract

Section: Sequential Bayes Factors: An Alternative Hypothesis Testing mentioning

confidence: 99%

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Schönbrodt

Wagenmakers

Zehetleitner

et al. 2017

Psychological Methods

Self Cite

422

466

View full text Add to dashboard Cite

show abstract

“…To advance in this path, research practices that are currently uncommon, such as the use of pre-registration for confirmatory research (Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012) and more transparent practices of data collection, analyses, and publication (Nosek & BarAnan, 2012;Nosek et al, 2015), may help producers of this literature gain a firmer grip on infant experimental data.…”

Section: Broader Implications For Infant Cognition Researchmentioning

confidence: 99%

Can infants learn phonology in the lab? A meta-analytic answer

Cristià

2018

Cognition

View full text Add to dashboard Cite

Two of the key tasks facing the language-learning infant lie at the level of phonology: establishing which sounds are contrastive in the native inventory, and determining what their possible syllabic positions and permissible combinations (phonotactics) are. In 2002-2003, two theoretical proposals, one bearing on how infants can learn sounds (Maye, Werker, & Gerken, 2002) and the other on phonotactics (K. E. Chambers, Onishi, & Fisher, 2003), were put forward on the pages of Cognition, each supported by two laboratory experiments, wherein a group of infants was briefly exposed to a set of pseudo-words, and plausible phonological generalizations were tested in a subsequent phase. These two papers have received considerable attention from the general scientific community, and inspired a flurry of follow-up work. In the context of questions regarding the replicability of psychological science, the present work uses a meta-analytic approach to appraise extant empirical evidence for infant phonological learning in the laboratory. It is found that neither seminal finding (on learning sounds and learning phonotactics) holds up when close methodological replications are integrated, although less close methodological replications do provide some evidence in favor of the sound learning strand of work. Implications for authors and readers of this literature are drawn out. It would be desirable that additional mechanisms for phonological learning be explored, and that future infant laboratory work should employ paradigms that rely on constrained and unambiguous links between experimental exposure and measured infant behavior.

show abstract

“…Fortunately, the problem of flexibility and potential for p-hacking that bedevil the fertility effect literature can be prevented in a simple and decisive fashion. What is needed is for investigators to conduct future studies using preregistration (Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012) of their definition of fertile periods and all other analytical methods, including plans for excluding subjects and potential moderator variables to be included in analyses. Preregistration is now possible and convenient through the Open Science Framework (https://OSF.io).…”

Section: Looking Forward To Better Research: How To Prevent Such Probmentioning

confidence: 99%

Elastic analysis procedures: An incurable (but preventable) problem in the fertility effect literature. Comment on Gildersleeve, Haselton, and Fales (2014).

Harris

Pashler

Mickes

2014

Psychological Bulletin

View full text Add to dashboard Cite

Gildersleeve, Haselton, and Fales (2014) presented a meta-analysis of the effects of fertility on mate preferences in women. Research in this area has categorized fertility using a great variety of methods, chiefly based on self-reported cycle length and time since last menses. We argue that this literature is particularly prone to hidden experimenter degrees of freedom. Studies vary greatly in the duration and timing of windows used to define fertile versus nonfertile phases, criteria for excluding subjects, and the choice of what moderator variables to include, as well as other variables. These issues raise the concern that many or perhaps all results may have been created by exploitation of unacknowledged degrees of freedom ("p-hacking"). Gildersleeve et al. sought to dismiss such concerns, but we contend that their arguments rest upon statistical and logical errors. The possibility that positive results in this literature may have been created, or at least greatly amplified, by p-hacking receives additional support from the fact that recent attempts at exact replication of fertility results have mostly failed. Our concerns are also supported by findings of another recent review of the literature (Wood, Kressel, Joshi, & Louie, 2014). We conclude on a positive note, arguing that if fertility-effect researchers take advantage of the rapidly emerging opportunities for study preregistration, the validity of this literature can be rapidly clarified.

show abstract

An Agenda for Purely Confirmatory Research

Cited by 916 publications

References 51 publications

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Can infants learn phonology in the lab? A meta-analytic answer

Elastic analysis procedures: An incurable (but preventable) problem in the fertility effect literature. Comment on Gildersleeve, Haselton, and Fales (2014).

Contact Info

Product

Resources

About