Detecting and avoiding likely false‐positive findings – a practical guide

Forstmeier, Wolfgang; Wagenmakers, Eric‐Jan; Parker, Timothy

doi:10.1111/brv.12315

Cited by 361 publications

(429 citation statements)

References 116 publications

Supporting

Mentioning

400

Contrasting

Unclassified

Order By: Relevance

“…We then selected fixed effects by hierarchically removing the effects whose 95% CI encompassed zero, starting with the effects with a posterior mode closer to zero. Because this stepwise method may increase the risk of type-I error (Mundry and Nunn, 2009;Forstmeier et al, 2017), we compared the 95% CI in the selected fixed structure to those obtained from the full models, but retained estimates from the selected models. Removing the genetic correlation effect after selecting the fixed effects did not change the results (results not detailed, but see Appendix S5 for the full model output).…”

Section: Implementation Of Modelsmentioning

confidence: 99%

Heterospecific Nest Site Copying Behavior in a Wild Bird: Assessing the Influence of Genetics and Past Experience on a Joint Breeding Phenotype

et al. 2018

View full text Add to dashboard Cite

Breeding site selection is often a joint decision of pair members in species with biparental care and the experience of both pair members may influence the use of information for site selection. Nevertheless, quantitative genetics of joint information use for site selection remains unexplored so far. We used an experimental approach to quantify the relative importance of genetics (heritability) and past experience (age, familiarity with the environment, previous breeding success, previous information use) in heterospecific social information use for nest site selection in wild collared flycatchers (Ficedula albicollis). Flycatchers collect social information from resident tits for nest site selection. We created an apparent preference of tits for a novel nest site feature and recorded choices of flycatchers (copying or rejecting the tit preference). Copying behavior was stronger for naive individuals but also differed between years, which could be explained by contrasting seasonality in the demonstrator species. Past experience as reflected by age affected subsequent use of social information: pairs with a yearling male were more likely to copy the heterospecific preference than pairs with older immigrant males. There was no general pattern in successive individual choices over the years. Accordingly, individual repeatability in copying tit preference was very low. At the pair level, we estimated sex-specific direct and indirect genetic effects on the joint nest site decision and found no sex-specific heritability and no cross-sex genetic correlation. Our results confirm the importance of past experience for social information use and suggest that social information use is highly plastic and most likely not genetically inherited in collared flycatchers. Whether individuals use social information should be related to environmentally-induced changes in the quality of information and thus be context-dependent. Selection may therefore act on the ability to optimally use social information in varying environments and on the processes underlying such adjustment, such as learning, rather than the use of information itself.

show abstract

Section: Implementation Of Modelsmentioning

confidence: 99%

Heterospecific Nest Site Copying Behavior in a Wild Bird: Assessing the Influence of Genetics and Past Experience on a Joint Breeding Phenotype

et al. 2018

View full text Add to dashboard Cite

show abstract

“…As stated above, social psychological studies are frequently underpowered given the small effect sizes observed in the field. This likely inflates the rate of false-positive findings in the published literature that are later unreplicable (Forstmeier, Wagenmakers, & Parker, 2016;Ioannidis, 2005;Lakens & Evers, 2014).…”

Section: The Important (Neglect Of) Statistical Powermentioning

confidence: 99%

Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future

Świątkowski¹,

Dompnier

2017

rips

View full text Add to dashboard Cite

Over the last few years, psychology researchers have become increasingly preoccupied with the question of whether findings from psychological studies are generally replicable. The debates have originated from some unfortunate events of scientific misconduct in the field, and they have reached a climax with the recent discovery of a relatively weak rate of replicability of published literature, leading to the so-called replicability crisis in psychology.The present paper is concerned with examining the issue of replicability in the field of social psychology. We begin by drawing a state of the art of the crisis in this field. We then highlight some possible causes for the crisis, discussing topics of statistical power, questionable research practices, publication standards, and hidden auxiliary assumptions of context-dependency of social psychological theories. Finally, we argue that given the absence of absolute falsification in science, social psychology could greatly benefit from adopting McGuire's perspectivist approach to knowledge construction.

show abstract

“…The problem that has been pointed out by Nosek, Wagenmakers, and others (Bender & Lange, 2001;Dahl et al, 2008;De Groot, 1956/2014Forstmeier et al, 2016;Nosek et al, 2017;Nosek & Lakens, 2014;Wagenmakers, 2016) is that although the precise number of tests is known in preregistered confirmatory analyses, it is not usually known in exploratory analyses. This is because exploratory analyses tend to involve a lot of tests, and only a subset of those tests is documented.…”

Section: Why Is Multiple Testing Problematic In Exploratory Analyses?mentioning

confidence: 99%

“…The replication crisis (e.g., Munafò et al, 2017) has led several researchers to conclude that it is inappropriate to interpret p values in exploratory analyses (Nosek, Ebersole, DeHaven, & Mellor, 2017;Nosek & Lakens, 2014;Forstmeier, Wagenmakers, & Parker, 2016;Wagenmakers, 2016; see also Dahl, Grotle, Benth, & Natvig, 2008;De Groot, 1956/2014. These researchers argue that "in exploratory analysis, p-values lose their meaning due to an unknown inflation of the alpha-level" (Nosek & Lakens, 2014, p. 138).…”

mentioning

confidence: 99%

Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate

Rubin¹

2017

Preprint

View full text Add to dashboard Cite

Citation: Rubin, M. (2017). Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate. Review of General Psychology, 21, 269-275. doi: 10.1037/gpr0000123 Abstract Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014;Wagenmakers, 2016). For this argument to be tenable, the familywise error rate must be defined in relation to the number of hypotheses that are tested in the same study or article. Under this conceptualization, the familywise error rate is usually unknowable in exploratory analyses because it is usually unclear how many hypotheses have been tested on a spontaneous basis and then omitted from the final research report. In the present article, I argue that it is inappropriate to conceptualize the familywise error rate in relation to the number of hypotheses that are tested. Instead, it is more appropriate to conceptualize familywise error in relation to the number of different tests that are conducted on the same null hypothesis in the same study. Under this conceptualization, alpha level adjustments in exploratory analyses are (a) less necessary and (b) objectively verifiable. As a result, p values do not lose their meaning in exploratory analyses.

show abstract

Detecting and avoiding likely false‐positive findings – a practical guide

Cited by 361 publications

References 116 publications

Heterospecific Nest Site Copying Behavior in a Wild Bird: Assessing the Influence of Genetics and Past Experience on a Joint Breeding Phenotype

Heterospecific Nest Site Copying Behavior in a Wild Bird: Assessing the Influence of Genetics and Past Experience on a Joint Breeding Phenotype

Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future

Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate

Contact Info

Product

Resources

About